Data processing method, device, equipment and readable storage medium

Abstract

The embodiments of the present application provide a data processing method, apparatus, device and readable storage medium, wherein the client refreshes the target web page in response to a preset refresh instruction, and the target web page is the web page indicated by the preset target URL. Target data is obtained, and the target data is the data located at a preset target position in the target web page. Based on the target data and historical target data, it is determined whether there is a data update, and the historical target data is the data located at the target position before the target web page is refreshed. If there is a data update, a change notification is sent to the server. In response to receiving the change notification, the first preset process is executed according to the change notification, and the change notification indicates that there is a data update on the target web page. As can be seen from the above, the client determines whether the target web page has a data change based on the target data, and the server processes the changed web page data, without the need to collect all the page data, thereby improving monitoring efficiency and saving computing resources.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, and readable storage medium. Background Art

[0002] With the rapid development of the data and information industry, web crawlers have become an indispensable tool for people to access information in the vast online world. In existing technologies, because clients are unable to perform complex calculations, servers typically use web crawlers to periodically collect full data from web pages and analyze whether any data changes have occurred. However, due to the large amount of computation required and the time-consuming nature of full data collection and analysis, web page data changes cannot be detected in a timely manner. As a result, existing technologies are not efficient and cannot process data changes in a timely manner. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, and readable storage medium for improving the efficiency of data processing, as follows:

[0004] A data processing method, applied to a client, comprising:

[0005] Refreshing a target webpage in response to a preset refresh instruction, wherein the target webpage is a webpage indicated by a preset target URL;

[0006] Acquiring target data, wherein the target data is data located at a preset target location in the target webpage;

[0007] Determining whether there is data update based on the target data and historical target data, wherein the historical target data is the data located at the target location before refreshing the target webpage;

[0008] If there is a data update, a change notification is sent to the server, so that the server responds to the change notification, obtains the webpage data of the target webpage, compares the webpage data of the target webpage with the historical data of the target webpage, and obtains the change data as the data processing result.

[0009] Optionally, before refreshing the target webpage in response to a preset refresh instruction, the method further includes:

[0010] Establishing a monitoring task based on preset parameters; the preset parameters include a preset refresh cycle, the target website, and the target location;

[0011] The refresh instruction is issued in response to the refresh time indicated by the preset refresh cycle.

[0012] Optionally, the preset parameters also include: heartbeat cycle and heartbeat information;

[0013] After the monitoring task is established, the method further includes: sending the heartbeat information to the server in response to reaching the sending time indicated by the heartbeat cycle.

[0014] A data processing method, applied to a server, comprising:

[0015] In response to receiving the change notification, obtaining webpage data of the target webpage according to the change notification; the change notification is sent to the server by the client after determining that there is data update based on the target data and historical target data, the target data being data located at a preset target location in the target webpage;

[0016] The webpage data of the target webpage is compared with the historical data of the target webpage to obtain change data as a data processing result.

[0017] Optionally, it also includes:

[0018] In response to receiving the change notification, generating a push message based at least on the change notification, the push message at least indicating that data on the target webpage has been updated;

[0019] Send the push message to a preset account.

[0020] Optionally, it also includes:

[0021] In response to receiving the change notification, obtaining a first processing identifier and a second processing identifier of the change notification, wherein the first processing identifier is one of a valid identifier and an invalid identifier, and the second processing identifier is one of a completed identifier and an uncompleted identifier;

[0022] The comparing the webpage data of the target webpage with the historical data of the target webpage to obtain the change data as the data processing result includes:

[0023] If the first processing identifier is the valid identifier and the second processing identifier is the unfinished identifier, the webpage data of the target webpage is compared with the historical data of the target webpage to obtain change data as a data processing result.

[0024] Optionally, it also includes:

[0025] In response to receiving a heartbeat message sent by a client, determining, based on the heartbeat message, whether the heartbeat message is the same as a preset heartbeat message; the heartbeat message is sent by the client in response to reaching a sending time indicated by a preset heartbeat period;

[0026] If the heartbeat information is different from the preset heartbeat information, a preset abnormal warning message is sent to the preset account, and the abnormal warning message indicates that the client is running abnormally.

[0027] A data processing system includes a server and at least one client in communication with the server, wherein the client is pre-configured in a preset browser in the form of a plug-in;

[0028] The client is used to implement each step of the data processing method described above;

[0029] The server is used to implement each step of the data processing method described above.

[0030] A client, comprising:

[0031] A page refresh unit, configured to refresh a target webpage in response to a preset refresh instruction, wherein the target webpage is a webpage indicated by a preset target URL;

[0032] a target data acquisition unit, configured to acquire target data, wherein the target data is data located at a preset target position in the target webpage;

[0033] an update determination unit, configured to determine whether there is a data update based on the target data and historical target data, wherein the historical target data is data located at the target location before refreshing the target webpage;

[0034] The notification unit is used to send a change notification to the server if there is a data update, so that the server responds to the change notification, obtains the web page data of the target web page, compares the web page data of the target web page with the historical data of the target web page, and obtains the change data as the data processing result.

[0035] A server includes:

[0036] an update processing unit configured to, in response to receiving a change notification, obtain webpage data of a target webpage based on the change notification; the change notification is sent to the server by the client after determining that there is a data update based on the target data and historical target data, the target data being data located at a preset target location in the target webpage;

[0037] The result acquisition unit is used to compare the webpage data of the target webpage with the historical data of the target webpage, and acquire the change data as the data processing result.

[0038] A data processing device comprising: a memory and a processor;

[0039] The memory is used to store programs;

[0040] The processor is used to execute the program to implement the various steps of the data processing method described above.

[0041] A readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, the computer program implements the various steps of the data processing method described above.

[0042] It can be seen from the above technical solution that the data processing method, apparatus, device and readable storage medium provided by the embodiment of the present application, the client responds to a preset refresh instruction to refresh the target web page, and the target web page is the web page indicated by the preset target URL. The target data is obtained, and the target data is the data located at the preset target position in the target web page. Based on the target data and the historical target data, it is determined whether there is a data update, and the historical target data is the data located at the target position before the target web page is refreshed. If there is a data update, a change notification is sent to the server. Since the change notification indicates that there is a data update on the target web page, the server responds to the receipt of the change notification, and based on the change notification, obtains the web page data of the target web page, compares the web page data of the target web page with the historical data of the target web page, and obtains the change data as the data processing result. From the above, it can be seen that since the target data is the data at the target location, that is, this method can determine whether the data of the target web page has changed without the client collecting and analyzing the full amount of page data. In this way, changes in web page data can be discovered in a timely manner, thereby improving monitoring efficiency. Furthermore, after the client determines the data change, a change notification is sent to the server. The server obtains the data processing result based on the change notification by comparing the web page data of the target web page with the historical data of the target web page, thereby avoiding the defect that the client is unable to perform complex calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 A schematic diagram of the structure of a data processing system provided in an embodiment of the present application;

[0045] Figure 2 A flowchart of a configuration method provided in an embodiment of the present application;

[0046] Figure 3 A flowchart of a specific implementation method of a data processing method provided in an embodiment of the present application;

[0047] Figure 4 A flowchart of a specific implementation method of another data processing method provided in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of the structure of a client provided in an embodiment of the present application;

[0049] Figure 6 A schematic diagram of the structure of a server provided in an embodiment of the present application;

[0050] Figure 7 A flowchart of a data processing method provided in an embodiment of the present application;

[0051] Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0052] Figure 9 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0054] The data processing method provided in the embodiment of the present application can be applied to a data processing system, which includes a client and a server, wherein the client is pre-configured in a preset browser in the form of a browser plug-in, and the server runs on a server (including a single-point server or an integrated server) or in the cloud. This solution realizes real-time monitoring of at least one web page in the browser through communication between the client and the server.

[0055] like Figure 1 As shown, the data processing system includes a server and N clients respectively connected to the server for communication. Each client is pre-configured in a browser. After the server completes the task configuration of the client, a data processing method provided in an embodiment of the present application is implemented through data interaction between the server and the client. Figure 2 The following is a schematic diagram illustrating a server configuring a target client, wherein the target client is any client that is connected to the server for communication.

[0056] like Figure 2 As shown, this method specifically includes:

[0057] S201 : The server sends preset parameters of a preset task to a target client in response to a configuration completion instruction.

[0058] In this embodiment, the preset parameters include: a target URL, a heartbeat interface, a heartbeat period, a data interface, an anchor point location (referred to as anchor point xpath), and a target location (referred to as target xpath).

[0059] It should be noted that the configuration completion instruction is issued in response to the user clicking a preset configuration completion button on a preset human-computer interaction interface, which is used to provide an editable interface for the user to input preset parameters. Figure 2 An optional human-computer interaction interface diagram is illustrated.

[0060] like Figure 2 As shown, the task description input box 21 is used to input a task description of a preset task, for example, "monitoring the update status of the first information of the first webpage".

[0061] The target URL input box 22 is used to input a preset target URL, which is the URL of the target web page.

[0062] The heartbeat interface input box 23 is used to input a preset heartbeat interface, which is the server heartbeat interface address.

[0063] The heartbeat cycle input box 24 is used to input a preset heartbeat cycle, where the heartbeat cycle is the time interval between the sending times of two adjacent heartbeat messages.

[0064] The data interface input box 25 is used to input a preset data interface, which is the submission address of the change notification.

[0065] The anchor xpath input box 26 is used to input a preset anchor xpath, where the anchor xpath is the position of the element indicating that the target webpage has been loaded, on the target webpage.

[0066] The target xpath input box 27 is used to input a preset target xpath, where the target xpath is the position of the element to be monitored on the target web page.

[0067] It should be noted that the anchor XPath and the target XPath satisfy the following requirement: when refreshing the target webpage, the loading order of the anchor XPath and the target XPath is that the target XPath is loaded before the anchor XPath.

[0068] The save button 28 is a preset configuration completion button, which is used to issue a configuration completion instruction in response to a click operation.

[0069] It should be noted that Figure 2 The human-computer interaction interface shown is pre-configured on the server. In response to the user completing the above inputs on the human-computer interaction interface of the server and clicking the Save button, the server sends the above preset parameters to the target client to implement the operation configuration of the target client.

[0070] S202: The target client receives preset parameters of the preset task and establishes a monitoring task.

[0071] It should be noted that S201 to S202 are the process of configuring the target client on the server side, that is, the process of establishing a monitoring task on the target client on the server side. In this embodiment, the data processing system may include multiple clients, and the server side may establish multiple different tasks on each client side. The process of the server side establishing each task on each client side can be found in the above S201 to S202, which will not be elaborated on in this embodiment.

[0072] It should be noted that Figure 2 This is only an example of an optional process for configuring a client to establish a monitoring task. In actual applications, the process of establishing a monitoring task may also include the client updating the monitoring task in response to a parameter modification instruction, where the parameter modification instruction is input by the user through a preset interactive interface on the client. In other words, the client provides an interactive interface for the user to modify various parameters in the monitoring task.

[0073] Figure 3 An example of a specific implementation method of the data processing method provided in the embodiment of the present application is given. Figure 3 In the process shown, any client (referred to as the target client) in the data processing system is still taken as an example to exemplify the data processing method provided in an embodiment of the present application by data interaction between the client and the server.

[0074] S301: The client refreshes a target webpage in response to a preset refresh instruction.

[0075] In this embodiment, the client monitors and refreshes the target webpage according to the target URL, and the refresh instruction is pre-configured to be generated periodically, for example, a refresh instruction is generated every hour.

[0076] S302: In response to monitoring that the anchor point xpath is refreshed, the client obtains target data.

[0077] In this embodiment, the target data is obtained by parsing the element at the target xpath in the target webpage. It should be noted that the client monitors the target webpage according to the target URL, and when the anchor xpath is loaded, it is determined that the anchor xpath is refreshed.

[0078] For specific monitoring methods and methods for obtaining target elements, please refer to the prior art.

[0079] S303: The client determines whether there is data update based on the target data and historical target data.

[0080] In this embodiment, the historical target data is the target data obtained by the client when refreshing the target webpage last time, that is, the data obtained by parsing the element at the target xpath in the target webpage last time. The specific method for determining whether there is data update can be referred to the prior art.

[0081] S304: If the client determines that there is a data update, it sends a change notification to the server.

[0082] In this embodiment, the change notification indicates that there is a data update on the target webpage. It should be noted that the change notification may specifically include an identifier that uniquely indicates the target webpage, such as a preset identifier of the client, a preset identifier of the target webpage, and / or a preset identifier of the monitoring task, and the client sends the change notification via a preset data interface. For details, please refer to the prior art.

[0083] S305: In response to receiving the change notification, the server obtains the first processing identifier and the second processing identifier of the change notification.

[0084] In this embodiment, the first processing flag of the change notification is one of a valid flag or an invalid flag of the change notification, where the valid flag indicates that the change notification is valid, and the invalid flag indicates that the change notification is invalid. The second processing flag is one of a completed flag or an incomplete flag, where the completed flag indicates that the change notification has been processed, and the incomplete flag indicates that the change notification has not been processed.

[0085] Methods for determining whether a change notification is valid include:

[0086] Determine whether the change notification is sent repeatedly. If so, it is invalid.

[0087] Determine whether the change notification includes the preset update information, if not, it is invalid.

[0088] S306: If the first processing identifier of the change notification is a valid identifier and the second processing identifier is an unfinished identifier, the server executes the first preset process.

[0089] The first preset process at least includes: obtaining webpage data of the target webpage, comparing the webpage data of the target webpage with historical data of the target webpage, and obtaining change data as a data processing result.

[0090] Specifically, the server calls the crawler platform to collect web page data of the target web page, verifies the parsed page data one by one with the historical data stored in the preset database, obtains the deleted, updated or newly added data as the change data, and synchronizes the change data to the preset database to update the preset database.

[0091] Optionally, the first preset process further includes: if an exception occurs in the processing of the first preset process, the server performs a preset fault-tolerant processing process.

[0092] Optionally, the first preset process also includes: generating a push message and sending it to a preset account to notify the preset person in charge that there is a data update on the target web page. For example, the push message is sent via SMS to the device bound to the mobile phone number of the preset operation and maintenance personnel.

[0093] S307: The server sends a completion status.

[0094] It should be noted that the server sends the completion status to the preset account to notify the operation and maintenance personnel that the server has completed the response to the change notification.

[0095] It can be seen from the above technical solution that a data processing method provided by the present application is applied to a data processing system, and the data processing system includes a client and a server. The client senses whether there is a data change on the target web page based on the preset parameters of the preset task obtained, and when there is a data change on the target web page, sends a change notification to the server, and the server processes the data change. Since the client determines whether the page to be monitored has changed through the target data and historical target data, if there is a data change, it sends a change notification to the server, and the server executes a predetermined processing process (first preset process) for the data change.

[0096] This solution has the following beneficial effects compared to the existing technology:

[0097] 1. This method involves refreshing the target web page and collecting target data on the client side. By comparing the target data with historical target data, it is determined whether data changes have occurred. Compared with the method in which the server periodically collects all web page data and compares the data to determine whether the page data has changed, this method does not require full collection of page data, thereby improving monitoring efficiency and saving computing resources.

[0098] 2. The server only needs to obtain the web page data of the target web page indicated by the change notification, and compare the web page data of the target web page with the historical data of the target web page to obtain the change data as the data processing result. On the one hand, it reduces the computing task of the server, and on the other hand, it avoids the defect that the client is unable to handle complex calculations.

[0099] In summary, this method integrates the server's ability to undertake large-scale computing and the client's ability to efficiently monitor data changes, thereby improving data processing efficiency and saving computing resources.

[0100] 3. After determining that the first processing identifier of the change notification is a valid identifier and the second processing identifier is an incomplete identifier, the server executes the first preset process, and after executing the first preset process, updates the second processing identifier to avoid wasting computing resources and time due to processing invalid change notifications, and also avoid processing errors caused by repeated processing of change notifications.

[0101] It should be noted that Figure 3 The method shown is only an optional specific implementation method of the data processing method provided in this application. Figure 4 This paper illustrates a specific implementation method of another data processing method provided by this application. Figure 3 Compared with the process shown in FIG, this method adds the following S401 to S403.

[0102] S401: In response to reaching the sending time indicated by the heartbeat cycle, the client sends a preset heartbeat message to the server.

[0103] In this embodiment, the heartbeat cycle is the time interval between the sending times of two adjacent heartbeat messages. Optionally, the client sends the heartbeat message for the first time at a preset initial time, and periodically sends the heartbeat message according to the heartbeat cycle.

[0104] It should be noted that the heartbeat information is pre-configured information, and the client sends the heartbeat information through a preset heartbeat interface. The heartbeat information may include a client identifier or address. For details, please refer to the prior art and this embodiment does not limit it.

[0105] S402: In response to receiving the heartbeat information, the server determines the client operation status based on the heartbeat information.

[0106] In this embodiment, the method for determining the client operation status based on the heartbeat information includes:

[0107] Determine whether the heartbeat information is consistent with the preset heartbeat information. If so, it is determined that the client operation status is normal. If not, it is determined that the client operation status is abnormal.

[0108] S403: In response to the abnormal operation state of the client, the server executes a second preset process.

[0109] It should be noted that the second preset process includes:

[0110] 1. Obtain the webpage data of the target webpage, compare the webpage data of the target webpage with the historical webpage data of the target webpage, and determine whether the data is updated.

[0111] 2. Generate an abnormal warning message and send it to a preset account to notify the preset person in charge that the client's operating status is abnormal. For example, a push message is sent via SMS to the device bound to the preset operation and maintenance personnel's mobile phone number.

[0112] It should be noted that Figure 4An example is given of a specific method for determining whether the client is running normally based on heartbeat information. The present application also includes other specific methods for determining whether the client is running normally. For example, the server does not receive heartbeat information within a preset delay time and determines that the client's operating status is abnormal, wherein the preset delay time is obtained based on the heartbeat period. Optionally, the preset delay time is greater than n (n is greater than 1) times the heartbeat period.

[0113] Depend on Figure 4 As can be seen from the illustrated process, an embodiment of the present application provides a data processing method in which a client periodically sends preset heartbeat information to a server. The server determines the client's operating status based on the heartbeat information and, if the operating status is abnormal, executes a second preset process. This avoids the server not receiving timely notification of client changes when the client's operating status is abnormal, thereby preventing the server from failing to promptly execute the first preset process. Alternatively, it avoids the server failing to promptly perceive data changes on the target webpage due to abnormal client operation. Therefore, the accuracy of the data processing method is improved.

[0114] In summary, the data processing method provided by the present application includes a variety of optional specific implementation methods. Figure 5 is an optional structural diagram of a client for implementing a data processing method, such as Figure 5 As shown, the client includes:

[0115] The heartbeat sending module CP10 is used to submit preset heartbeat information to the server to report the client's operating status, thereby enabling the server to monitor the server's operating status. Among them, the heartbeat information is preset according to actual needs, for example, including location information, monitoring range and other information, see the above embodiment for details.

[0116] The automatic monitoring module CP20 is used to obtain and analyze page data (including but not limited to target data) of a target web page according to preset parameters of a preset task and to determine whether data changes occur.

[0117] Specifically, CP20 includes a page collection submodule CP201 ​​for refreshing and collecting target data, and a page analysis and judgment submodule CP202 for analyzing the collected target data in real time and comparing it with historical target data to determine whether the page data has changed. For details, see the above embodiment.

[0118] The event push module CP30 is used to submit the change event to the server after the client perceives the data change.

[0119] Figure 6 This is an optional structural diagram of a server for implementing a data processing method, such as Figure 6As shown, the server includes: a heartbeat monitoring module SP10, which is used to receive and process heartbeat information to monitor the running status of the client.

[0120] The event processing module SP20 is used to respond to the change notification after receiving the change notification submitted by the client. The specific response method is shown in the above embodiment.

[0121] The message push module SP30 is used to send push messages, for example, to push messages indicating abnormal client operation status to the account of the operation and maintenance personnel.

[0122] According to the above embodiments, the data processing method provided by the embodiment of the present application can be summarized as follows: Figure 7 The process shown in FIG. 1 may specifically include:

[0123] S701: The client responds to a preset refresh instruction and refreshes the target web page.

[0124] In this embodiment, the target webpage is a webpage indicated by a preset target URL.

[0125] S702: The client obtains target data.

[0126] In this embodiment, the target data is data located at a preset target position in the target webpage.

[0127] It should be noted that the target URL and target location are configured by the server and sent to the client. The client monitors the target web page based on the target URL. For an optional configuration process, see Figure 2 The process shown.

[0128] S703: The client determines whether there is data update based on the target data and the historical target data.

[0129] In this embodiment, the historical target data is the data located at the target location before refreshing the target webpage.

[0130] It should be noted that there are many methods for determining whether data has been updated. For example, comparing the keywords in the target data and the historical target data to see if they are the same, if not, then determining that data has been updated. For specific methods for determining whether data has been updated, refer to the prior art.

[0131] S704: If there is any data update on the client, the client sends a change notification to the server.

[0132] It should be noted that the method of sending the change notification to the server can be referred to in the prior art, for example, the change notification is sent via a preset data interface.

[0133] S705 : In response to receiving the change notification, the server obtains webpage data of the target webpage according to the change notification.

[0134] S706: The server compares the webpage data of the target webpage with the historical data of the target webpage, and obtains the change data as the data processing result.

[0135] From the above, it can be seen that since the target data is the data at the target location, that is, this method does not need to collect and analyze the full amount of page data, the client can determine whether the data of the target web page has changed, thereby timely discovering changes in web page data and improving monitoring efficiency. Furthermore, after the client determines the data change, it sends a change notification to the server, and the server processes the change notification to avoid the defect that the client is unable to handle complex calculations.

[0136] The present application also provides a client and a server. Figure 8 The schematic diagram of the structure of the client 81 and the server 82 is illustrated.

[0137] like Figure 8 As shown, the client 81 includes:

[0138] A page refresh unit 811 is configured to refresh a target web page in response to a preset refresh instruction, wherein the target web page is a web page indicated by a preset target URL;

[0139] A target data acquisition unit 812 is configured to acquire target data, wherein the target data is data located at a preset target position in the target webpage;

[0140] An update determination unit 813 is configured to determine whether there is a data update based on the target data and historical target data, where the historical target data is the data at the target location before the target webpage is refreshed;

[0141] The notification unit 814 is used to send a change notification to the server if there is a data update, so that the server responds to the change notification, obtains the web page data of the target web page, compares the web page data of the target web page with the historical data of the target web page, and obtains the change data as the data processing result.

[0142] Optionally, the client also includes:

[0143] The task establishing unit is used to establish a monitoring task according to preset parameters before refreshing the target web page in response to a preset refresh instruction; the preset parameters include a preset refresh cycle, the target website address and the target location.

[0144] The refresh instruction issuing unit is configured to issue the refresh instruction in response to the refresh time indicated by the preset refresh cycle.

[0145] Optionally, the preset parameters further include: heartbeat cycle and heartbeat information;

[0146] The client also includes: a heartbeat sending unit, which is used to send the heartbeat information to the server in response to reaching the sending time indicated by the heartbeat cycle after the monitoring task is established.

[0147] like Figure 8 As shown, the server 82 includes:

[0148] An update processing unit 821 is configured to, in response to receiving a change notification, obtain webpage data of the target webpage according to the change notification; the change notification is sent to the server by the client after determining that there is a data update based on the target data and historical target data, wherein the target data is the data located at a preset target location in the target webpage;

[0149] The result acquisition unit 822 is configured to compare the webpage data of the target webpage with the historical data of the target webpage, and acquire change data as a data processing result.

[0150] Optionally, the server further includes: an update message push unit, configured to:

[0151] In response to receiving the change notification, a push message is generated based at least on the change notification, the push message at least indicating that there is a data update on the target webpage; and the push message is sent to a preset account.

[0152] Optionally, the server further includes: a status acquisition unit, configured to: in response to receiving the change notification, acquire a first processing identifier and a second processing identifier of the change notification, wherein the first processing identifier is one of a valid identifier and an invalid identifier, and the second processing identifier is one of a completed identifier and an uncompleted identifier;

[0153] The result acquisition unit is used to compare the webpage data of the target webpage with the historical data of the target webpage, and obtain the change data as the data processing result, including: the result acquisition unit is specifically used to:

[0154] If the first processing identifier is the valid identifier and the second processing identifier is the unfinished identifier, the webpage data of the target webpage is compared with the historical data of the target webpage to obtain change data as a data processing result.

[0155] Optionally, the server further includes: a heartbeat abnormality judgment unit, configured to:

[0156] In response to receiving a heartbeat message sent by a client, determining, based on the heartbeat message, whether the heartbeat message is the same as a preset heartbeat message; the heartbeat message is sent by the client in response to reaching a sending time indicated by a preset heartbeat period;

[0157] If the heartbeat information is different from the preset heartbeat information, a preset abnormal warning message is sent to the preset account, and the abnormal warning message indicates that the client is running abnormally.

[0158] It should be noted that the data processing system provided in the embodiment of the present application includes a server and at least one client communicating with the server, and the client is pre-configured in a preset browser in the form of a plug-in (such as Figure 1 As shown), the structure of the server and any client can be found in Figure 8 .

[0159] Figure 9 Schematic diagram of the structure of the data processing device is shown, which may include: at least one processor 901, at least one communication interface 902, at least one memory 903 and at least one communication bus 904;

[0160] In the embodiment of the present application, the number of the processor 901, the communication interface 902, the memory 903, and the communication bus 904 is at least one, and the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904;

[0161] The processor 901 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention;

[0162] The memory 903 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0163] The memory stores a program, and the processor can execute the program stored in the memory to implement the steps of a data processing method provided in an embodiment of the present application, as follows:

[0164] A data processing method, applied to a client, comprising:

[0165] Refreshing a target webpage in response to a preset refresh instruction, wherein the target webpage is a webpage indicated by a preset target URL;

[0166] Acquiring target data, wherein the target data is data located at a preset target location in the target webpage;

[0167] Determining whether there is data update based on the target data and historical target data, wherein the historical target data is the data located at the target location before refreshing the target webpage;

[0168] If there is a data update, a change notification is sent to the server, so that the server responds to the change notification, obtains the webpage data of the target webpage, compares the webpage data of the target webpage with the historical data of the target webpage, and obtains the change data as the data processing result.

[0169] Optionally, before refreshing the target webpage in response to a preset refresh instruction, the method further includes:

[0170] Establishing a monitoring task based on preset parameters; the preset parameters include a preset refresh cycle, the target website, and the target location;

[0171] The refresh instruction is issued in response to the refresh time indicated by the preset refresh cycle.

[0172] Optionally, the preset parameters also include: heartbeat cycle and heartbeat information;

[0173] After the monitoring task is established, the method further includes: sending the heartbeat information to the server in response to reaching the sending time indicated by the heartbeat cycle.

[0174] A data processing method, applied to a server, comprising:

[0175] In response to receiving the change notification, obtaining webpage data of the target webpage according to the change notification; the change notification is sent to the server by the client after determining that there is data update based on the target data and historical target data, the target data being data located at a preset target location in the target webpage;

[0176] The webpage data of the target webpage is compared with the historical data of the target webpage to obtain change data as a data processing result.

[0177] Optionally, it also includes:

[0178] In response to receiving the change notification, generating a push message based at least on the change notification, the push message at least indicating that data on the target webpage has been updated;

[0179] Send the push message to a preset account.

[0180] Optionally, it also includes:

[0181] In response to receiving the change notification, obtaining a first processing identifier and a second processing identifier of the change notification, wherein the first processing identifier is one of a valid identifier and an invalid identifier, and the second processing identifier is one of a completed identifier and an uncompleted identifier;

[0182] The comparing the webpage data of the target webpage with the historical data of the target webpage to obtain the change data as the data processing result includes:

[0183] If the first processing identifier is the valid identifier and the second processing identifier is the unfinished identifier, the webpage data of the target webpage is compared with the historical data of the target webpage to obtain change data as a data processing result.

[0184] Optionally, it also includes:

[0185] In response to receiving a heartbeat message sent by a client, determining, based on the heartbeat message, whether the heartbeat message is the same as a preset heartbeat message; the heartbeat message is sent by the client in response to reaching a sending time indicated by a preset heartbeat period;

[0186] If the heartbeat information is different from the preset heartbeat information, a preset abnormal warning message is sent to the preset account, and the abnormal warning message indicates that the client is running abnormally.

[0187] The present application also provides a readable storage medium, which may store a computer program suitable for execution by a processor. When the computer program is executed by the processor, the steps of a data processing method provided in the present application are as follows:

[0188] A data processing method, applied to a client, comprising:

[0189] Refreshing a target webpage in response to a preset refresh instruction, wherein the target webpage is a webpage indicated by a preset target URL;

[0190] Acquiring target data, wherein the target data is data located at a preset target location in the target webpage;

[0191] Determining whether there is data update based on the target data and historical target data, wherein the historical target data is the data located at the target location before refreshing the target webpage;

[0192] If there is a data update, a change notification is sent to the server, so that the server responds to the change notification, obtains the webpage data of the target webpage, compares the webpage data of the target webpage with the historical data of the target webpage, and obtains the change data as the data processing result.

[0193] Optionally, before refreshing the target webpage in response to a preset refresh instruction, the method further includes:

[0194] Establishing a monitoring task based on preset parameters; the preset parameters include a preset refresh cycle, the target website, and the target location;

[0195] The refresh instruction is issued in response to the refresh time indicated by the preset refresh cycle.

[0196] Optionally, the preset parameters also include: heartbeat cycle and heartbeat information;

[0197] After the monitoring task is established, the method further includes: sending the heartbeat information to the server in response to reaching the sending time indicated by the heartbeat cycle.

[0198] A data processing method, applied to a server, comprising:

[0199] In response to receiving the change notification, obtaining webpage data of the target webpage according to the change notification; the change notification is sent to the server by the client after determining that there is data update based on the target data and historical target data, the target data being data located at a preset target location in the target webpage;

[0200] The webpage data of the target webpage is compared with the historical data of the target webpage to obtain change data as a data processing result.

[0201] Optionally, it also includes:

[0202] In response to receiving the change notification, generating a push message based at least on the change notification, the push message at least indicating that data on the target webpage has been updated;

[0203] Send the push message to a preset account.

[0204] Optionally, it also includes:

[0205] In response to receiving the change notification, obtaining a first processing identifier and a second processing identifier of the change notification, wherein the first processing identifier is one of a valid identifier and an invalid identifier, and the second processing identifier is one of a completed identifier and an uncompleted identifier;

[0206] The comparing the webpage data of the target webpage with the historical data of the target webpage to obtain the change data as the data processing result includes:

[0207] If the first processing identifier is the valid identifier and the second processing identifier is the unfinished identifier, the webpage data of the target webpage is compared with the historical data of the target webpage to obtain change data as a data processing result.

[0208] Optionally, it also includes:

[0209] In response to receiving a heartbeat message sent by a client, determining, based on the heartbeat message, whether the heartbeat message is the same as a preset heartbeat message; the heartbeat message is sent by the client in response to reaching a sending time indicated by a preset heartbeat period;

[0210] If the heartbeat information is different from the preset heartbeat information, a preset abnormal warning message is sent to the preset account, and the abnormal warning message indicates that the client is running abnormally.

[0211] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0212] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0213] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, characterized in that: Applied to the client, including: Refreshing a target webpage in response to a preset refresh instruction, the target webpage being a webpage indicated by a preset target URL, wherein the client monitors and refreshes the target webpage according to the target URL, and the server pre-configures a monitoring task on the client; In response to monitoring that the anchor point position is refreshed, acquiring target data, the target data being data located at a preset target position in the target webpage; Determining whether there is data update based on the target data and historical target data, wherein the historical target data is the data located at the target location before refreshing the target webpage; If there is a data update, a change notification is sent to the server, so that the server, in response to the change notification, obtains the webpage data of the target webpage, compares the webpage data of the target webpage with the historical data of the target webpage, and obtains the change data as a data processing result; in response to reaching the sending time indicated by the heartbeat cycle, a heartbeat message is sent to the server, so that the server determines the operating status of the client based on the heartbeat information, and when it is determined that the operating status of the client is abnormal, executes a second preset process, the second preset process including: obtaining the webpage data of the target webpage, comparing the webpage data of the target webpage with the historical webpage data of the target webpage, and determining whether a data update occurs; the heartbeat information is pre-configured information, including a client identifier or address; Before refreshing the target webpage in response to the preset refresh instruction, the method further includes: Establishing a monitoring task based on preset parameters; the preset parameters include a preset refresh cycle, a heartbeat interface, a heartbeat cycle, a data interface, an anchor point location, the target URL, and the target location; the preset parameters are sent by the server to the target client in response to a configuration completion instruction; the configuration completion instruction is issued in response to a user clicking a preset configuration completion button on a preset human-computer interaction interface, the human-computer interaction interface being used to provide an editable interface for the user to input the preset parameters; the process of establishing the monitoring task further includes: the client updating the monitoring task in response to the parameter modification instruction; the anchor point location is the location on the target webpage of an element indicating that the target webpage has been loaded; The refresh instruction is issued in response to the refresh time indicated by the preset refresh cycle.

2. A data processing method, characterized in that: Applied to the server, including: In response to a configuration completion instruction, preset parameters of a preset task are sent to a target client, wherein the preset parameters include a preset refresh period, a heartbeat interface, a heartbeat period, a data interface, an anchor position, a target URL, and a target position; the configuration completion instruction is issued in response to a user clicking a preset configuration completion button on a preset human-computer interaction interface, wherein the human-computer interaction interface is used to provide an editable interface for the user to input the preset parameters; wherein the target client refreshes a target webpage in response to the preset refresh instruction, wherein the target webpage is a webpage indicated by a preset target URL, wherein the target client monitors and refreshes the target webpage according to the target URL, the server pre-configures a monitoring task on the client, and the anchor position is the position on the target webpage of an element indicating that the target webpage has been loaded; In response to receiving the change notification, obtaining webpage data of the target webpage according to the change notification; the change notification is sent to the server by the client after determining that there is data update based on the target data and historical target data, the target data being data located at a preset target location in the target webpage; Comparing the webpage data of the target webpage with the historical data of the target webpage to obtain change data as a data processing result; In response to receiving the heartbeat information sent by the client, based on the heartbeat information, it is determined whether the heartbeat information is the same as the preset heartbeat information. When it is determined that the client's operating status is abnormal, a second preset process is executed. The second preset process includes: obtaining the web page data of the target web page, comparing the web page data of the target web page with the historical web page data of the target web page, and determining whether a data update occurs. The heartbeat information is sent by the client in response to reaching the sending time indicated by the preset heartbeat cycle; the heartbeat information is pre-configured information, including a client identifier or address.

3. The method according to claim 2, characterized in that Also includes: In response to receiving the change notification, generating a push message based at least on the change notification, the push message at least indicating that data on the target webpage has been updated; Send the push message to a preset account.

4. The method according to claim 3, characterized in that Also includes: In response to receiving the change notification, obtaining a first processing identifier and a second processing identifier of the change notification, wherein the first processing identifier is one of a valid identifier and an invalid identifier, and the second processing identifier is one of a completed identifier and an uncompleted identifier; The comparing the webpage data of the target webpage with the historical data of the target webpage to obtain the change data as the data processing result includes: If the first processing identifier is the valid identifier and the second processing identifier is the unfinished identifier, the webpage data of the target webpage is compared with the historical data of the target webpage to obtain change data as a data processing result.

5. The method according to claim 3, characterized in that Also includes: If the heartbeat information is different from the preset heartbeat information, a preset abnormal warning message is sent to the preset account, and the abnormal warning message indicates that the client is running abnormally.

6. A data processing system, characterized in that: It includes a server and at least one client connected to the server for communication, wherein the client is pre-configured in a preset browser in the form of a plug-in; The client is used to implement each step of the data processing method according to claim 1; The server is used to implement each step of the data processing method according to any one of claims 2 to 5.

7. A client, characterized in that include: a page refresh unit, configured to refresh a target webpage in response to a preset refresh instruction, wherein the target webpage is a webpage indicated by a preset target URL, wherein the client monitors and refreshes the target webpage according to the target URL, and the server pre-configures a monitoring task on the client; a target data acquisition unit, configured to acquire target data in response to monitoring that the anchor point position is refreshed, wherein the target data is data located at a preset target position in the target webpage; an update determination unit, configured to determine whether there is a data update based on the target data and historical target data, wherein the historical target data is data located at the target location before refreshing the target webpage; a notification unit, configured to send a change notification to a server if there is a data update, so that the server, in response to the change notification, obtains webpage data of the target webpage, compares the webpage data of the target webpage with historical data of the target webpage, and obtains change data as a data processing result; The client is further configured to send heartbeat information to the server in response to reaching the sending time indicated by the heartbeat cycle, so that the server determines the operating status of the client based on the heartbeat information, and executes a second preset process when it is determined that the operating status of the client is abnormal. The second preset process includes: obtaining web page data of the target web page, comparing the web page data of the target web page with historical web page data of the target web page, and determining whether data update occurs; the heartbeat information is pre-configured information, including a client identifier or address; Before refreshing the target webpage in response to the preset refresh instruction, the method further includes: A monitoring task is established based on preset parameters; the preset parameters include a preset refresh cycle, a heartbeat interface, a heartbeat cycle, a data interface, an anchor point position, the target URL and the target position; the preset parameters are sent by the server to the target client in response to a configuration completion instruction; the configuration completion instruction is issued in response to a user clicking a preset configuration completion button on a preset human-computer interaction interface, and the human-computer interaction interface is used to provide an editable interface for the user to input preset parameters; the process of establishing the monitoring task also includes: the client responding to the parameter modification instruction to update the monitoring task; the anchor point position is the position on the target web page of the element indicating that the target web page has been loaded.

8. A server, characterized in that: include: The server is configured to send preset parameters of the preset task to the target client in response to the configuration completion instruction, wherein the preset parameters include a preset refresh period, a heartbeat interface, a heartbeat period, a data interface, an anchor point location, a target website address, and a target location; The configuration completion instruction is issued in response to a user clicking a preset configuration completion button on a preset human-computer interaction interface, the human-computer interaction interface being used to provide an editable interface for the user to input preset parameters; wherein the target client refreshes a target webpage in response to a preset refresh instruction, the target webpage being a webpage indicated by a preset target URL, wherein the target client monitors and refreshes the target webpage according to the target URL, the server pre-configuring a monitoring task on the client, and the anchor point position being the position on the target webpage of an element indicating that loading of the target webpage is complete; an update processing unit configured to, in response to receiving a change notification, obtain webpage data of a target webpage based on the change notification; the change notification is sent to the server by the client after determining that there is a data update based on the target data and historical target data, the target data being data located at a preset target location in the target webpage; a result acquisition unit, configured to compare the webpage data of the target webpage with the historical data of the target webpage, and acquire change data as a data processing result; The server is further configured to, in response to receiving heartbeat information sent by the client, determine, based on the heartbeat information, whether the heartbeat information is the same as preset heartbeat information; and, when it is determined that the client's operating status is abnormal, execute a second preset process, the second preset process including: obtaining web page data of the target web page, comparing the web page data of the target web page with the historical web page data of the target web page, and determining whether a data update has occurred, wherein the heartbeat information is sent by the client in response to reaching the sending time indicated by the preset heartbeat cycle; and the heartbeat information is pre-configured information including a client identifier or address.

9. A data processing device, characterized in that: include: memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the data processing method according to any one of claims 1 or 2 to 5.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the data processing method according to any one of claims 1 or 2 to 5 is implemented.

Images