A data processing method and related device

Abstract

The embodiment of the present application discloses a data processing method, the method comprising: determining that a target website updates interactive data; acquiring the interactive data; if an abnormality is detected in the process of acquiring the interactive data, determining the historical operation behavior corresponding to the abnormality; and executing the historical operation behavior. Through this method, after determining that the target website has updated the interactive data, the processing device can automatically acquire the data and detect the data acquisition process. If an abnormality occurs, the historical operation behavior corresponding to the abnormality can be automatically repeated, thereby retrying to acquire data, improving the success rate and efficiency of data processing, and reducing the consumption of human resources.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and related devices. Background Art

[0002] In the Internet age, data collection and analysis are the main ways for relevant personnel to obtain information. For example, after an advertisement is placed, the effectiveness of the advertisement can be analyzed by collecting information such as the advertisement conversion rate and click-through rate.

[0003] In related technologies, data collection and analysis are mainly completed manually by relevant personnel. When the amount of data is large, this processing method will consume a lot of human resources and the efficiency of data processing is low. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a data processing method. After determining that the target website has updated the interactive data, the processing device can automatically obtain the changed data and detect the data acquisition process. If an abnormal situation occurs, the historical operation behavior corresponding to the abnormal situation can be automatically repeated, thereby retrying data acquisition, thereby improving the success rate and efficiency of data processing and reducing the consumption of human resources.

[0005] The embodiments of the present application disclose the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising:

[0007] Determine the target website to update interaction data;

[0008] Acquiring the interaction data;

[0009] If an abnormal situation occurs during the process of obtaining the interaction data, determining a historical operation behavior corresponding to the abnormal situation;

[0010] Execute the historical operation behavior.

[0011] In a possible implementation, the historical operation behavior is a historical operation behavior that is related to the abnormal situation and has a shortest time interval with a time node at which the abnormal situation occurs.

[0012] In a possible implementation, the method further includes:

[0013] Verifying whether the interaction data meets data acquisition requirements, wherein the data acquisition requirements include any one or more combinations of whether the amount of the interaction data meets a first preset value and whether the accuracy of the interaction data meets a second preset value;

[0014] If not satisfied, a first alarm message is sent to the user, where the first alarm message is used to instruct the user to check the interaction data.

[0015] In a possible implementation, the method further includes:

[0016] If an abnormal situation is detected in the process of obtaining the interaction data, a second alarm message is sent to the user.

[0017] In a possible implementation, the method further includes:

[0018] If an abnormal situation is detected, and the abnormal situation does not occur in the process of obtaining the interaction data, executing the step of determining the target website to update the interaction data;

[0019] A third alarm message is sent to the user.

[0020] In a possible implementation, the method further includes:

[0021] Converting the data format of the interactive data into a preset format;

[0022] The interactive data is transmitted to the application party via a data interface corresponding to the preset format.

[0023] In a possible implementation, the method further includes:

[0024] If it is determined that the interactive data transmission is completed, a data transmission completion message is sent to the user.

[0025] In a possible implementation, the method is implemented based on RPA technology.

[0026] In a second aspect, an embodiment of the present application provides a data processing device, the device comprising a first determining unit, an acquiring unit, a second determining unit, and a first executing unit:

[0027] The first determining unit is used to determine the target website to update the interaction data;

[0028] The acquisition unit is used to acquire the interaction data;

[0029] The second determining unit is configured to determine a historical operation behavior corresponding to the abnormal situation if an abnormal situation occurs during the acquisition of the interaction data;

[0030] The first execution unit is used to execute the historical operation behavior.

[0031] In a possible implementation, the historical operation behavior is a historical operation behavior that is related to the abnormal situation and has a shortest time interval with a time node at which the abnormal situation occurs.

[0032] In a possible implementation, the device further includes a verification unit and a first alarm unit:

[0033] The verification unit is used to verify whether the interaction data meets the data acquisition requirement, and the data acquisition requirement includes any one or more combinations of whether the amount of the interaction data meets the first preset value and whether the accuracy of the interaction data meets the second preset value;

[0034] The first alarm unit is used to send a first alarm message to the user if the conditions are not met, wherein the first alarm message is used to instruct the user to check the interaction data.

[0035] In a possible implementation, the device further includes a second alarm unit:

[0036] The second alarm unit is used to send a second alarm message to the user if an abnormal situation is detected in the process of obtaining the interaction data.

[0037] In a possible implementation, the device further includes a second execution unit and a third alarm unit:

[0038] The second execution unit is configured to execute the step of determining the target website to update the interaction data if an abnormal situation is detected and the abnormal situation does not occur in the process of obtaining the interaction data;

[0039] The third alarm unit is used to send third alarm information to the user.

[0040] In a possible implementation manner, the device further includes a conversion unit and a transmission unit:

[0041] The conversion unit is used to convert the data format of the interactive data into a preset format;

[0042] The transmission unit is used to transmit the interactive data to the application party through the data interface corresponding to the preset format.

[0043] In a possible implementation, the device further includes a sending unit:

[0044] The sending unit is used to send data transmission completion information to the user if it is determined that the interactive data transmission is completed.

[0045] In a possible implementation, the device is implemented based on RPA technology.

[0046] It can be seen from the above technical solution that the present application provides a data processing method, which allows the processing device to replace manual data processing. First, the processing device can determine that the target website updates the interactive data, and then automatically obtain the interactive data. During the acquisition process, the processing device can perform real-time monitoring. If an abnormal situation is detected in the process of obtaining the interactive data, the historical operation behavior corresponding to the abnormal situation can be determined, and the historical operation behavior can be executed. Among them, the historical operation behavior is the operation behavior that causes the abnormal situation to occur, so that the processing device can repeatedly try the historical operation behavior to try to successfully acquire the data without manual re-operation, which improves the processing efficiency while ensuring the success rate of data processing and saves the consumption of human resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of a data processing method in an actual application scenario provided by an embodiment of the present application;

[0050] Figure 3 A structural block diagram of a data processing device provided in an embodiment of the present application;

[0051] Figure 4 A structural block diagram of a data processing device provided in an embodiment of the present application;

[0052] Figure 5 A structural block diagram of a data processing device provided in an embodiment of the present application;

[0053] Figure 6 A structural block diagram of a data processing device provided in an embodiment of the present application;

[0054] Figure 7 A structural block diagram of a data processing device provided in an embodiment of the present application;

[0055] Figure 8 A structural block diagram of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The embodiments of the present application are described below in conjunction with the accompanying drawings.

[0057] In order to process and analyze data, in related technologies, it is usually necessary to manually collect relevant data from relevant web pages and analyze them. However, due to the Internet era, the number of Internet users has grown exponentially, which has led to a large amount of data to be processed in various websites or software. It is difficult to process data quickly and accurately with manual capabilities alone.

[0058] In order to solve the above technical problems, the present application provides a data processing method. After determining that the target website has updated the interactive data, the processing device can automatically obtain the changed data and detect the data acquisition process. If an abnormal situation occurs, the historical operation behavior corresponding to the abnormal situation can be automatically repeated, thereby retrying data acquisition, thereby improving the success rate and efficiency of data processing and reducing the consumption of human resources.

[0059] It is understandable that the method can be applied to a processing device, which is a processing device with a data processing function, for example, a terminal device or a server with a data processing function. The method can be applied to a network scenario in which a terminal device and a server communicate, and the terminal device and the server cooperate to operate. Among them, the terminal device can be a mobile phone, a desktop computer, a personal digital assistant (PDA for short), a tablet computer and other devices. The server can be understood as an application server or a Web server. In actual deployment, the server can be an independent physical server or a server cluster or distributed system composed of multiple physical servers. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here.

[0060] Next, a data processing method provided in an embodiment of the present application will be introduced in conjunction with the accompanying drawings.

[0061] See also Figure 1 , Figure 1 A flowchart of a data processing method provided in an embodiment of the present application, the method comprising:

[0062] S101: Determine the target website to update interaction data.

[0063] Among them, the target website is a website that can provide the data required for this data processing. For example, after an advertisement is placed on a website, the website can provide conversion rate, click-through rate, pageviews and other data related to the advertisement. Relevant personnel need to use these data to judge the effectiveness of the advertisement. At this time, the website can be set as the target website.

[0064] The processing device can log in to the target website through the target website's corresponding account and password, and monitor the data status on the target website in real time. If it is determined that the target website has updated the interaction data, the subsequent steps can be executed. Among them, the interaction data refers to the data generated by the user when browsing the target website.

[0065] S102: Acquire interaction data.

[0066] After determining that the target website has interaction data, the processing device may automatically obtain the interaction data from the website.

[0067] There are multiple methods for enabling the processing device to automatically acquire interactive data. In one possible implementation, relevant personnel can design a program code that enables the processing device to simulate the manual acquisition of data to acquire data, such as downloading relevant data by clicking on relevant locations in the website. In this way, the processing device can acquire data from the target website more reasonably, thereby improving the success rate of data acquisition.

[0068] S103: If an abnormal situation occurs during the process of detecting and acquiring the interactive data, determine the historical operation behavior corresponding to the abnormal situation.

[0069] In order to reduce the failure rate in the data acquisition process, the processing device can monitor the process of acquiring interactive data in real time. It is understandable that in the process of acquiring data, network anomalies, code errors, etc. may occur, which may cause the data acquisition action to fail. At this time, in order to eliminate the abnormal situation, the processing device can determine the historical operation behavior corresponding to the abnormal situation. The historical operation behavior refers to the operation behavior performed by the processing device in the process of acquiring interactive data before the abnormal situation occurs, such as click, copy, paste, pull, etc. The processing device can record the operation behavior it performs, and when an abnormal situation occurs, the historical operation behavior corresponding to the abnormal situation can be obtained from the record.

[0070] It is understandable that the abnormal situation may be caused by accidental problems. For example, when the abnormal situation is caused by network fluctuations and a certain data cannot be successfully downloaded, if the download action is repeated, the data may be successfully obtained after the problem disappears. It can be seen that by analyzing the historical operation behavior, it is helpful for the processing device to solve the abnormal situation that occurs in the process.

[0071] S104: Execute historical operation behavior.

[0072] After determining the historical operation behavior, in order to eliminate abnormal situations and successfully obtain interaction data, the processing device can execute the historical operation behavior. During the execution process, if abnormal situations still occur, the above steps of determining and executing the historical operation behavior will be repeated, thereby achieving the purpose of repeatedly executing the historical operation behavior before eliminating the abnormal situation.

[0073] In order to improve data processing efficiency, the historical operation behavior can be set more accurately. For example, in one possible implementation, the closer the historical operation behavior is to the time when the abnormal situation occurs, the less time it takes to repeat the operation. Therefore, the historical operation behavior can be the historical operation behavior that is related to the abnormal situation and has the smallest time interval with the time node that caused the abnormal situation.

[0074] In the process of executing the historical operation behavior, in order to improve the efficiency of data processing, a reasonable execution stop condition can also be set. For example, in a possible implementation, the execution of the historical operation behavior can be stopped after the interactive data is successfully obtained or the historical operation behavior is executed for a preset number of times, thereby avoiding some abnormal situations that cannot be solved by repeated execution of operations and wasting data processing time.

[0075] It can be seen from the above technical solution that the present application provides a data processing method, which allows the processing device to replace manual data processing. First, the processing device can determine that the target website updates the interactive data, and then automatically obtain the interactive data. During the acquisition process, the processing device can perform real-time monitoring. If an abnormal situation is detected in the process of obtaining the interactive data, the historical operation behavior corresponding to the abnormal situation can be determined, and the historical operation behavior can be executed. Among them, the historical operation behavior is the operation behavior that causes the abnormal situation to occur, so that the processing device can repeatedly try the historical operation behavior to try to successfully acquire the data without manual re-operation, which improves the processing efficiency while ensuring the success rate of data processing and saves the consumption of human resources.

[0076] In order to improve the data quality of the acquired interaction data, in a possible implementation, the processing device may also verify whether the interaction data meets the data acquisition requirements, and the data acquisition requirements are used to determine the data quality of the acquired interaction data. The data acquisition requirements may include any one or more combinations of whether the amount of interaction data meets a first preset value and whether the accuracy of the interaction data meets a second preset value, thereby improving the integrity and accuracy of the data to a certain extent.

[0077] For example, when the interaction data is data related to a certain advertisement, the interaction data may involve multiple data of multiple cities. At this time, the processing device can analyze the acquired interaction data to determine whether the number of cities and the number of categories of interaction data meet the requirements; the number of categories of data can be determined by information such as the number of tables and the number of rows in the table.

[0078] If not, the processing device may further send a first alarm message to the user, where the first alarm message is used to instruct the user to check the interaction data, thereby further improving the validity of the interaction data.

[0079] It can be understood that in addition to sending an alarm message when the interaction data is abnormal, in one possible implementation, in order to enable relevant users to be aware of abnormal situations in the data acquisition process as soon as possible, if an abnormal situation is detected in the process of acquiring interaction data, the processing device can also send a second alarm message to the user.

[0080] In addition, it is understandable that abnormal situations may occur not only in the process of acquiring interactive data, but also in other processes such as determining that the target website updates interactive data, executing historical operation behaviors, etc. Therefore, in order to further improve the success rate of data processing, in a possible implementation, if an abnormal situation is detected and the abnormal situation does not occur in the process of acquiring interactive data, the processing device can also perform the step of determining that the target website updates interactive data, so that when problems occur in other processes, the processing device can rerun the entire data processing process, thereby improving the success rate of data processing.

[0081] At the same time, the processing device may also send a third alarm message to the user. Since the third alarm message, the second alarm message and the first alarm message are different alarm messages, the relevant user may also be informed of the specific type of abnormal situation through the category of the alarm message.

[0082] In addition, after successfully acquiring the relevant data, in order to enable the relevant data to be transmitted through certain specific interfaces, the processing device can also convert the data format of the interactive data into a preset format, and transmit the interactive data to the application party through the data interface corresponding to the preset format. The preset format can be a format corresponding to the API interface, etc. Thus, the specific transmission of data can be completed without the relevant user performing data processing, further improving the efficiency of data processing.

[0083] After the data is sent, in order to enable the relevant users to be informed of the completion of the data sending in the first place, in a possible implementation, if it is determined that the interactive data is transmitted, the processing device may also send data transmission completion information to the user.

[0084] It is understandable that in the above technical solution, the automatic execution of the processing device can be achieved in a variety of ways. In one possible implementation, the method can be implemented based on RPA technology, for example, the processing device can be a certain RPA robot. The characteristic of RPA robotic process automation is that it can replace or assist humans in completing repetitive work and tasks in digital devices such as computers and RPA phones.

[0085] As long as the usage rules are designed in advance, RPA can simulate manual operations such as copying, pasting, clicking, and inputting, and assist humans in completing a large number of tasks with "relatively fixed rules, high repetition, and low added value." RPA technology has two major characteristics: non-intrusiveness and flexible configuration: when enterprises deploy RPA, they do not need to change their existing information systems, thus avoiding the legacy system iceberg; RPA technology has very strong flexible configuration and can be very close to the enterprise's own business to achieve seamless integration. These two characteristics can enable RPA technology to be implemented in enterprises from small to large.

[0086] In order to facilitate understanding of the technical solution provided by the present application, a data processing method provided by an embodiment of the present application will be introduced below in combination with an actual application scenario.

[0087] See also Figure 2 , Figure 2 A schematic diagram of a data processing method in an actual application scenario provided in an embodiment of the present application, in which the processing device is an RPA robot.

[0088] After starting processing, the RPA robot can first determine whether the third-party website, such as an advertising website, has updated the interactive data. The interactive data can be the conversion rate, click-through rate, pageviews, etc. of a certain advertisement. If not updated, an alarm message is sent to the relevant personnel; if updated, the RPA robot can obtain the required interactive data and determine whether there is an abnormality in the data acquisition. If so, the action of obtaining the interactive data is re-executed; if not, it can be verified whether the data is completely obtained. If not completely obtained, an alarm message can be sent to the relevant personnel to prompt the relevant personnel to check the interactive data or code; if it has been completely obtained, the data format can be converted and the data can be compressed and sent to the application party. At the same time, in order to meet the data needs of other relevant personnel, the RPA robot can also send the compressed data to the relevant personnel by email and send notification information to the relevant personnel.

[0089] Based on the data processing method provided in the above embodiment, the present application embodiment also provides a data processing device, see Figure 3 , Figure 3This is a structural block diagram of a data processing device 300 provided in the present application, the device 300 includes a first determining unit 301, an acquiring unit 302, a second determining unit 303 and a first executing unit 304:

[0090] The first determination unit 301 is used to determine the target website update interaction data;

[0091] The acquisition unit 302 is used to acquire the interaction data;

[0092] The second determining unit 303 is configured to determine a historical operation behavior corresponding to the abnormal situation if an abnormal situation occurs during the process of acquiring the interaction data;

[0093] The first execution unit 304 is used to execute the historical operation behavior.

[0094] In a possible implementation, the historical operation behavior is a historical operation behavior that is related to the abnormal situation and has a shortest time interval with a time node at which the abnormal situation occurs.

[0095] In one possible implementation, see Figure 4 , the device 300 further includes a verification unit 305 and a first alarm unit 306:

[0096] The verification unit 305 is used to verify whether the interaction data meets the data acquisition requirement, and the data acquisition requirement includes any one or more combinations of whether the amount of the interaction data meets the first preset value and whether the accuracy of the interaction data meets the second preset value;

[0097] The first alarm unit 306 is used to send a first alarm message to the user if the conditions are not met, wherein the first alarm message is used to instruct the user to check the interaction data.

[0098] In one possible implementation, see Figure 5 , the device 300 further includes a second alarm unit 307:

[0099] The second alarm unit 307 is configured to send a second alarm message to the user if an abnormality is detected in the process of acquiring the interaction data.

[0100] In one possible implementation, see Figure 6 , the device 300 further includes a second execution unit 308 and a third alarm unit 309:

[0101] The second execution unit 308 is configured to execute the step of determining the target website to update the interaction data if an abnormality is detected and the abnormality does not occur in the process of obtaining the interaction data;

[0102] The third alarm unit 309 is used to send third alarm information to the user.

[0103] In one possible implementation, see Figure 7 , the device 300 further includes a conversion unit 310 and a transmission unit 311:

[0104] The conversion unit 310 is used to convert the data format of the interaction data into a preset format;

[0105] The transmission unit 311 is used to transmit the interactive data to the application party through the data interface corresponding to the preset format.

[0106] In one possible implementation, see Figure 8 , the apparatus 300 further includes a sending unit 312:

[0107] The sending unit 321 is configured to send data transmission completion information to the user if it is determined that the interactive data transmission is completed.

[0108] In a possible implementation, the method is implemented based on RPA technology.

[0109] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the above-mentioned storage medium can be at least one of the following media: read-only memory (English: read-only memory, abbreviated: ROM), RAM, magnetic disk or optical disk, etc. Various media that can store program codes.

[0110] It should be noted that each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, in which the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0111] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Determine the target website to update interaction data; Acquiring the interaction data; Verifying whether the interaction data meets data acquisition requirements, wherein the data acquisition requirements include any one or more combinations of whether the amount of the interaction data meets a first preset value and whether the accuracy of the interaction data meets a second preset value; If not satisfied, sending a first alarm message to the user, wherein the first alarm message is used to instruct the user to check the interaction data; If an abnormal situation occurs during the process of acquiring the interaction data, determine a historical operation behavior corresponding to the abnormal situation, wherein the historical operation behavior is a historical operation behavior related to the abnormal situation and having a minimum time interval with a time node at which the abnormal situation occurs, and the historical operation behavior is specifically an operation behavior performed during the process of acquiring the interaction data before the abnormal situation occurs; Execute the historical operation behavior.

2. The method according to claim 1, characterized in that The method further comprises: If an abnormal situation is detected in the process of obtaining the interaction data, a second alarm message is sent to the user.

3. The method according to claim 1, characterized in that The method further comprises: If an abnormal situation is detected, and the abnormal situation does not occur in the process of obtaining the interaction data, executing the step of determining the target website to update the interaction data; A third alarm message is sent to the user.

4. The method according to claim 1, characterized in that: The method further comprises: Converting the data format of the interactive data into a preset format; The interactive data is transmitted to the application party via a data interface corresponding to the preset format.

5. The method according to claim 4, characterized in that The method further comprises: If it is determined that the interactive data transmission is completed, a data transmission completion message is sent to the user.

6. The method according to any one of claims 1 to 5, characterized in that: The method is implemented based on RPA technology.

7. A data processing device, characterized in that: The device comprises a first determining unit, an acquiring unit, a verifying unit, a first alarming unit, a second determining unit and a first executing unit: The first determining unit is used to determine the target website to update the interaction data; The acquisition unit is used to acquire the interaction data; The verification unit is used to verify whether the interaction data meets the data acquisition requirement, and the data acquisition requirement includes any one or more combinations of whether the amount of the interaction data meets the first preset value and whether the accuracy of the interaction data meets the second preset value; The first alarm unit is used to send a first alarm message to the user if the conditions are not met, wherein the first alarm message is used to instruct the user to check the interaction data; The second determining unit is configured to determine, if an abnormal situation occurs during the process of acquiring the interaction data, a historical operation behavior corresponding to the abnormal situation, wherein the historical operation behavior is a historical operation behavior related to the abnormal situation and having a minimum time interval with a time node at which the abnormal situation occurs, and the historical operation behavior is specifically an operation behavior performed during the process of acquiring the interaction data before the abnormal situation occurs; The first execution unit is used to execute the historical operation behavior.

8. The device according to claim 7, characterized in that The device also includes a second alarm unit: The second alarm unit is used to send a second alarm message to the user if an abnormal situation is detected in the process of obtaining the interaction data.

9. The device according to claim 7, characterized in that The device also includes a second execution unit and a third alarm unit: The second execution unit is configured to execute the step of determining the target website to update the interaction data if an abnormal situation is detected and the abnormal situation does not occur in the process of obtaining the interaction data; The third alarm unit is used to send third alarm information to the user.

10. The device according to claim 7, characterized in that The device also includes a conversion unit and a transmission unit: The conversion unit is used to convert the data format of the interactive data into a preset format; The transmission unit is used to transmit the interactive data to the application party through the data interface corresponding to the preset format.

Images