Data processing task construction and execution method and device, storage medium and product

Abstract

One or more embodiments of the invention provide a data processing task construction and execution method and device, a storage medium and a product. The data processing task construction method comprises the steps that a predetermined task framework is obtained, the task framework comprises multiple operators executed according to a preset sequence, and the multiple operators at least comprise an input operator, a data processing operator and an output operator; receiving execution information which is written by calling the interface of each operator and corresponds to each operator, the execution information corresponding to each operator at least comprises a data structure and an analysis rule of input data corresponding to the input operator, a data processing rule corresponding to the data processing operator and an output data packaging rule corresponding to the output operator; and constructing a data processing task based on the task framework and the execution information corresponding to each operator in the task framework. The workload of manual coding is reduced, and the efficiency and accuracy of task construction are improved.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of computer software technology, and in particular, to a method for constructing a data processing task, a method for executing a data processing task, a method for constructing an advertising traffic anomaly identification task, a method for executing an advertising traffic anomaly identification task, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the rapid development of artificial intelligence and big data technology, the demand for data processing and analysis in all walks of life is growing. For example, with the rapid growth of Internet users, especially mobile Internet users, Internet advertising has become an important form of advertising, and the amount of advertising has also shown a rapid growth trend. However, with the expansion of the scale of advertising, the black industry of advertising has become increasingly rampant, which not only causes advertisers to suffer huge economic losses, but also undermines the reputation and market order of advertising platforms. Therefore, the processing and analysis of advertising traffic data has become an urgent need.

[0003] The current data processing task construction process usually relies on developers to manually write program code for each task. In scenarios where the number of advertising traffic processing tasks is large and highly reusable, this manual coding method is inefficient and difficult to meet the needs. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a data processing task construction method, a data processing task execution method, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a data processing task construction method is proposed, including:

[0007] Acquire a predetermined task framework, wherein the task framework includes a plurality of operators executed in a preset order, and the plurality of operators include at least an input operator, a data processing operator, and an output operator;

[0008] receiving execution information corresponding to each of the operators written by calling the interface of each of the operators, wherein the execution information corresponding to each of the operators at least includes: a data structure and parsing rule of input data corresponding to the input operator, a data processing rule corresponding to the data processing operator, and an output data encapsulation rule corresponding to the output operator;

[0009] A data processing task is constructed based on the task framework and the execution information corresponding to each of the operators in the task framework.

[0010] According to a second aspect of an embodiment of this specification, a method for executing a data processing task is provided, wherein the data processing task is constructed based on the data processing task construction method described in the first aspect, and the data processing task includes: an input operator including a data structure of input data and a parsing rule, a data processing operator including a data processing rule, and an output operator including an output data encapsulation rule;

[0011] The method comprises:

[0012] The input operator receives input data and parses the input data based on the data parsing rule to obtain data to be processed;

[0013] The data processing operator processes the data to be processed based on the data processing rule to obtain a processing result;

[0014] The output operator encapsulates and outputs the processing result based on the output data encapsulation rule.

[0015] According to a third aspect of an embodiment of the present specification, a method for constructing an advertising traffic anomaly identification task is provided, wherein the advertising traffic anomaly identification task is used to identify abnormal traffic in advertising traffic; the advertising traffic is used to describe user behavior data generated during advertising exposure, clicks, or conversions; the abnormal traffic refers to abnormal user behavior data during advertising exposure, clicks, or conversions; the method includes:

[0016] Acquire a predetermined task framework, wherein the task framework includes a plurality of operators executed in a preset order, wherein the plurality of operators include at least an input operator, an anomaly recognition operator, and an output operator;

[0017] Receiving execution information corresponding to each of the operators written by calling the interface of each of the operators, the execution information corresponding to each of the operators at least includes: a data structure and parsing rule of input data corresponding to the input operator, an exception recognition rule corresponding to the exception recognition operator, and an output data encapsulation rule corresponding to the output operator;

[0018] Based on the task framework and the execution information corresponding to each operator in the task framework, an advertising traffic anomaly identification task is constructed.

[0019] According to a fourth aspect of an embodiment of this specification, a method for executing an advertising traffic anomaly identification task is provided, wherein the advertising traffic anomaly identification task is constructed based on the construction method of the advertising traffic anomaly identification task described in the third aspect, and the advertising traffic anomaly identification task includes: an input operator including a data structure of input data and a parsing rule, an anomaly identification operator including an anomaly identification rule, and an output operator including an output data encapsulation rule;

[0020] The method comprises:

[0021] The input operator receives the advertisement traffic data, and parses the advertisement traffic data based on the data parsing rule to obtain data to be processed;

[0022] The anomaly identification operator performs an anomaly identification process on the data to be processed based on the anomaly identification rule to obtain an anomaly identification process result;

[0023] The output operator encapsulates and outputs the abnormality identification processing result based on the output data encapsulation rule.

[0024] According to a fifth aspect of the embodiments of this specification, an electronic device is provided, including:

[0025] processor;

[0026] a memory for storing processor-executable instructions;

[0027] When the processor executes the executable instructions, it is used to implement the method described in the first aspect.

[0028] According to a sixth aspect of the embodiments of this specification, there is provided a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any of the above methods are implemented.

[0029] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program, which implements the steps of any of the above methods when executed by a processor.

[0030] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0031] In the embodiments of the present specification, a pre-set task framework is used, and the task framework includes multiple operators executed in a preset order, and the multiple operators include at least an input operator, a data processing operator, and an output operator; and by calling an operator interface to write execution information corresponding to the operator, such as input data structure, data processing rules, output data encapsulation rules, etc., data processing tasks can be quickly constructed, reducing the workload of manual coding and improving the efficiency and accuracy of task construction.

[0032] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A software architecture diagram is provided by an exemplary embodiment.

[0034] Figure 2 It is a schematic diagram of other basic services connected with data processing tasks provided by an exemplary embodiment.

[0035] Figure 3 It is a flowchart of a method for constructing a data processing task provided by an exemplary embodiment.

[0036] Figure 4 It is a schematic diagram of a task framework provided by an exemplary embodiment.

[0037] Figure 5 is a schematic diagram of frequency calculation provided by an exemplary embodiment.

[0038] Figure 6 is a schematic diagram of repeated marking provided by an exemplary embodiment.

[0039] Figure 7 It is a schematic diagram of the execution order of multiple operators provided by an exemplary embodiment.

[0040] Figure 8 It is a flowchart of a method for executing a data processing task provided by an exemplary embodiment.

[0041] Fig. 9 It is a schematic structural diagram of an electronic device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0042] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0043] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0044] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0045] With the rapid development of artificial intelligence and big data technologies, the demand for data processing and analysis in all walks of life is increasing.

[0046] For example, in the advertising industry, advertisers need to promote products or services by pushing advertisements to users. Advertisers are parties that have the need to place advertisements and pay for the corresponding fees. Advertisers hope that every paid ad click is a valid click by a real user, not a cheating click. Ad clicks refer to users visiting the advertiser's page by clicking on an advertisement on a user-side device (such as a smartphone, tablet, or other terminal device).

[0047] With the rapid growth of Internet users, especially mobile Internet users, Internet advertising has become an important form of advertising, and the amount of advertising has also shown a rapid growth trend. However, with the expansion of the scale of advertising, advertising black industry has become increasingly rampant. For example, in order to obtain more advertising revenue, some users will use advertising fraud to maliciously operate the advertisements placed on the traffic to increase advertising behavior indicators such as exposure, clicks or conversion rate. Therefore, it is necessary to conduct advertising anti-fraud, identify abnormal advertising traffic, and reduce advertising losses. The processing and analysis of advertising traffic has become an urgent need.

[0048] Among them, advertising fraud refers to the behavior of users in the aspects of advertising exposure, clicks, and effects that can increase indicators such as advertising exposure, advertising clicks, advertising click-through rate, and conversion rate for some malicious purposes. This malicious behavior of cheating users is called advertising fraud.

[0049] Ad anti-fraud refers to checking the ad exposure, clicks, and conversions to determine whether ad exposure, clicks, conversions, etc. are triggered by normal user visits or are achieved by cheating users through ad cheating methods.

[0050] However, the current data processing task construction process usually relies on developers to manually write program code for each task. In scenarios where the number of advertising traffic processing tasks is large and highly reusable, this manual coding method is inefficient and difficult to meet the needs.

[0051] Based on this, an embodiment of this specification provides a method for constructing a data processing task. Through a pre-set task framework, the task framework includes multiple operators executed in a preset order, and the execution information corresponding to the operator is written by calling the operator interface. This can quickly construct a data processing task, reduce the workload of manual coding, and improve the efficiency and accuracy of task construction.

[0052] See also Figure 1 , shows a schematic diagram of a software architecture, and uses the software architecture to exemplify the application in the advertising field. Based on the software structure, the task of identifying abnormal traffic in advertising traffic can be implemented. Advertising traffic is used to describe user behavior data generated during advertising exposure, clicks, or conversions. Abnormal traffic refers to abnormal user behavior data during advertising exposure, clicks, or conversions. Such abnormal behavior may include invalid clicks, robot traffic, false displays (or virtual exposures), repeated clicks, malicious clicks, etc.

[0053] The software architecture includes application layer, API layer, core layer, basic service layer, engine layer and storage layer.

[0054] (1) The application layer is responsible for handling various specific needs in the advertising traffic anomaly identification scenario. Customized configuration and development can be performed by combining the API layer interface and the configuration information for data processing tasks. For example, multiple tasks can be built to detect ad click fraud, ad exposure fraud, and ad conversion anomalies. The flexible combination of configuration information and API layer interfaces enables the rapid development of multiple online data processing tasks for different needs to adapt to the ever-changing advertising fraud patterns.

[0055] (2) The API layer provides interfaces corresponding to the various operators in the task framework. These interfaces enable developers to customize functions such as data parsing, data encapsulation, data preprocessing, rule filtering, and policy identification. For example, the API layer can provide interfaces for parsing ad click data, encapsulating anti-cheating results after processing, or calling pre-defined rule filters to perform preliminary screening of ad traffic. Through the API layer, the application layer can quickly call and integrate underlying functions to implement complex processing logic.

[0056] (3) The core layer includes a task framework, which is composed of multiple operators executed in a preset order. The operators include input operators, data processing operators and output operators. Exemplarily, the data processing operator may include at least one of the following: a data filtering operator, a deduplication operator, a feature selection operator, a data preprocessing operator, a first anomaly recognition operator and a second anomaly recognition operator. The data preprocessing operator includes at least one basic component such as a frequency calculation component, a duplicate marking component and a reference data acquisition component. The specific functions of each operator will be described later.

[0057] (4) The basic service layer includes other basic services that interface with data processing tasks. This layer ensures that data processing tasks can be seamlessly connected with other systems and provides stable operation support for the core layer.

[0058] See for example Figure 2 , which shows a schematic diagram of other basic services that interface with data processing tasks. Basic services include but are not limited to offline processing services, report display services, monitoring services, reference databases, and model services.

[0059] The offline processing service can perform data mining on some historical advertising traffic data to obtain some reference data and store it in the reference database for use in the execution of data processing tasks. The offline processing service can also perform supervised training based on some historical advertising traffic data marked with abnormal identification tags to obtain an abnormal identification processing model, and deploy it to the model server, which provides model services for the execution of data processing tasks.

[0060] The monitoring service collects specified monitoring indicators from each operator and can evaluate the stability of data processing tasks by analyzing these monitoring indicators. The report display service can display the execution results of data processing tasks, etc.

[0061] (5) The engine layer is responsible for real-time data processing and analysis. The Flink engine was selected as the data processing engine. Flink has the advantages of high throughput and low latency, and is very suitable for processing large-scale traffic data in the advertising industry. At the engine layer, Flink is responsible for scheduling and executing the anomaly identification tasks built by the core layer to ensure real-time response and processing of abnormal traffic.

[0062] (6) The storage layer is the basis for data storage. It includes various storage systems for storing and managing advertising traffic data and intermediate data generated during the processing, thereby providing efficient and reliable data support.

[0063] Next, the data processing task construction method provided in this specification is described as follows: Figure 3, shows a flow chart of a data processing task construction method, which can be executed by an electronic device, including but not limited to a physical server, a server cluster, a cloud server, a virtual server, a smart phone / mobile phone, a tablet computer, a personal digital assistant (PDA), a laptop computer, and a desktop computer. The electronic device has Figure 1 The software structure shown in FIG. 1 includes:

[0064] In S301, a predetermined task framework is obtained, where the task framework includes a plurality of operators executed in a preset order, and the plurality of operators include at least an input operator, a data processing operator, and an output operator.

[0065] The task framework is an abstract structure designed to perform a series of ordered data processing operations. Figure 4 ,The task framework contains at least the input operator, data processing operator and output operator that are executed sequentially, and each operator is responsible for a specific task.

[0066] For example, the input operator is used to receive and parse input data. The input operator can obtain initial input data from an external data source and perform preliminary parsing on it for subsequent processing.

[0067] The data processing operator is used to perform specific data processing operations on the data received by the input operator, such as data cleaning, conversion, calculation or filtering. Exemplarily, the data processing operator may include at least one of the following: a data filtering operator, a deduplication operator, a feature selection operator, a data preprocessing operator, a first anomaly recognition operator, and a second anomaly recognition operator.

[0068] The output operator is used to encapsulate the data processed by the data processing operator and output it to the specified target location. The output operator ensures that the processing results can be stored or transmitted according to the predetermined format or protocol.

[0069] In S302, execution information corresponding to each operator written by calling the interface of each operator is received, and the execution information corresponding to each operator includes at least: the data structure and parsing rules of the input data corresponding to the input operator, the data processing rules corresponding to the data processing operator, and the output data encapsulation rules corresponding to the output operator.

[0070] In this step, the execution information corresponding to each operator can be written by calling the interface of each operator.

[0071] The data structure and parsing rules of the input data corresponding to the input operator are used to define the structure of the input data (such as field type, data format, etc.) and how to parse the data. This information can ensure that the input data can be correctly interpreted and prepared for subsequent processing.

[0072] The data processing rules corresponding to the data processing operators define how the data processing operators should operate on the data, including, for example, filtering conditions, conversion logic, calculation formulas, etc., which determine the specific transformation method of the data during the processing.

[0073] The output data encapsulation rules corresponding to the output operators define the encapsulation method and output form of the processed data, and the output data encapsulation rules ensure that the results of data processing can be encapsulated in the expected format and output to the target location, such as a database, file system or other external interface. In S303, a data processing task is constructed based on the task framework and the execution information corresponding to each operator in the task framework.

[0074] In this step, the input operator, data processing operator, and output operator can be combined in sequence according to the preset order in the task framework to form an orderly and executable task flow, and the execution information corresponding to each operator is embedded into the operator, so that each operator can perform its task according to the predefined rules and configurations; for example, the input operator will interpret the input data according to the parsing rules, the data processing operator will operate the data according to the processing rules, and the output operator will output the data according to the encapsulation rules. After completing the combination of all operators and the embedding of information, the electronic device generates a complete data processing task. This task can be executed in a real environment, process data in sequence, and output the final result.

[0075] In some embodiments, taking the advertising field as an example, by applying the above data processing task construction method, an identification task for abnormal traffic can be created. The input operator in this task framework will parse the advertising traffic data, the data processing operator will apply the preset abnormal identification rules to detect abnormal behavior, and the final output operator will encapsulate and output the identification results. It can effectively improve the efficiency of advertising traffic data processing, and through automated and modular task construction, it can effectively deal with the problem of advertising black industry, reduce the economic losses of advertisers, and enhance the market competitiveness and reputation of advertising platforms.

[0076] For example, in an advertisement exposure scenario, advertisement traffic data records detailed information about the user's interaction with the advertisement. An advertisement traffic data item related to an advertisement exposure scenario may include any one or more of the following information: client identifier, advertisement identifier, advertisement exposure location, advertisement exposure duration, advertisement exposure time point, exposure page URL (Uniform Resource Locator), and user behavior, but is not limited thereto.

[0077] Among them, (1) the client identifier is used to uniquely identify the client. (2) the ad identifier is used to uniquely identify the ad. (3) the ad exposure position refers to the specific display position of the ad on the page or application, including but not limited to the top of the screen, the sidebar, the bottom banner, etc., which can be further refined to the specific partition of the page. (4) the ad exposure duration refers to the length of time the ad is displayed on the page or application, usually in milliseconds, seconds or minutes, which helps to measure the user's actual attention to the ad. (5) the ad exposure time point records the specific time point when the ad begins to be exposed, usually recorded in the form of a timestamp, including the date and specific time. (6) the exposure page URL is the page address of the page where the ad is located. (7) the user behavior is used to record whether the user has interacted with the ad, such as clicking, scrolling or pausing the ad.

[0078] For example, in an ad click scenario, ad traffic data records detailed information about a user clicking on an ad. An ad traffic data item related to an ad click scenario may include any one or more of the following information: client ID, ad ID, click time, click location, click source page, ad target URL, and click type, but is not limited thereto.

[0079] Among them, (1) the client identifier is used to uniquely identify the client. (2) the ad identifier is used to uniquely identify the ad. (3) the click time is used to record the specific time when the user clicks the ad. (4) the click location is used to record the specific location where the ad is clicked on the page or application, such as a banner ad, sidebar ad, etc. (5) the click source page is used to record the URL of the page where the user was before clicking the ad, which can help understand where the user came from to reach the ad. (6) the ad target URL refers to the target page URL that the user jumps to after clicking the ad. This is usually the page that the advertiser wants the user to visit, such as a product details page or an event page. (7) the click type records the type of user click, whether it is a valid click (such as a one-time click, repeated click, long press, etc.).

[0080] For example, in an advertising conversion scenario, advertising traffic data records information about a user completing a specific conversion behavior after clicking on an advertisement. The conversion behavior is usually an operation that the advertiser wants the user to perform, such as purchase, registration, download, etc. An advertising traffic data item related to an advertising conversion scenario may include any one or more of the following information: client identifier, advertising identifier, conversion time point, conversion type, conversion value, conversion target ID, and conversion path, but is not limited thereto.

[0081] Among them, (1) the client ID is used to uniquely identify the client. (2) the ad ID is used to uniquely identify the ad. (3) the conversion time point records the specific time when the user completes the conversion behavior. (4) the conversion type records the type of conversion behavior performed by the user, such as purchase, registration, download, etc. (5) the conversion value records the value information related to the conversion behavior, such as purchase amount, subscription fee, etc. (6) the conversion target ID is used to distinguish different conversion events. (7) the conversion path records the path information of the user from clicking the ad to completing the conversion, such as the order of visiting multiple pages.

[0082] In some embodiments, the multiple operators in the data processing task include at least an input operator, a data processing operator, and an output operator. Here is an exemplary description of the data processing operator:

[0083] See also Figure 3 , data processing operators and data processing rules corresponding to data processing operators include but are not limited to:

[0084] (1) Data filter operator. The data filter operator has an API interface. By calling the interface of the data filter operator, at least one of the data format to be retained and the data type to be retained corresponding to the data filter operator can be written. In the process of executing the data processing task, the data filter operator can filter out illegal data written by the input operator, including data with mismatched data format or mismatched data type. In this way, the data filter operator can effectively ensure that only data that meets the requirements is processed in the subsequent processing stage, thereby improving the accuracy and efficiency of data processing.

[0085] (2) Deduplication operator: The deduplication operator is used to deduplicate multiple identical advertising traffic data sent repeatedly for the same user behavior. It can be run according to a predefined fixed logic without providing an interface. The deduplication operator can avoid analysis result deviation and resource waste caused by duplicate data.

[0086] For example, during the execution of a data processing task, due to network problems, system failures, etc., for example, during network transmission, advertising traffic data may be lost, causing the relevant device to resend the same advertising traffic data to ensure integrity; or due to network delays or instability, advertising traffic data may be sent multiple times to confirm receipt; or application crashes or restarts may cause repeated submission of advertising traffic data; or advertising applications or browsers on user devices may repeatedly send the same advertising traffic data in a short period of time, etc., resulting in multiple identical advertising traffic data being sent repeatedly for the same user behavior. This embodiment can identify and process these repeated transmission situations in the data processing task through the deduplication operator, thereby ensuring the accuracy of the advertising data and the efficiency of the system.

[0087] After receiving a piece of advertising traffic data, the deduplication operator can perform a hash operation on the advertising traffic data to obtain a hash value, and group it according to the hash value to determine the group to which the advertising traffic data belongs. It should be noted that if multiple identical advertising traffic data are sent repeatedly for the same user behavior, the hash values ​​and groups corresponding to the multiple identical advertising traffic data are the same. After the group to which the advertising traffic data belongs, it can be further determined whether the storage location corresponding to the group to which the advertising traffic data belongs has data stored. If data has been stored, the advertising traffic data is discarded; if data is not stored, the hash value of the advertising traffic data can be written to the storage location.

[0088] The judgment period of the deduplication operator may be a preset time period, such as 1 hour or 30 minutes, etc., that is, at the end of each judgment period, the hash value written in the judgment period is cleared to restart the deduplication judgment in the next judgment period. For example, for the storage location corresponding to each group, a timer can be started after the hash value is written for the first time, and the written hash value can be cleared when the timer returns to zero. Or other methods can also be used.

[0089] (3) Feature selection operator. The feature selection operator has an API interface. The feature selection rule corresponding to the feature selection operator can be written by calling the interface of the feature selection operator. The feature selection rule is used to specify at least one dimension of features for subsequent data processing from the advertising traffic data. The feature selection rule can select features of one or more dimensions in the advertising traffic data for subsequent preprocessing. For example, in the advertising exposure scenario, features of dimensions such as client ID, advertising ID, and advertising exposure position are selected.

[0090] (4) Data preprocessing operator. The data preprocessing operator has an API interface. The preprocessing rules corresponding to the data preprocessing operator can be written by calling the interface of the data preprocessing operator. The preprocessing rules corresponding to the data preprocessing operator are used to define the statistical period for the advertising traffic and / or define the type of reference data to be introduced from the reference database, so that the data preprocessing operator calls the predetermined basic components according to the preprocessing rules during the task running process to obtain the preprocessing results.

[0091] For example, see Figure 1 , the basic components include at least one of the following: a frequency calculation component, a repeat marking component, and a reference data acquisition component.

[0092] The frequency calculation component is used to count the frequency of advertising traffic data received from the same user within the statistical period. Figure 5As shown in the figure, the time axis starts from 00:00, and the numbers above indicate the number of occurrences in this time block. The red vertical line indicates the frequency of occurrence within 10 minutes at the time corresponding to the time axis. In order to save storage space, for example, a storage granularity of 5 minutes can be used, such as a number in the time range of [00:00, 00:05) to indicate the number of occurrences during this period, and the result of the frequency statistics is the sum of the numbers in the time block within the statistical period. Among them, the result of the frequency statistics can be stored as an intermediate result in a specified storage location.

[0093] The duplicate marking component is used to identify whether the advertising traffic data received within the statistical period is duplicate data from the same user, and mark the currently identified advertising traffic data according to the identification result. For example, see Figure 6 , within the statistical period (assuming 1 minute), the first piece of advertising traffic data from the same user is marked as non-duplicate data, and other advertising traffic data except the first piece of advertising traffic data is marked as duplicate data.

[0094] Exemplarily, the feature selection rules applied to the feature selection operator are used to specify the features of at least one dimension in the advertising traffic data as the statistical basis for the frequency calculation component and the duplicate marking component. For example, in an advertising exposure scenario, a piece of advertising traffic data related to the advertising exposure scenario includes information such as client identification, advertising identification, advertising exposure position, advertising exposure duration, advertising exposure time point, exposure page URL, and user behavior. The feature selection rule can specify "client identification" as the statistical basis for the frequency calculation component and the duplicate marking component in the preprocessing operator, which means that the subsequent preprocessing steps will focus on these two features for more effective data statistics and duplicate data identification. For example, the frequency calculation component can count the frequency of advertising traffic data received from the same user in each statistical period based on the client identification in the advertising traffic data; the duplicate marking component can identify whether the advertising traffic data received in the statistical period is duplicate data from the same user based on the client identification in the advertising traffic data. The collaborative work of feature selection operators and data preprocessing operators ensures that advertising traffic data is fully cleaned and prepared before analysis, thereby improving the accuracy and efficiency of data analysis. This structured processing method enables data processing tasks to flexibly adapt to different data processing requirements and effectively handle complex data traffic.

[0095] The reference data acquisition component is used to obtain reference data related to advertising traffic data from the reference database according to the pre-defined reference data type. Figure 2 ,The data stored in the reference database can be obtained by an offline ,processing service based on data mining of several historical ,advertising traffic data.

[0096] Exemplarily, the reference data may be a sequence of historical advertising traffic data of the user corresponding to the piece of advertising traffic data, or the reference data may be a relationship graph consisting of historical advertising traffic data of other users related to the user corresponding to the piece of advertising traffic data. The use of the user's historical advertising traffic data sequence and the relationship graph between the user and related users can provide deep insights into advertising data analysis, identify user behavior patterns and discover potential abnormal activities. The use of these reference data not only improves the accuracy of analysis, but also enhances the understanding of user behavior.

[0097] For example, the preprocessing rule defines that the type of reference data to be introduced from the reference database is a historical advertising traffic data sequence. Assuming that the device is analyzing the advertising click behavior of a certain user (user A), and the advertising traffic data includes the click record and timestamp of user A, the preprocessing module can obtain the historical advertising traffic data sequence of user A from the reference database according to the preprocessing rule.

[0098] For example, the currently received advertising traffic data includes: {User ID: User A; Advertising ID: Advertising 123; Click time: 2024-08-20, 10:15:00; Advertising position: top of page}. User A's historical advertising traffic data sequence includes: {(1) User ID: User A; Advertising ID: Advertising 123; Click time: 2024-08-19, 08:30:00; Advertising position: bottom of page. (2) User ID: User A; Advertising ID: Advertising 456, Advertising 789; Click time: 2024-08-19, 10:00:00; Advertising position: side of page. (3) User ID: User A; Advertising ID: Advertising 789; Click time: 2024-08-19, 16:45:00; Advertising position: top of page}. But not limited to this. By analyzing user A's historical click data, we can determine user A's click frequency and ad click position preference over the past period of time, and help identify whether there are abnormal click patterns, such as excessively high click frequency or unusual ad position preference.

[0099] As another example, the preprocessing rules define that the type of reference data that needs to be introduced from the reference database is the historical advertising traffic data relationship diagram between the current user to be analyzed and other related users. Assuming that the device is analyzing the advertising click behavior of a certain user (user A), and the advertising traffic data includes the click record and timestamp of user A, the preprocessing module can obtain the advertising traffic data of other users related to user A (user B, user C, etc.) from the reference database according to the preprocessing rules. The advertising traffic data of user A, user B, and user C can be mapped on a graph to show the distribution of information such as advertising ID and click time, as well as the similarity of click behavior between users. By comparing the click behavior of user A and other users on the same advertisement, identify whether there is an abnormal pattern. For example, if the click time of user A is significantly different from that of other users, it may indicate abnormal behavior.

[0100] (5) A first exception recognition operator. The first exception recognition operator has an API interface. The first exception recognition rule corresponding to the first exception recognition operator can be written by calling the interface of the first exception recognition operator.

[0101] (6) A second anomaly recognition operator. The second anomaly recognition operator has an API interface. The second anomaly recognition rule corresponding to the second anomaly recognition operator can be written by calling the interface of the second anomaly recognition operator.

[0102] The complexity of the second anomaly identification rule is higher than the complexity of the first anomaly identification rule.

[0103] The first anomaly identification rule is used for fast and basic anomaly detection, and usually defines simple matching or filtering rules to identify abnormal behavior. For example, the first anomaly identification rule includes at least one of a threshold judgment rule, a blacklist filtering rule, and a regular expression match; for example, the threshold judgment rule defines a threshold rule for the frequency of ad clicks. If the same user clicks on the same ad more than 10 times within 1 minute, it is identified as abnormal behavior. The blacklist filtering rule defines a preset blacklist of IP addresses, device IDs, or user IDs. If the relevant fields in the ad traffic data match the records in the blacklist, it is directly marked as an anomaly. Regular expression matching implements pattern matching using regular expressions or simple logical expressions, such as checking whether a URL contains suspicious parameters or character combinations.

[0104] The second anomaly identification rule is used for more complex and sophisticated anomaly detection, and is usually based on a machine learning model or a deep learning model for identification. For example, the second anomaly identification rule includes a model identifier of a trained anomaly identification processing model, and the second anomaly identification rule indicates that anomaly identification processing is performed based on the prediction result of the anomaly identification processing model.

[0105] Among them, see Figure 2The anomaly identification processing model is obtained by supervised training of several historical advertising traffic data marked with anomaly identification labels by offline processing services. The trained anomaly identification processing model can be deployed in a preset model server, so that the model server can provide model services.

[0106] The first anomaly identification rule is usually applied to basic, rule-driven detection scenarios, suitable for quickly filtering data with obvious anomalies. The rules are simple and efficient, and suitable for preliminary screening of large-scale data. The second anomaly identification rule is applicable to more complex scenarios, requiring in-depth analysis and comprehensive judgment. It relies on complex models or algorithms, can identify more hidden abnormal behaviors, and is suitable for further mining deep-level anomalies in data after preliminary screening.

[0107] Those skilled in the art will appreciate that one or more of the data filtering operator, the deduplication operator, the feature selection operator, the data preprocessing operator, the first anomaly recognition operator, and the second anomaly recognition operator may be selected for use according to actual needs, and this embodiment does not impose any restrictions on this.

[0108] Exemplarily, when the data processing operator includes a data filtering operator, a deduplication operator, a feature selection operator, a data preprocessing operator, a first anomaly recognition operator, and a second anomaly recognition operator, the multiple operators in the data processing task can be as follows: Figure 7 The execution is performed in the order shown, that is, input operator → data filtering operator → deduplication operator → feature selection operator → data preprocessing operator → first anomaly recognition operator → second anomaly recognition operator → output operator. This embodiment adopts a highly general and modular framework design concept, combined with a flexible interface development strategy, to ensure that the task framework can easily adapt to the needs of various scenarios, and can quickly respond and expand functions when faced with new or changed requirements, and by reducing repeated development work, significantly reduce the time and cost investment in subsequent upgrades, expansions and maintenance processes.

[0109] During the execution of the data processing task, if the first anomaly identification operator identifies that the advertising traffic data is abnormal traffic based on the first anomaly identification rule, the anomaly identification result can be directly transmitted to the output operator without the need to perform the next step of processing the traffic, which is beneficial to improving data processing efficiency; if the first anomaly identification operator does not identify that the advertising traffic data is abnormal traffic based on the first anomaly identification rule, the advertising traffic data can be transmitted to the second anomaly identification operator for further identification and processing.

[0110] Those skilled in the art will appreciate that at least one of the above-mentioned operators may be executed at least once in a data processing task, and the present embodiment does not impose any restrictions on this. For example, if the first anomaly identification operator does not identify the advertising traffic data as abnormal traffic based on the first anomaly identification rule, the data preprocessing operator may preprocess the advertising traffic data again based on the preprocessing rule, and then transmit the preprocessing result to the second anomaly identification operator for further anomaly identification.

[0111] In some embodiments, see Figure 1 as well as Figure 7 The multiple operators in the data processing task also include at least one of a debugging operator and a reporting operator, and the debugging operator and the reporting operator are executed after the abnormality identification data processing operator.

[0112] The debugging operator is used to output the first specified data generated during the execution of the data processing task to the offline processing service; wherein the offline processing service is used to perform data mining based on the first specified data, generate reference data for updating the reference database, and / or, to optimize the training of the abnormal identification processing model based on the first specified data. Specifically, the debugging operator is mainly used to output key intermediate data or result data to the offline processing service during the execution of the data processing task for further analysis, mining and optimization. For example, in the advertising click scenario, the debugging operator can output the feature vectors of certain click behaviors and their identification as normal or abnormal. These data are transmitted to the offline processing service for further analysis of the click behavior pattern and adjustment of the model or rules; if a large number of abnormal behaviors are detected within a certain period of time, the debugging operator can output detailed log information of these behaviors for the offline processing service to conduct in-depth mining and discover new abnormal patterns.

[0113] The report operator is used to output the second specified data that can be used for report display to the report display service. The second specified data can be processed summary data, statistical results, or specific anomaly detection results. These data will be used to generate visual reports to display the overall performance of advertising traffic, abnormal behavior statistics, user behavior analysis, etc. The reports generated by the report display service based on the data output by the report operator can help enterprises better understand the advertising effect, abnormal behavior distribution and other key indicators, and support further strategic adjustments and operational optimization.

[0114] For example, in an ad exposure scenario, a reporting operator can output aggregated ad exposure frequency, click-through rate, and abnormal behavior rate data, which are transmitted to the report display service to generate daily or weekly ad performance reports. Alternatively, in an ad click and conversion scenario, a reporting operator can output click-through rates and identified abnormal click behavior statistics for different ad campaigns to generate key performance indicator reports for management to view.

[0115] In other words, the main function of the debugging operator is to provide detailed intermediate data for offline optimization and model improvement, helping developers to continuously optimize the anomaly recognition model and update the reference database to improve the processing capabilities of subsequent tasks. The main function of the reporting operator is to convert the processing results into visual data reports and provide them to developers to help them make accurate judgments on the effectiveness of advertising and abnormal behaviors, thereby supporting more scientific decision-making.

[0116] For example, see Figure 2 During the execution of multiple operators in the data processing task, they can also output the specified monitoring indicators of the execution process to the monitoring service; the specified monitoring indicators include but are not limited to processing delay, operator throughput, input and output data volume, triggering frequency of abnormal events, resource utilization (such as CPU and memory usage), error rate, data loss rate, etc. Different operators can output different monitoring indicators according to their functions and processing order.

[0117] The monitoring service collects specified monitoring indicators from each operator. By analyzing these monitoring indicators, it can evaluate the stability of data processing tasks and detect whether there are potential performance problems or abnormal situations. When the monitoring service detects that certain indicators exceed the preset threshold (such as too long processing delay, too high error rate, etc.), it can immediately trigger the abnormal alarm mechanism to notify the system administrator or automatically execute fault handling measures, thereby improving the overall processing efficiency and data processing accuracy.

[0118] Exemplarily, the electronic device may embed the execution information corresponding to each operator in the task framework into each operator in the form of a plug-in to form a data processing task.

[0119] The electronic device embeds a plug-in for the input operator, which contains the data structure definition and parsing rules of the input data. By embedding the plug-in into the input operator, the input operator can accurately parse the initial input data based on the information provided by the plug-in. The plug-in defines the field name, data type, and how to convert the original data format into the standard format in the task framework.

[0120] The electronic device embeds the corresponding plug-in for the data processing operator. This plug-in contains specific data processing rules, such as filtering conditions, data conversion logic, calculation formulas, etc. After embedding this plug-in, the data processing operator can perform corresponding processing operations according to the rules, process the data parsed by the input operator, and generate intermediate results or directly output results.

[0121] The electronic device embeds an encapsulation rule plug-in for the output operator. This plug-in defines how to encapsulate the result data generated by the data processing operator into a predetermined format and output it to a specified storage location or transmit it to the next processing link. By embedding the plug-in into the output operator, it is ensured that the output data meets the requirements of the task framework and the specifications of the external interface.

[0122] In some embodiments, after creating a data processing task, the electronic device can obtain configuration information for the data processing task, the configuration information is used to indicate the resource configuration of the data processing task, and then deploy the data processing task to a preset data processing engine based on the configuration information of the data processing task. In this embodiment, the electronic device automatically obtains the configuration information of the data processing task, and deploys the data processing task to the preset data processing engine based on the configuration information, which reduces the workload of manual configuration and deployment. This automated process improves the development and deployment efficiency of data processing tasks, especially in scenarios that require rapid response and processing of large-scale data, and can significantly shorten the time it takes to launch a task.

[0123] Exemplarily, the data processing engine includes a flink engine. The flink engine is an open source stream processing framework that focuses on real-time data processing, can process large-scale data streams, supports low-latency, high-throughput stream processing tasks, and provides fault tolerance and state management.

[0124] Exemplarily, the configuration information includes at least one of the following: input data configuration information, output data configuration information, storage configuration information, and computing resource configuration information. Flexible configuration of configuration information allows developers to flexibly adjust the use of input, output, storage, and computing resources according to different needs.

[0125] The input data configuration information is used to indicate at least one of a storage location and an input method of the input data, and the input method includes but is not limited to concurrent reading or batch reading.

[0126] The output data configuration information is used to indicate at least one of a storage location and an output mode of the output data. The output mode includes but is not limited to concurrent writing, multi-channel output or compressed output.

[0127] Storage configuration information is used to indicate the storage location of intermediate data generated during the execution of data processing tasks; for example, whether to use the Flink engine's memory or other external storage media. Intermediate data refers to temporary data generated during the execution of a task. This data may be part of the final result data or may be data used for subsequent processing steps.

[0128] The computing resource configuration information is used to indicate the usage of computing resources of each operator in the abnormal identification data processing task. Computing resources include but are not limited to CPU resources, memory resources, network resources, and storage I / O resources.

[0129] Those skilled in the art will appreciate that the data processing task construction method provided in this specification can be applied to other scenarios in addition to advertising exposure, click or conversion scenarios, and this embodiment does not impose any limitations on this.

[0130] For example, in network security scenarios, the data processing task construction method can be used to construct tasks to monitor network traffic and detect potential intrusion behaviors. The input operator receives network traffic data, filters invalid or abnormal traffic, the deduplication operator removes duplicate packet transmission requests, the feature selection operator extracts traffic features such as IP address, port number, protocol type, etc., the preprocessing operator analyzes the traffic pattern, and the anomaly recognition operator detects abnormal traffic, such as DDoS attacks, unauthorized access, etc., and finally the output operator outputs the final detection results.

[0131] For example, in financial transaction scenarios, the data processing task construction method can be used to construct tasks for detecting abnormal transaction behaviors. For example, in stock trading and payment systems, transaction data can be analyzed in real time, data with abnormal formats or contents can be filtered out, duplicate transaction requests can be removed, key features such as transaction amount, time, location, etc. can be extracted, and transaction frequency and patterns can be counted through preprocessing operators. Subsequently, anomaly identification operators are used to detect potential fraudulent behaviors, such as a large number of transactions in a short period of time or abnormal transaction paths.

[0132] In some embodiments, the embodiments of this specification further provide a method for constructing an advertising traffic anomaly identification task, where the advertising traffic anomaly identification task is used to identify abnormal traffic in advertising traffic; advertising traffic is used to describe user behavior data generated during advertising exposure, clicks, or conversions; abnormal traffic refers to abnormal user behavior data during advertising exposure, clicks, or conversions; the method includes:

[0133] Acquire a predetermined task framework, the task framework including a plurality of operators executed in a preset order, the plurality of operators including at least an input operator, an anomaly recognition operator, and an output operator;

[0134] Receive execution information corresponding to each operator written by calling the interface of each operator, where the execution information corresponding to each operator at least includes: a data structure and parsing rule of input data corresponding to the input operator, an exception recognition rule corresponding to the exception recognition operator, and an output data encapsulation rule corresponding to the output operator;

[0135] Based on the task framework and the execution information corresponding to each operator in the task framework, an advertising traffic anomaly identification task is constructed.

[0136] In one implementation, the anomaly recognition operator includes at least one of the following: a data filtering operator, a feature selection operator, a data preprocessing operator, a first anomaly recognition operator, and a second anomaly recognition operator.

[0137] The anomaly identification rules corresponding to the anomaly identification operator include at least one of the following: (1) a data format to be retained and / or a data type to be retained corresponding to the data filtering operator; (2) a feature selection rule corresponding to the feature selection operator, which is used to specify at least one dimension of features for subsequent data processing from the advertising traffic data; (3) a preprocessing rule corresponding to the data preprocessing operator, which is used to define the statistical period for the advertising traffic and / or the type of reference data related to the advertising traffic data that needs to be introduced from the reference database, wherein the data stored in the reference database is obtained by the offline processing service based on data mining of a number of historical advertising traffic data; (4) a first anomaly identification rule corresponding to the first identification operator; and (5) a second anomaly identification rule corresponding to the second identification operator, wherein the complexity of the second anomaly identification rule is higher than the complexity of the first anomaly identification rule. The second anomaly identification rule includes a model identifier of a trained anomaly identification processing model, and the second anomaly identification rule indicates that anomaly identification processing is performed based on the prediction result of the anomaly identification processing model, and the anomaly identification processing model is obtained by the offline processing service through supervised training based on a number of historical advertising traffic data labeled with anomaly identification labels.

[0138] In one implementation, the data preprocessing operator includes at least one basic component, which includes at least one of the following: (1) a frequency calculation component, which is used to count the frequency of advertising traffic data received from the same user within a statistical period; (2) a duplicate marking component, which is used to identify whether the advertising traffic data received within the statistical period is duplicate data from the same user, and mark the currently identified advertising traffic data according to the identification result; and (3) a reference data acquisition component, which is used to obtain reference data related to the advertising traffic data from a reference database according to a predefined reference data type.

[0139] In one implementation, the anomaly identification operator also includes a deduplication operator; the deduplication operator is executed before the data preprocessing operator and is used to deduplicate multiple identical advertising traffic data repeatedly sent for the same user behavior.

[0140] In one implementation, the multiple operators also include at least one of the following: (1) a debugging operator, which is executed after the data processing operator and is used to output first specified data generated during the execution of the data processing task to an offline processing service, wherein the offline processing service is used to generate reference data for updating a reference database and / or optimizing an anomaly identification processing model based on the first specified data; (2) a reporting operator, which is executed after the data processing operator and is used to output second specified data that can be used for report presentation to a report presentation service.

[0141] In some embodiments, see Figure 8 The embodiment of this specification also provides a method for executing a data processing task. The data processing task is constructed based on the above-mentioned method for constructing a data processing task. The data processing task includes: an input operator including a data structure of input data and parsing rules, a data processing operator including data processing rules, and an output operator including output data encapsulation rules. The method for executing a data processing task includes:

[0142] In S801, the input operator receives input data and parses the input data based on data parsing rules to obtain data to be processed.

[0143] In this step, the input operator is the entry point of the data processing task and is used to receive input data. The input data can come from various data sources, such as databases, message queues, log files, etc. The data parsing rules are used to guide the input operator on how to decode and interpret the input data. For example, if the input data is a log file in JSON format, the parsing rules will define how to parse the JSON data into structured key-value pairs for subsequent processing.

[0144] For example, in the ad anti-fraud scenario, the input operator receives ad exposure log data. The parsing rules will guide the input operator to parse these log data into various fields, such as ad ID, exposure time, user ID, etc. Only the data that is parsed successfully will enter the next step of processing.

[0145] In S802, the data processing operator processes the data to be processed based on the data processing rule to obtain a processing result.

[0146] In this step, the data processing operator is the core part of the data processing task and is responsible for executing specific data processing logic. The processing logic can be flexibly defined according to requirements.

[0147] The data processing operator may include multiple operators, such as data filtering operators, feature selection operators, deduplication operators, anomaly identification operators, etc. These operators are executed in a predetermined order.

[0148] In S803, the output operator encapsulates and outputs the processing result based on the output data encapsulation rule.

[0149] In this step, the output operator is responsible for converting the processing results into a data format that meets the requirements of the target system or storage medium, and outputting it. The output can be stored in a database, written to a message queue, sent to an external system, etc. The output data encapsulation rules define how to organize and format the processing results. For example, the rules may require that the processing results be encapsulated in a specific JSON structure, or that the data be compressed before storage.

[0150] In some embodiments, see Figure 7 , the data processing operators include data filtering operators, deduplication operators, feature selection operators, data preprocessing operators, first anomaly recognition operators, and second anomaly recognition operators. Among them, the data processing task is to identify abnormal traffic in advertising traffic. Take the identification of false exposure events in advertising exposure scenarios as an example for illustrative explanation:

[0151] First, the input operator receives the ad exposure data. Assume that the received data includes information such as ad ID, client ID, exposure location, exposure duration, and exposure time. The input operator parses this data according to predefined parsing rules and generates data to be processed in a unified format for use by subsequent operators.

[0152] Next, the data filter operator will check the parsed data and filter out data that does not match the format or type. For example, if a piece of ad exposure data is missing a key field (such as an ad ID) or the data format does not meet expectations (such as an incorrect time format), the data filter operator will filter out this data to avoid affecting subsequent processing.

[0153] After data filtering, the deduplication operator will check whether the ad exposure data is repeated. For example, if there are multiple identical ad traffic data sent for the same user behavior, the deduplication operator will check whether these data are repeated through hash operations. If there are duplicate data, the deduplication operator will discard them and only keep one piece of data for subsequent processing.

[0154] After deduplication, the ad exposure data will enter the feature selection operator. This operator will extract features of specific dimensions from the data according to the feature selection rules. For example, in the ad exposure scenario, the client ID and ad ID can be selected as features for subsequent frequency calculation.

[0155] After feature selection is completed, the data preprocessing operator will preprocess the selected features. For example, the number of exposures within a certain time period can be counted based on the client identifier in the ad exposure data, or reference data, such as the user's historical ad exposure behavior, can be introduced for further analysis and processing.

[0156] Next, the first anomaly identification operator detects anomalies in the data based on simple matching rules. For example, if a user's ad exposure increases abnormally in a short period of time, this situation may be detected as a potential abnormal behavior by a simple rule.

[0157] If the first anomaly identification operator does not identify abnormal behavior, the second anomaly identification operator will apply a more complex model for further anomaly detection. The second anomaly identification operator may use a trained anomaly identification processing model to identify more hidden abnormal patterns by analyzing the user's historical exposure behavior and other related data.

[0158] Finally, after all the processing steps, the output operator will encapsulate and output the processing results. Through the sequential execution of these operators, the advertising exposure data can be effectively processed and analyzed, invalid or abnormal data can be filtered out, and the accuracy of advertising delivery and the effectiveness of anti-cheating can be improved.

[0159] Similarly, the process of identifying false click events in advertising click scenarios and false conversion events in advertising conversion scenarios is similar to the above process. The execution information related to the operator can be written based on the processing logic in the actual scenario to construct data processing tasks for different scenarios.

[0160] In some embodiments, the embodiments of this specification also provide a method for executing an advertising traffic anomaly identification task, the advertising traffic anomaly identification task is constructed based on the above-mentioned advertising traffic anomaly identification task construction method, and the advertising traffic anomaly identification task includes: an input operator including a data structure of input data and a parsing rule, an anomaly identification operator including an anomaly identification rule, and an output operator including an output data encapsulation rule;

[0161] The method includes:

[0162] The input operator receives the advertising traffic data and parses the advertising traffic data based on the data parsing rules to obtain the data to be processed;

[0163] The anomaly identification operator performs anomaly identification processing on the data to be processed based on the anomaly identification rule to obtain an anomaly identification processing result;

[0164] The output operator encapsulates and outputs the abnormality identification processing results based on the output data encapsulation rules.

[0165] The various technical features in the above embodiments can be arbitrarily combined as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, any combination of the various technical features in the above embodiments also falls within the scope of this specification.

[0166] Based on the same concept as the above method, this specification also provides an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of the method described in any of the above embodiments by running the executable instructions. Fig. 9 is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Fig. 9 At the hardware level, the device includes a processor 902, an internal bus 904, a network interface 906, a memory 909, and a non-volatile memory 910, and may also include hardware required for other functions. One or more embodiments of this specification may be implemented based on software, such as the processor 902 reading the corresponding computer program from the non-volatile memory 910 into the memory 909 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0167] Exemplarily, the data processing task construction device can be applied to Fig. 9 In the electronic device shown in the figure, the technical solution of this specification is implemented. Among them, the data processing task construction device may include:

[0168] A task framework acquisition module, used to acquire a predetermined task framework, the task framework includes a plurality of operators executed in a preset order, the plurality of operators at least including an input operator, a data processing operator and an output operator;

[0169] An execution information receiving module, used to receive execution information corresponding to each operator written by calling the interface of each operator, and the execution information corresponding to each operator at least includes: a data structure and parsing rule of input data corresponding to the input operator, a data processing rule corresponding to the data processing operator, and an output data encapsulation rule corresponding to the output operator;

[0170] The task construction module is used to construct a data processing task based on the task framework and the execution information corresponding to each operator in the task framework.

[0171] Exemplarily, the data processing task execution device can be applied to Fig. 9In the electronic device shown in the figure, the technical solution of this specification is implemented. Among them, the data processing task execution device may include:

[0172] An input module is used to receive input data from an input operator and parse the input data based on data parsing rules to obtain data to be processed;

[0173] A processing module is used for processing the data to be processed by the data processing operator based on the data processing rules to obtain a processing result;

[0174] The output module is used for the output operator to encapsulate and output the processing results based on the output data encapsulation rules.

[0175] Exemplarily, the construction device of the advertising traffic anomaly identification task can be applied to Fig. 9 In the electronic device shown, the technical solution of this specification is implemented. Among them, the construction device of the advertising traffic anomaly identification task may include:

[0176] The task framework acquisition module is used to acquire a predetermined task framework, where the task framework includes multiple operators executed in a preset order, and the multiple operators include at least an input operator, an abnormality recognition operator, and an output operator.

[0177] An execution information receiving module is used to receive execution information corresponding to each operator written by calling the interface of each operator. The execution information corresponding to each operator includes at least: the data structure and parsing rules of the input data corresponding to the input operator, the exception recognition rules corresponding to the exception recognition operator, and the output data encapsulation rules corresponding to the output operator.

[0178] The task construction module is used to construct an advertising traffic anomaly identification task based on the task framework and the execution information corresponding to each operator in the task framework.

[0179] For example, the advertising traffic anomaly identification task execution device can be applied to Fig. 9 In the electronic device shown in the figure, the technical solution of this specification is implemented. Among them, the advertising traffic anomaly identification task execution device may include:

[0180] An input module, which is used to receive advertisement traffic data through an input operator, and parse the advertisement traffic data based on data parsing rules to obtain data to be processed;

[0181] A processing module, configured to perform anomaly identification processing on the data to be processed by an anomaly identification operator based on anomaly identification rules to obtain an anomaly identification processing result;

[0182] The output module is used to encapsulate and output the abnormality identification processing results based on the output data encapsulation rules by the output operator.

[0183] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0184] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0185] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0186] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

[0187] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A method for constructing a data processing task, comprising: Acquire a predetermined task framework, wherein the task framework includes a plurality of operators executed in a preset order, and the plurality of operators include at least an input operator, a data processing operator, and an output operator; receiving execution information corresponding to each of the operators written by calling the interface of each of the operators, wherein the execution information corresponding to each of the operators at least includes: a data structure and parsing rule of input data corresponding to the input operator, a data processing rule corresponding to the data processing operator, and an output data encapsulation rule corresponding to the output operator; A data processing task is constructed based on the task framework and the execution information corresponding to each of the operators in the task framework.

2. According to the method of claim 1, the data processing task is a task for identifying abnormal traffic in advertising traffic; The advertising traffic is used to describe the user behavior data generated during the advertising exposure, click or conversion process; The abnormal traffic refers to abnormal user behavior data during the process of advertisement exposure, click or conversion.

3. The method according to claim 2, wherein the data processing operator comprises at least one of the following: a data filtering operator, a feature selection operator, a data preprocessing operator, a first anomaly recognition operator, and a second anomaly recognition operator; The data processing rules corresponding to the data processing operator include at least one of the following: The data format to be retained and / or the data type to be retained corresponding to the data filtering operator; A feature selection rule corresponding to the feature selection operator, used to specify features of at least one dimension for subsequent data processing from the advertising traffic data; A preprocessing rule corresponding to the data preprocessing operator, used to define a statistical period for advertising traffic and / or a type of reference data related to advertising traffic data that needs to be introduced from a reference database, wherein the data stored in the reference database is obtained by an offline processing service based on data mining of a number of historical advertising traffic data; a first anomaly identification rule corresponding to the first identification operator; and a second anomaly identification rule corresponding to the second identification operator, the complexity of the second anomaly identification rule being higher than the complexity of the first anomaly identification rule; Among them, the second anomaly identification rule includes the model identifier of the trained anomaly identification processing model, and the second anomaly identification rule indicates to perform anomaly identification processing based on the prediction result of the anomaly identification processing model. The anomaly identification processing model is obtained by supervised training by the offline processing service based on a number of historical advertising traffic data marked with anomaly identification labels.

4. According to the method of claim 3, the data preprocessing operator includes at least one basic component, and the basic component includes at least one of the following: A frequency calculation component, used to count the frequency of advertisement traffic data received from the same user within the statistical period; A repeat marking component, used to identify whether the advertisement traffic data received within the statistical period is repeated data from the same user, and mark the currently identified advertisement traffic data according to the identification result; as well as The reference data acquisition component is used to acquire the reference data related to the advertisement traffic data from the reference database according to the predefined type of the reference data.

5. A method for executing a data processing task, wherein the data processing task is constructed based on the data processing task construction method according to any one of claims 1 to 4, and the data processing task comprises: An input operator including the data structure and parsing rules of input data, a data processing operator including data processing rules, and an output operator including output data encapsulation rules; The method comprises: The input operator receives input data and parses the input data based on the data parsing rule to obtain data to be processed; The data processing operator processes the data to be processed based on the data processing rule to obtain a processing result; The output operator encapsulates and outputs the processing result based on the output data encapsulation rule.

6. A method for constructing an advertising traffic anomaly identification task, wherein the advertising traffic anomaly identification task is used to identify abnormal traffic in advertising traffic; the advertising traffic is used to describe user behavior data generated during advertising exposure, clicks or conversions; The abnormal traffic refers to abnormal user behavior data during the process of advertisement exposure, click or conversion; the method includes: Acquire a predetermined task framework, wherein the task framework includes a plurality of operators executed in a preset order, wherein the plurality of operators include at least an input operator, an anomaly recognition operator, and an output operator; Receiving execution information corresponding to each of the operators written by calling the interface of each of the operators, the execution information corresponding to each of the operators at least includes: a data structure and parsing rule of input data corresponding to the input operator, an exception recognition rule corresponding to the exception recognition operator, and an output data encapsulation rule corresponding to the output operator; Based on the task framework and the execution information corresponding to each operator in the task framework, an advertising traffic anomaly identification task is constructed.

7. A method for executing an advertising traffic anomaly identification task, wherein the advertising traffic anomaly identification task is constructed based on the construction method of the advertising traffic anomaly identification task according to claim 6, and the advertising traffic anomaly identification task comprises: An input operator including the data structure and parsing rules of input data, an anomaly identification operator including anomaly identification rules, and an output operator including output data encapsulation rules; The method comprises: The input operator receives the advertisement traffic data, and parses the advertisement traffic data based on the data parsing rule to obtain data to be processed; The anomaly identification operator performs an anomaly identification process on the data to be processed based on the anomaly identification rule to obtain an anomaly identification process result; The output operator encapsulates and outputs the abnormality identification processing result based on the output data encapsulation rule.

8. An electronic device comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 7 by executing the executable instructions.

9. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images