Data processing method, device, equipment, medium and computer program product
By collecting metric data in the streaming link of the advertising system for evaluation and correction processing, the correctness and consistency problems caused by out-of-order data are solved, and the accurate and reliable streaming calculation of the data is achieved.
Patent Information
- Application Number
- CN202411284518.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-12-06
AI Technical Summary
In commercial advertising systems, due to data back-pass delay, long data links and complex data strategies, there is a problem of out-of-order in the data system, making it difficult to verify the correctness and consistency of the data, and thus it is difficult to determine the root of abnormal data. Due to repair.
By collecting metric data in the streaming link of the advertising system for data evaluation, identifying abnormal phenomena, and performing correction processing, the target data flow is obtained to ensure the integrity and correctness of the data, thereby realizing streaming calculation.
It effectively eliminates the impact of data exceptions on streaming calculations, ensures the accuracy and consistency of data, and improves the accuracy and reliability of data processing.
Smart Images

Figure CN119090560A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, in particular to the field of big data and information flow technology, and specifically to data processing methods, devices, equipment, media and computer program products. Background Art
[0002] In commercial advertising systems, data streams play the role of linking users, advertising systems, and advertisers. Log data generated by the interaction between platform users and advertisements enters the data stream through end-point logging, advertiser feedback, and other methods, and is then uniformly calculated and standardized by the streaming system and distributed to different businesses for application. Summary of the invention
[0003] The present disclosure provides a data processing method, apparatus, device, medium and computer program product.
[0004] According to one aspect of the present disclosure, there is provided a data processing method, comprising: in response to a streaming computing request, performing data evaluation based on indicator data collected from a streaming link of an advertising system to obtain a data evaluation result; when the data evaluation result indicates that an abnormality exists in a data stream of the advertising system, performing correction processing on the data stream to obtain a target data stream; and performing streaming computing based on the target data stream.
[0005] According to another aspect of the present disclosure, there is provided a data processing device, comprising:
[0006] A data evaluation module, for responding to a streaming computing request, performing data evaluation based on indicator data collected from a streaming link of an advertising system, and obtaining a data evaluation result;
[0007] A data correction module, used to correct the data stream to obtain a target data stream when the data evaluation result indicates that there is an abnormality in the data stream of the advertising system, and
[0008] The streaming computing module is used to perform streaming computing based on the target data stream.
[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.
[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method as described above.
[0011] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method described above when executed by a processor.
[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0014] Figure 1 An exemplary system architecture to which the data processing method and device according to an embodiment of the present disclosure can be applied is schematically shown;
[0015] Figure 2 A flowchart schematically shows a data processing method according to an embodiment of the present disclosure;
[0016] Figure 3 A curve diagram schematically shows the inlet data and the outlet data of data aligned based on the event execution time according to an embodiment of the present disclosure;
[0017] Figure 4A A schematic diagram schematically shows a job data flow according to an embodiment of the present disclosure;
[0018] Figure 4B A schematic diagram schematically shows a job data flow according to another embodiment of the present disclosure;
[0019] Figure 5 A flowchart schematically shows a data processing method according to another embodiment of the present disclosure;
[0020] Figure 6 A block diagram schematically shows a data processing device according to an embodiment of the present disclosure; and
[0021] Figure 7 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION
[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0023] In the process of implementing the present disclosure, the inventors found that there are at least the following problems in the prior art. In actual business scenarios, due to the data's own return delay, excessively long data links, streaming splicing windows and complex data strategies, the data is often partially disordered in the system, making it difficult for the system to verify the correctness and consistency of the data. Therefore, in the event of data anomalies, it is difficult to determine whether to repair the abnormal data based on the root cause of the data anomaly.
[0024] Figure 1 An exemplary system architecture to which the data processing method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.
[0025] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, the exemplary system architecture to which the content processing method and apparatus can be applied may include a terminal device, but the terminal device may implement the content processing method and apparatus provided by the embodiments of the present disclosure without interacting with the server.
[0026] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.
[0027] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).
[0028] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.
[0029] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for the content browsed by the user using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device.
[0030] It should be noted that the content processing method provided in the embodiment of the present disclosure can generally be executed by the terminal device 101, 102, or 103. Accordingly, the content processing apparatus provided in the embodiment of the present disclosure can also be set in the terminal device 101, 102, or 103.
[0031] Alternatively, the content processing method provided in the embodiment of the present disclosure may also be generally executed by the server 105. Accordingly, the content processing device provided in the embodiment of the present disclosure may generally be arranged in the server 105. The content processing method provided in the embodiment of the present disclosure may also be executed by a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the content processing device provided in the embodiment of the present disclosure may also be arranged in a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.
[0032] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0033] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.
[0034] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.
[0035] Figure 2 The flowchart of the content processing method according to the embodiment of the present disclosure is schematically shown.
[0036] like Figure 2 As shown, the method includes operations S210 to S230.
[0037] In operation S210, in response to a streaming computing request, data evaluation is performed based on indicator data collected from a streaming link of an advertising system to obtain a data evaluation result.
[0038] According to an embodiment of the present disclosure, an advertising system may include multiple advertisements and indicator data corresponding to the multiple advertisements, and the indicator data includes data for characterizing effects of advertisement conversion data flows of the multiple advertisements.
[0039] According to an embodiment of the present disclosure, data can be transmitted in a streaming link in the form of a continuous data stream, so the streaming link does not need to wait for a large amount of data to be generated and then packaged and sent, but can send the data in real time after it is generated.
[0040] According to an embodiment of the present disclosure, the indicator data collected from the streaming link of the advertising system can be streaming data, and the streaming computing request can be used to perform streaming computing on the indicator data collected from the streaming link of the advertising system to obtain streaming computing results, wherein streaming computing is a computing model for real-time processing of continuous data streams, without the need to wait until the continuous data stream arrives completely before starting processing.
[0041] According to an embodiment of the present disclosure, data evaluation is performed based on the indicator data collected from the streaming link of the advertising system to obtain a data evaluation result, wherein the data evaluation may include evaluating the integrity of the indicator data, and may also include evaluating the correctness of the indicator data. The data evaluation result can be used to characterize whether there are abnormal phenomena in the indicator data, and the abnormal phenomena may include missing data, abnormal data values, etc.
[0042] In operation S220, when the data evaluation result indicates that there is an abnormality in the data stream of the advertisement system, correction processing is performed on the data stream to obtain a target data stream.
[0043] According to an embodiment of the present disclosure, based on the data evaluation result, it is determined whether there are abnormal phenomena in the data flow of the advertising system. When it is determined that there are abnormal phenomena in the data flow, correction processing is performed on the database to obtain a target data flow, wherein the correction processing includes processing the integrity of the data and processing the correctness of the data, and the data of the target data flow does not have abnormal phenomena such as missing data and data anomalies.
[0044] According to an embodiment of the present disclosure, when the data evaluation result indicates that there is no abnormality in the data flow of the advertisement system, the data flow is determined as the target data flow.
[0045] In operation S230 , stream computing is performed based on the target data stream.
[0046] According to an embodiment of the present disclosure, the target data stream is calculated in real time based on the received target data stream.
[0047] According to an embodiment of the present disclosure, in response to a streaming computing request, the indicator data collected in the streaming link of the advertising system is evaluated, and the data evaluation result is determined. By collecting data in the streaming link of the advertising system, errors caused by data delays can be reduced. In the case where there are abnormalities in the data stream of the advertising system, the data stream is corrected to obtain a target data stream to eliminate the impact of the abnormality on the data stream, so that the data of the target data stream is accurate, and consistency and correctness problems of the data are avoided, thereby ensuring that the streaming computing based on the target data stream is more accurate and reliable.
[0048] According to an embodiment of the present disclosure, indicator data includes a first category of indicator data and a second category of indicator data, the first category of indicator data is data aligned based on the event triggering time, and the second category of indicator data is data aligned based on the event execution time; in response to a streaming computing request, data evaluation is performed based on the indicator data collected from the streaming link of the advertising system to obtain data evaluation results, including: obtaining an integrity evaluation result based on the first category of indicator data; obtaining a correctness evaluation result based on the second category of indicator data; and obtaining a data evaluation result based on the integrity evaluation result and the correctness evaluation result.
[0049] According to an embodiment of the present disclosure, indicator data includes a first category of indicator data and a second category of indicator data, wherein the first category of indicator data is data obtained by aligning multiple events according to the time interval to which the event triggering moment belongs, and the second category of indicator data is data obtained by aligning multiple times according to the time interval described by the event execution moment, the event triggering moment is the moment when the event occurs and the data corresponding to the event is generated, and the event execution moment is the moment when the data corresponding to the event starts to be processed, wherein the data can be processed locally on the electronic device, or the data can be sent to a server for processing.
[0050] According to the embodiments of the present disclosure, the server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.
[0051] According to an embodiment of the present disclosure, since the first category of indicator data is aligned based on the event triggering moment, and the event triggering moment is the moment when the event occurs and the data is generated, the first category of indicator data can more accurately reflect the original state and order of the data, so that the use of the first category of indicator data can ensure that the indicator data is completely collected and processed within a specific time window, and therefore, an integrity assessment result can be obtained based on the first category of indicator data, wherein the integrity assessment result is used to characterize whether there is any missing data in the indicator data.
[0052] According to the embodiments of the present disclosure, since the time when data is processed in the server is the system time of the server, the time is certain and is not affected by the timestamp of the external data source. In addition, in a distributed system, the processing time can be used to synchronize data processing on different nodes and different data sources. Since the processing time is the system time of the server, the processing time can ensure that data processing is performed in the same order on different nodes and different data sources, thereby ensuring the correctness of the data.
[0053] According to the embodiments of the present disclosure, since the second category of indicator data is aligned based on the event execution time, and the event execution time is the time when the data is sent to the server for processing, the second category of indicator data can more accurately reflect the system status when the streaming job processes the data stream, and thereby reflect whether the streaming job on the link has stability problems. Therefore, based on the second category of indicator data, a correctness evaluation result can be obtained, wherein the correctness evaluation result is used to characterize whether there are data errors in the indicator data.
[0054] According to an embodiment of the present disclosure, a data evaluation result may be determined based on an integrity evaluation result for characterizing whether the indicator data has data missing conditions and a correctness evaluation result for characterizing whether the indicator data has data errors.
[0055] According to an embodiment of the present disclosure, based on the first category of indicator data and the second category of indicator data in the indicator data, the integrity assessment results and the correctness assessment results are respectively determined, and the data evaluation results are further determined. This can fully utilize the characteristics of the time bases of the first category of indicator data and the second category of indicator data, respectively, to accurately determine the integrity assessment results and the correctness assessment results, thereby improving the accuracy of the data evaluation results and further improving the accuracy of data processing.
[0056] According to an embodiment of the present disclosure, an integrity assessment result is obtained based on the first category of indicator data, including: based on the integrity indicator, first target indicator data is screened from the first category of indicator data; first entry data related to the entry end of the advertising system and first exit data related to the exit end of the advertising system are extracted from the first target indicator data; based on the maximum delay time window, second entry data is intercepted from the first entry data, and second exit data is intercepted from the first exit data, and the maximum delay time window is determined based on the triggering time of the streaming computing request and the maximum delay duration of the advertising system; and the integrity assessment result is obtained based on the comparison result of the second entry data and the second exit data.
[0057] According to an embodiment of the present disclosure, based on the integrity index, first target index data for characterizing data integrity is screened from the first category of index data. Based on the inlet and outlet of the advertising system, first inlet data related to the inlet and first outlet data related to the outlet are extracted from the first target index data, respectively, wherein the inlet and outlet of the advertising system correspond to the inlet and outlet of the index data in and out of the link, respectively.
[0058] According to an embodiment of the present disclosure, based on the maximum delay time window, a time window before the maximum delay time window is selected from the first inlet data, and the data of the time window is intercepted to obtain the second inlet data. A time window before the maximum delay time window is selected from the first outlet data, and the data of the time window is intercepted to obtain the second outlet data.
[0059] According to an embodiment of the present disclosure, the maximum delay time window is determined based on the triggering moment of the streaming computing request and the maximum delay duration of the advertising system. The delay duration can be determined based on the transmission delay of the link data, the processing delay of the link data, and the delay time of the splicing window. When the transmission delay of the link data, the processing delay of the link data, and the delay time of the splicing window take their maximum values, the maximum delay duration can be determined. At the triggering moment of the streaming computing request, the maximum delay duration is determined, thereby determining the maximum delay time window. For example, when the transmission delay of the link data is x, the processing delay of the link data is y, the delay time of the splicing window is z, and x, y, and z all take their maximum values, x+y+z can be determined as the maximum delay time window.
[0060] According to an embodiment of the present disclosure, the second entry data and the second exit data are compared to obtain a comparison result, and an integrity assessment result is obtained based on the comparison result of the second entry data and the second exit data, wherein, when the comparison result characterizes that the second entry data and the second exit data are consistent in terms of data volume and other feature dimensions, it can be determined that the integrity assessment result is that there is no data missing in the indicator data; when the comparison result characterizes that the second entry data and the second exit data are inconsistent in terms of data volume and other feature dimensions, it can be determined that the integrity assessment result is that there is data missing in the indicator data.
[0061] According to an embodiment of the present disclosure, based on the integrity index, the first target index data is screened from the first category of index data, and the first entry data and the first exit data related to the entry and exit of the advertising system are extracted therefrom, and after the second entry data and the second exit data are further determined, the integrity assessment result is determined. Since the second entry data and the second exit data are determined based on the maximum delay time window, it can be determined that there is no integrity problem with the data at the maximum delay time window, and it is only necessary to determine the integrity problem of the data at the previous time window. By comparing the data volume and other feature dimensions of the second entry data and the second exit data, the integrity assessment result of the data can be determined more comprehensively and accurately.
[0062] Due to the out-of-order data in the data stream and the delays caused by link transmission and processing, the integrity of the data in the overall streaming system cannot be determined using data aligned based on the event execution time.
[0063] Figure 3 A curve chart of input data and output data of data aligned based on the event execution time according to an embodiment of the present disclosure is schematically shown.
[0064] like Figure 3 As shown, curve 301 is the input data of the data aligned based on the event execution time, curve 302 is the output data of the data aligned based on the event execution time, and area 303 is a time window, wherein curve 301 is a solid line and curve 302 is a dotted line. The horizontal axis of the coordinate system is time, and the vertical axis is the value of feature A.
[0065] Based on the part of curve 301 and curve 302 in region 303, it can be found that at the starting position of region 303, the values of feature A of curve 301 and curve 302 are similar, and at the end position of region 303, the value of feature A of curve 302 is significantly higher than the value of feature A of curve 301. This may be because when data is processed in the streaming link, some data is processed quickly, some data is processed slowly, and multiple steps are required, resulting in adjacent data in the inlet data being located in different time windows in the outlet data, but no data loss is found. Therefore, it is easy to make mistakes when using data aligned based on the event execution time to judge the integrity of the data of the streaming system.
[0066] According to an embodiment of the present disclosure, a correctness evaluation result is obtained based on the second category of indicator data, including: based on the correctness indicator, second target indicator data is screened from the second category of indicator data to obtain second target indicator data, the streaming link includes multiple jobs, and the second target indicator data includes sub-indicator data related to each of the multiple jobs; the sub-indicator data related to each of the multiple jobs are combined to obtain combined data; and the combined data is input into a data model to obtain a correctness evaluation result.
[0067] According to an embodiment of the present disclosure, a streaming link includes multiple jobs, such as a splicing job, an anti-cheating job, a data labeling job, a data deduplication job, a data distribution job, and other streaming jobs, wherein the splicing job is used to merge data streams from different data sources or different time points into a unified data stream for subsequent analysis and processing; the anti-cheating job is used to identify and filter out illegal or abnormal data to ensure the quality and accuracy of the data; the data labeling job is used to add additional information or metadata to the data for subsequent query, analysis, and processing; the data deduplication job is used to remove duplicate records in the data stream to reduce data redundancy and improve processing efficiency; and the data distribution job is used to distribute the processed data to different targets, such as different databases, message queues, or services.
[0068] According to an embodiment of the present disclosure, sub-indicator data related to each of the multiple jobs in the streaming link is screened from the second type of indicator data to determine the second target indicator data. The multiple sub-indicator data included in the second target indicator data are combined to obtain combined data.
[0069] According to an embodiment of the present disclosure, the combined data is input into a data model, the combined data is analyzed using the data model, and the combined data is compared with sample combined data corresponding to the combined data to determine whether the combined data represents that there is a correctness problem with the indicator data, wherein the sample combined data is data that is of the same type as multiple jobs corresponding to the combined data and does not have any correctness problem. When there is no excessive data difference between the combined data and the sample combined data, it can be determined that there is no correctness problem with the indicator data.
[0070] According to an embodiment of the present disclosure, the second type of indicator data is given, sub-indicator data related to each of the multiple jobs in the streaming link is determined, and the multiple sub-indicator data are combined, and the correctness evaluation result is determined using a data model. This allows the correctness evaluation result to take into account the data conditions of each of the multiple jobs at the same time, thereby improving the comprehensiveness and correctness of data processing.
[0071] According to an embodiment of the present disclosure, a streaming link includes multiple jobs, and two adjacent jobs among the multiple jobs interact through a message middleware; the data processing method also includes: collecting data on the job data streams recorded in each of the multiple message middleware included in the streaming link to obtain first-category indicator data and second-category indicator data, respectively.
[0072] According to an embodiment of the present disclosure, in a plurality of jobs of a streaming link, two adjacent jobs interact with each other through a message middleware. After an interaction is completed between the two jobs, the job data stream corresponding to the interaction can be recorded in the message middleware, wherein the message middleware includes a message queue or a message delivery system. Through the message middleware, asynchronous, reliable and scalable data exchange can be performed between the two jobs. In addition, interaction through the message middleware can also achieve decoupling between message producers and consumers, that is, two adjacent jobs can send and receive messages without having to know each other's work details.
[0073] According to an embodiment of the present disclosure, a streaming link includes multiple jobs, and data is exchanged between every two adjacent jobs through a message middleware. Therefore, the streaming link includes multiple message middlewares, and data collection is performed on the job data streams recorded by each of the multiple message middlewares in the streaming link, and the data included in the job data stream is divided into first-category indicator data and second-category indicator data according to the event trigger time and the time execution time.
[0074] According to an embodiment of the present disclosure, the interaction between multiple jobs in the streaming link is recorded and collected through the message middleware, thereby ensuring that the first type of indicator data and the second type of indicator data are collected correctly.
[0075] According to an embodiment of the present disclosure, data collection is performed on the job data streams recorded in each of the multiple message middlewares included in the streaming link to obtain first-category indicator data and second-category indicator data, respectively, including: taking the event triggering moment as the time reference, data collection is performed on the job data streams recorded in each of the multiple message middlewares to obtain first-category indicator data; and taking the event execution moment as the time reference, data collection is performed on the job data streams recorded in each of the multiple message middlewares to obtain second-category indicator data.
[0076] According to an embodiment of the present disclosure, the job data streams recorded in each of the plurality of message middlewares are classified according to the alignment benchmark of the indicator data, and indicator data based on the event triggering time and indicator data based on the time execution time are obtained respectively.
[0077] According to an embodiment of the present disclosure, since the event triggering time of an event is an attribute of the data itself, indicator data based on the event triggering time can be collected from the job data streams recorded in multiple message middlewares to obtain first-category indicator data.
[0078] According to the embodiments of the present disclosure, since the event triggering time of the event is generated when the data starts to be processed, it is not an attribute of the data itself, that is, the data does not have this attribute when it is in the message middleware. Therefore, with the time execution time as the time reference, data is collected for each of the job data streams recorded in the multiple message middlewares to obtain the second type of indicator data.
[0079] According to an embodiment of the present disclosure, data collection is performed on the job data streams recorded in each of multiple message middlewares based on the event triggering time and the event execution time, so as to accurately determine the first category indicator data and the second category indicator data based on different time bases.
[0080] Figure 4A The diagram schematically shows a job data flow according to an embodiment of the present disclosure.
[0081] like Figure 4A As shown, the job data stream includes 5 job data, namely data1~data5, where the event triggering time of data1 is 1 and the event execution time is 1, the event triggering time of data2 is 1 and the event execution time is 2, the event triggering time of data3 is 1 and the event execution time is 3, the event triggering time of data4 is 2 and the event execution time is 4, and the event triggering time of data5 is 2 and the event execution time is 5. All data are normal state data. Since data1~data5 constitute the job data stream in sequence, the integrity of the job data stream can be determined according to the event triggering time.
[0082] Figure 4B The figure schematically shows a job data flow according to another embodiment of the present disclosure.
[0083] like Figure 4B As shown in the figure, the job data stream includes 5 job data, namely data1~data5, among which the event triggering time of data1 is 1 and the event execution time is 1, the event triggering time of data2 is 1 and the event execution time is 6, the event triggering time of data3 is 2 and the event execution time is 5, the event triggering time of data4 is 2 and the event execution time is 2, and the event triggering time of data5 is 2 and the event execution time is 4. All data are normal state data. Since there is disorder between data1~data5 in the job data stream arriving at the event execution time, and the data with the event execution time 3 is missing between data4 and data5, the integrity of the job data stream cannot be accurately judged based on the job data stream arriving at the event execution time. However, since the job data stream arriving at the event execution time can reflect the system state when the streaming job processes the data stream, it is possible to determine whether the job data in the job data stream is abnormal based on the job data stream arriving at the event execution time.
[0084] According to an embodiment of the present disclosure, taking the event triggering moment as a time reference, data collection is performed on the job data streams recorded in each of the multiple message middlewares to obtain the first category of indicator data, including: determining multiple job data located in the batch processing time window from the job data stream; aligning the event triggering moment information of the multiple job data based on the event triggering moment information represented by the batch processing time window to obtain multiple aligned job data; writing the multiple aligned job data into an offline data warehouse; and rolling up the aligned job data written in the offline data warehouse within a first preset time period to obtain the first category of indicator data.
[0085] According to an embodiment of the present disclosure, multiple job data located in a batch processing time window are determined from a job data stream. Based on the event triggering time information represented by the batch processing time window, multiple job data are aligned according to their respective event triggering times to obtain multiple aligned job data, and the aligned job data are written into an offline data warehouse, wherein the processing of the job data can be determined according to multiple event triggering times and a preset alignment period, for example, multiple event triggering times include multiple moments within 14:00~14:15, and the preset alignment period is 5 minutes, then the job data with event triggering times between 14:00~14:05 can be aligned to 14:00, the job data with event triggering times between 14:05~14:10 can be aligned to 14:05, and the job data with event triggering times between 14:10~14:15 can be aligned to 14:10, and the offline data warehouse is a data warehouse that can be used to process historical data.
[0086] According to an embodiment of the present disclosure, the aligned job data written in the offline data warehouse within a first preset time period is rolled up to obtain a first type of indicator data, wherein the first preset time period can be determined according to the degree of roll-up, and the first preset time period is greater than the preset alignment time period. For example, the first preset time period can be 1 hour, 6 hours, 1 day, etc. Data roll-up can be used to aggregate stream data from the bottom level to a higher level. The first type of indicator data includes job data within multiple preset alignment time periods, and the job data has been aligned according to the preset alignment time period.
[0087] According to an embodiment of the present disclosure, multiple job data are aligned using event triggering moment information represented in the job data stream, and the aligned job data are further rolled up through an offline data warehouse to determine first-category indicator data. Since the aligned data is event triggering moment information, it is possible to ensure that the time reference of the obtained first-category indicator data is the event triggering moment.
[0088] According to an embodiment of the present disclosure, taking the event execution time as the time reference, data collection is performed on the job data streams recorded in each of the multiple message middlewares to obtain the second category of indicator data, including: writing the job data included in the job data stream into the real-time data warehouse; and rolling up the job data written in the real-time data warehouse within a second preset time period to obtain the second category of indicator data.
[0089] According to an embodiment of the present disclosure, the job data included in the job data stream is written into a real-time data warehouse, wherein the real-time data warehouse is a data warehouse for processing and analyzing data in real time or near real time.
[0090] According to an embodiment of the present disclosure, the job data written in the real-time data warehouse within the second preset time period is rolled up to obtain the second type of indicator data, wherein the second preset time period is a shorter time period to achieve near real-time data processing. Preferably, the second preset time period can be 30 seconds, 1 minute, etc.
[0091] According to an embodiment of the present disclosure, a real-time data warehouse is used to roll up the job data in the job data stream to obtain the second type of indicator data, which can realize real-time or near real-time processing of the job data, thereby ensuring that the time base of the second type of indicator data is the event execution time.
[0092] According to an embodiment of the present disclosure, when a data evaluation result indicates that an abnormal phenomenon exists in the data stream of an advertising system, correction processing is performed on the data stream to obtain a target data stream, including: when a data evaluation result indicates that an abnormal phenomenon exists in the data stream of an advertising system, abnormal data is recalled from the advertising system; an inference result is obtained based on data attributes of the abnormal data and data attributes of normal data associated with the abnormal data in the data stream; and based on the inference result, the target data stream is obtained using the data stream and the abnormal data.
[0093] According to an embodiment of the present disclosure, when the data evaluation result indicates that there is an abnormality in the data flow of the advertising system, based on the data evaluation result, data corresponding to the data evaluation result is recalled from the advertising system to obtain abnormal data.
[0094] According to an embodiment of the present disclosure, normal data associated with the abnormal data is determined based on the category of the job corresponding to the abnormal data, wherein the category of the job corresponding to the normal data is the same as the category of the job corresponding to the abnormal data, and a data evaluation result of the normal data characterizes that there is no abnormal phenomenon in the data flow.
[0095] According to an embodiment of the present disclosure, data attributes of abnormal data and data attributes of normal data associated with the abnormal data in a data stream are acquired, wherein the data attributes include attributes related to the category of the data and the job. For example, when the category of the job is a splicing job, the job requires that there is an adjacent relationship in timestamps between multiple data to be spliced, and therefore the data attributes include the event execution times corresponding to the multiple data to be spliced.
[0096] According to an embodiment of the present disclosure, when there are errors in the data itself in the data flow or errors in job execution, the data evaluation result will indicate that there are abnormalities in the data flow of the advertising system, wherein, when the cause of the abnormality in the data flow causing the abnormal data is errors in the data itself, the data attributes of the abnormal data are different from the data attributes of normal data associated with the abnormal data, and when the cause of the abnormality in the data flow causing the abnormal data is errors in job execution, the data attributes of the abnormal data are the same as the data attributes of normal data associated with the abnormal data.
[0097] According to the embodiments of the present disclosure, based on the data attributes of the abnormal data and the data attributes of the normal data associated with the abnormal data in the data stream, a comparison result of the two can be determined, and further an inference result can be obtained.
[0098] According to an embodiment of the present disclosure, when the data evaluation result indicates that there is an abnormal phenomenon in the data flow of the advertising system, the abnormal data that causes the abnormal phenomenon is recalled from the advertising system, and based on the data attributes of the abnormal data and the normal data, the reasoning result is determined, and the target data flow is finally determined according to the reasoning result. By determining the abnormal data according to the abnormal phenomenon, and using the data attributes of the normal data associated with the abnormal data, it is possible to determine whether the data itself has an abnormality, and further determine the target data flow, so that the data processing is more comprehensive and accurate.
[0099] According to an embodiment of the present disclosure, when the inference result cannot be determined based on the data attributes of the abnormal data and the data attributes of the normal data associated with the abnormal data in the data stream, a clustering method can be used for multiple abnormal data to determine the abnormal connections between the multiple abnormal data, and based on the abnormal connections, the inference results of the multiple abnormal data can be determined.
[0100] For example, when the inference result cannot be determined based on the data attributes of the abnormal data and the data attributes of the normal data associated with the abnormal data in the data stream, for multiple abnormal data with the above problems, the clustering method can be used to determine that the multiple abnormal data are data returned by the same user. Further analysis determines that the event triggering times of the multiple abnormal data are all too early. Therefore, based on the abnormal connection, it can be determined that the inference result of the multiple abnormal data is that the reason for the abnormal phenomenon in the data stream of the abnormal data is that there are errors in the data itself.
[0101] According to an embodiment of the present disclosure, when a data evaluation result indicates that an abnormal phenomenon exists in the data flow of an advertising system, abnormal data is recalled from the advertising system, including: when a data evaluation result indicates that an abnormal phenomenon exists in the data flow of an advertising system, based on a target indicator indicated by the abnormal phenomenon, a reverse search is performed on the streaming link to obtain flow information of the target indicator; and based on the flow information of the target indicator, abnormal data is recalled from the advertising system.
[0102] According to an embodiment of the present disclosure, when the data evaluation result indicates that an abnormal phenomenon exists in the data flow of the advertising system, a target indicator indicated by the abnormal phenomenon is determined, wherein the target indicator includes an indicator corresponding to the abnormal phenomenon.
[0103] According to an embodiment of the present disclosure, based on the target indicator and the upstream and downstream topology of the streaming link, a reverse search is performed on the streaming link to obtain the flow information of the target indicator, wherein a reverse search can be performed job by job according to the upstream and downstream topology, and the flow information of the target indicator can be determined based on the situations of the target indicator in multiple jobs searched during the reverse search process.
[0104] According to an embodiment of the present disclosure, based on the flow information of the target indicator, the job corresponding to the first abnormality of the target indicator in the advertising system is determined, so that the data at the job is determined as abnormal data, and the abnormal data is recalled from the advertising system.
[0105] According to the embodiments of the present disclosure, when the data evaluation results indicate that there are abnormal phenomena in the data flow, a reverse search is performed in the streaming link through the target indicator indicated by the abnormal phenomenon to determine the flow information of the target indicator, so that the operation corresponding to the first abnormality of the data flow in the streaming link can be accurately determined, and the abnormality can be traced. Based on the flow information of the target indicator, the abnormal data is recalled from the advertising system, and the abnormal data can be accurately recalled according to the traceability results for subsequent analysis, thereby improving the accuracy of data processing.
[0106] According to an embodiment of the present disclosure, the inference result includes valid data and invalid data; based on the inference result, the target data stream is obtained by using the data stream and the abnormal data, including: when the inference result represents that the data is valid, the abnormal data is spliced into the data stream to obtain the target data stream; when the inference result represents that the data is invalid, the target data stream is obtained based on the data stream.
[0107] According to an embodiment of the present disclosure, when the data attributes of the abnormal data are different from the data attributes of the normal data associated with the abnormal data, the reason for the abnormal phenomenon in the data flow that causes the abnormal data is that the data itself has errors, so it can be determined that the data is invalid. When the data attributes of the abnormal data are the same as the data attributes of the normal data associated with the abnormal data, the reason for the abnormal phenomenon in the data flow that causes the abnormal data is that there are errors in job execution, so it can be determined that the data is valid.
[0108] According to an embodiment of the present disclosure, when the inference result represents that the data is valid, the job is re-executed until the data corresponding to the abnormal data no longer has an abnormality, and the data obtained by re-executing the job is spliced into the data stream to obtain the target data stream.
[0109] According to an embodiment of the present disclosure, when the inference result indicates that the data is invalid, the data may be discarded, thereby obtaining a target data stream based on the data stream.
[0110] According to the embodiments of the present disclosure, when the inference result represents invalid data, the abnormal data is discarded and the target data stream is determined, which can avoid the invalid erroneous data from affecting the data stream. When the inference result represents valid data, the abnormal data is reprocessed and spliced into the data stream to obtain the target data stream, which can minimize the discarded data while ensuring the correctness of the data stream, thereby improving data integrity.
[0111] Figure 5 The flowchart schematically shows a data processing method according to another embodiment of the present disclosure.
[0112] like Figure 5 As shown, the process of the data processing method of this embodiment includes a process of converting the data of the advertising system into a job data stream and a process of performing stream computing based on the job data stream.
[0113] In the process of converting the data of the advertising system into the job data stream, the interaction between two adjacent jobs is stored in the message middleware through the streaming data entry, and based on the interaction between the two jobs, the streaming job is performed. For example, the job includes data splicing, data anti-cheating, data labeling, data deduplication, data distribution, etc. Each time a streaming job is executed, the streaming job is stored in the message middleware.
[0114] After each update of the message middleware, multiple job data are read through the message middleware, and after processing the multiple job data according to their respective time trigger moments, the multiple aligned job data are stored in the offline data warehouse through the offline data warehouse manager. After the streaming job is completed, the job data is read through the message middleware after the last update, and the data is stored in the real-time data warehouse through the real-time data warehouse manager.
[0115] Based on the data of offline tables in the offline data warehouse and the data of online tables in the real-time data warehouse, feature storage is performed, and rollup and modeling are performed based on the feature storage to obtain rollup tables and data models respectively. In the feature storage process, since the main application scenarios of data processing are frequent batch writing, frequent queries, and few modifications, and query reading often only requires reading a small number of columns, online analysis and processing capabilities are required in long-term data analysis, and TTL (Time-To-Live) data expiration policies need to be supported. Therefore, the column-based database management system ClickHouse can be selected as the storage engine for feature storage, thereby avoiding the problem of high access and maintenance costs caused by inconsistent syntax when using time series databases. In addition, column storage can improve data query efficiency under large data volumes.
[0116] The roll-up table includes the first and second indicator data after roll-up. Based on the roll-up table, real-time multi-dimensional indicator analysis can be performed and a difference report can be generated. The real-time multi-dimensional indicators may include job traffic, latency, splicing green, cheating rate, deduplication rate, conversion rate, etc. Based on the difference report, the difference report of each job at the first preset time period level of the feature dimension can be determined.
[0117] According to the data model, trend mining can be performed on the advertising system to determine long-term product trends. Further, variable-cycle data modeling can be performed from a product perspective. For example, in the effective conversion indicator of the advertising conversion data flow, the alignment time of multiple job data in the batch processing time window is 5 minutes. Based on the characteristics of multiple job data in the batch processing time window, the changing trend of effective conversion volume indicators with days, weeks, months, and quarters as cycles under different feature combinations can be established. For feature combinations with obvious or continuous increases, we can pay attention to and add relevant conversion strategies to optimize the conversion effect in a targeted manner.
[0118] In addition, micro-investigation can be performed based on feature storage to determine abnormal data and inference results of abnormal phenomena, so as to obtain the target data flow based on the inference results, data flow and abnormal data.
[0119] Provide real-time multi-dimensional indicators, difference reports, micro-inspections and trend mining to users or advertisers so that they can determine optimization plans for advertising based on the above indicators.
[0120] Figure 6 The block diagram schematically shows a data processing device according to an embodiment of the present disclosure.
[0121] like Figure 6 As shown, the data processing device 600 of this embodiment includes a data evaluation module 610 , a data correction module 620 and a streaming computing module 630 .
[0122] The data evaluation module 610 is used to respond to the streaming computing request and perform data evaluation based on the indicator data collected from the streaming link of the advertising system to obtain a data evaluation result.
[0123] The data correction module 620 is used to correct the data stream to obtain a target data stream when the data evaluation result indicates that there is an abnormality in the data stream of the advertising system.
[0124] The stream computing module 630 is used to perform stream computing based on the target data stream.
[0125] According to an embodiment of the present disclosure, the data evaluation module 610 includes a first evaluation submodule, a second evaluation submodule and a data evaluation submodule.
[0126] The first evaluation submodule is used to obtain an integrity evaluation result based on the first type of indicator data.
[0127] The second evaluation submodule is used to obtain a correctness evaluation result based on the second type of indicator data.
[0128] The data evaluation submodule is used to obtain data evaluation results based on the integrity evaluation results and the correctness evaluation results.
[0129] According to an embodiment of the present disclosure, the first evaluation submodule includes a first data screening unit, a data extraction unit, a data interception unit and a data comparison unit.
[0130] The first data screening unit is used to screen the first target indicator data from the first category indicator data based on the integrity indicator.
[0131] The data extraction unit is used to extract first entry data related to the entry end of the advertisement system and first exit data related to the exit end of the advertisement system from the first target indicator data.
[0132] The data interception unit is used to intercept the second input data from the first input data and intercept the second output data from the first output data based on the maximum delay time window, and the maximum delay time window is determined based on the triggering time of the streaming computing request and the maximum delay duration of the advertising system.
[0133] The data comparison unit is used to obtain an integrity assessment result based on a comparison result between the second input data and the second output data.
[0134] According to an embodiment of the present disclosure, the second evaluation submodule includes a second data screening unit, a data combining unit and a second evaluation unit.
[0135] The second data screening unit is used to screen the second target indicator data from the second type of indicator data based on the correctness indicator, the streaming link includes multiple jobs, and the second target indicator data includes sub-indicator data related to each of the multiple jobs.
[0136] The data combining unit is used to combine the sub-indicator data related to each of the multiple operations to obtain combined data.
[0137] The second evaluation unit is used to input the combined data into the data model to obtain a correctness evaluation result.
[0138] According to an embodiment of the present disclosure, the data processing device further includes a data acquisition module.
[0139] The data collection module is used to collect data from the job data streams recorded in the multiple message middlewares included in the streaming link to obtain the first type of indicator data and the second type of indicator data respectively.
[0140] According to an embodiment of the present disclosure, the data acquisition module includes a first acquisition submodule and a second acquisition submodule.
[0141] The first collection submodule is used to collect data from the operation data streams recorded in the plurality of message middlewares respectively based on the event triggering time as the time reference, so as to obtain the first type of indicator data.
[0142] The second collection submodule is used to collect data from the job data streams recorded in the multiple message middlewares respectively based on the event execution time as the time reference, so as to obtain the second type of indicator data.
[0143] According to an embodiment of the present disclosure, the first acquisition submodule includes a job data determination unit, a data alignment unit, a first data writing unit and a first data rolling unit.
[0144] The job data determining unit is used to determine a plurality of job data located within a batch processing time window from the job data stream.
[0145] The data alignment unit is used to align the event triggering time of each of a plurality of job data based on the event triggering time information represented by the batch processing time window to obtain a plurality of aligned job data.
[0146] The first data writing unit is used to write the multiple aligned job data into the offline data warehouse.
[0147] The first data roll-up unit is used to roll up the aligned operation data written in the offline data warehouse within a first preset time period to obtain the first type of indicator data.
[0148] According to an embodiment of the present disclosure, the second acquisition submodule includes a second data writing unit and a second data rolling unit.
[0149] The second data writing unit is used to write the job data included in the job data stream into the real-time data warehouse.
[0150] The second data roll-up unit is used to roll up the operation data written in the real-time data warehouse within a second preset time period to obtain the second type of indicator data.
[0151] According to an embodiment of the present disclosure, the data correction module 620 includes a data recall submodule, a result reasoning submodule and a data correction submodule.
[0152] The data recall submodule is used to recall abnormal data from the advertising system when the data evaluation result indicates that there is an abnormal phenomenon in the data flow of the advertising system.
[0153] The result reasoning submodule is used to obtain the reasoning result based on the data attributes of the abnormal data and the data attributes of the normal data associated with the abnormal data in the data stream.
[0154] The data correction submodule is used to obtain the target data stream based on the inference result by using the data stream and the abnormal data.
[0155] According to an embodiment of the present disclosure, the data recall submodule includes a reverse search unit and a data recall unit.
[0156] The reverse search unit is used to reversely search the streaming link based on the target indicator indicated by the abnormal phenomenon when the data evaluation result indicates that there is an abnormal phenomenon in the data flow of the advertising system, so as to obtain the flow information of the target indicator.
[0157] The data recall unit is used to recall abnormal data from the advertising system based on the flow information of the target indicator.
[0158] According to an embodiment of the present disclosure, the data correction submodule includes a first data correction unit and a second data correction unit.
[0159] The first data correction unit is used to splice the abnormal data into the data stream to obtain the target data stream when the inference result represents that the data is valid.
[0160] The second data correction unit is used to obtain a target data stream based on the data stream when the inference result represents that the data is invalid.
[0161] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0162] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.
[0163] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method as described above.
[0164] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method as described above.
[0165] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0166] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0167] A number of components in the device 700 are connected to an input / output (I / O) interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0168] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).
[0169] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0170] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0171] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0172] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0173] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0174] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.
[0175] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0176] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A data processing method, comprising: In response to the streaming computing request, data evaluation is performed based on the indicator data collected from the streaming link of the advertising system to obtain a data evaluation result; When the data evaluation result indicates that there is an abnormality in the data stream of the advertising system, performing correction processing on the data stream to obtain a target data stream; as well as Stream computing is performed based on the target data stream.
2. The method according to claim 1, wherein: The indicator data includes first-category indicator data and second-category indicator data, wherein the first-category indicator data is data aligned based on the event triggering time, and the second-category indicator data is data aligned based on the event execution time; The step of performing data evaluation based on the indicator data collected from the streaming link of the advertising system in response to the streaming computing request to obtain the data evaluation result includes: Based on the first type of indicator data, an integrity assessment result is obtained; Based on the second type of indicator data, obtaining a correctness evaluation result; and The data evaluation result is obtained based on the integrity evaluation result and the correctness evaluation result.
3. The method according to claim 2, wherein: The integrity assessment result obtained based on the first type of indicator data includes: Based on the integrity index, first target index data is obtained by screening from the first category of index data; Extracting first entry data related to an entry end of the advertising system and first exit data related to an exit end of the advertising system from the first target indicator data; Based on a maximum delay time window, second inlet data is intercepted from the first inlet data, and second outlet data is intercepted from the first outlet data, wherein the maximum delay time window is determined based on a triggering time of the streaming computing request and a maximum delay duration of the advertising system; and The integrity assessment result is obtained based on a comparison result of the second input data and the second output data.
4. The method according to claim 2, wherein: The correctness evaluation result obtained based on the second type of indicator data includes: Based on the correctness index, second target index data is obtained by screening from the second type of index data, the streaming link includes a plurality of jobs, and the second target index data includes sub-index data related to each of the plurality of jobs; Combining the sub-indicator data related to each of the plurality of operations to obtain combined data; and The combined data is input into a data model to obtain the correctness evaluation result.
5. The method according to claim 2, wherein: The streaming link includes a plurality of jobs, and two adjacent jobs in the plurality of jobs interact with each other through a message middleware; The method further comprises: Data collection is performed on the job data streams recorded in each of the plurality of message middlewares included in the streaming link to obtain the first category indicator data and the second category indicator data respectively.
6. The method according to claim 5, wherein: The collecting data of the job data streams recorded in the plurality of message middlewares included in the streaming link to obtain the first category indicator data and the second category indicator data respectively includes: Taking the event triggering time as the time reference, respectively collect data from the job data streams recorded in the plurality of message middlewares to obtain the first category of indicator data; and Taking the event execution time as the time reference, data collection is performed on the job data streams recorded in each of the plurality of message middlewares to obtain the second category indicator data.
7. The method according to claim 6, wherein: The method of collecting data from the job data streams recorded in the plurality of message middlewares respectively based on the event triggering time as the time reference to obtain the first type of indicator data includes: Determine a plurality of job data within a batch processing time window from the job data stream; Based on the event triggering time information represented by the batch processing time window, aligning the event triggering time of each of the plurality of job data to obtain a plurality of aligned job data; Writing the plurality of aligned job data into an offline data warehouse; and The aligned operation data written in the offline data warehouse within a first preset time period is rolled up to obtain the first category indicator data.
8. The method according to claim 6, wherein: The method of collecting data from the job data streams recorded in the plurality of message middlewares respectively based on the event execution time as the time reference, and obtaining the second type of indicator data, includes: Writing the job data included in the job data stream into a real-time data warehouse; and The operation data written in the real-time data warehouse within a second preset time period is rolled up to obtain the second category indicator data.
9. The method according to claim 1, wherein: When the data evaluation result indicates that there is an abnormality in the data stream of the advertising system, the data stream is corrected to obtain a target data stream, including: When the data evaluation result indicates that there is an abnormality in the data stream of the advertising system, recalling the abnormal data from the advertising system; Obtaining an inference result based on data attributes of the abnormal data and data attributes of normal data associated with the abnormal data in the data stream; and Based on the inference result, the target data stream is obtained by using the data stream and the abnormal data.
10. The method according to claim 9, wherein: When the data evaluation result indicates that there is an abnormal phenomenon in the data flow of the advertising system, recalling the abnormal data from the advertising system includes: When the data evaluation result indicates that an abnormal phenomenon exists in the data stream of the advertising system, based on the target indicator indicated by the abnormal phenomenon, reverse search is performed on the streaming link to obtain the flow information of the target indicator; and Based on the flow information of the target indicator, the abnormal data is retrieved from the advertising system.
11. The method according to claim 9, wherein: The inference results include valid data and invalid data; Wherein, obtaining the target data stream by using the data stream and the abnormal data based on the inference result includes: When the inference result represents that the data is valid, the abnormal data is spliced into the data stream to obtain the target data stream; In the case where the inference result represents that the data is invalid, the target data flow is obtained based on the data flow.
12. A data processing device, comprising: A data evaluation module, for responding to a streaming computing request, performing data evaluation based on indicator data collected from a streaming link of an advertising system, and obtaining a data evaluation result; A data correction module, configured to correct the data stream to obtain a target data stream when the data evaluation result indicates that there is an abnormality in the data stream of the advertising system; as well as A stream computing module is used to perform stream computing based on the target data stream.
13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.
14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.
15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.
Citation Information
Cited By
Service performance evaluation method, computer program product and electronic device
CN120723342A
Service performance evaluation method, computer program product, and electronic device
CN120723342B
