Data processing method and apparatus, device, and storage medium

Abstract

This application provides a data processing method, apparatus, device, and storage medium. The method includes: collecting business data from each target business terminal in a data platform based on preset business tracking points; obtaining target business fields and field annotation information from the business data; extracting abnormal business data from the business data based on the target business fields and the field annotation information; annotating the abnormal business data to obtain updated business data for the target business terminal; and responding to a data distribution request by distributing the updated business data to a preset downstream data terminal. The technical solution provided by this application can improve the timeliness and accuracy of updating annotation information in business data by detecting unannotated abnormal business data in real time and annotating the abnormal business data.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] Currently, with the rapid development of the computer and AI fields, enterprises require an increasing number of data sources to conduct various business operations. Data platforms acquire business data from various business data sources, organize the data, and make it available to other business parties. However, as data sources iterate on business data, new values ​​may appear for the same fields. Other business parties may not be aware of these new values ​​and their meaning, leading to data analysis errors. This could cause other business parties to flag these values ​​as abnormal, impacting business operations. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, and storage medium, aiming to solve the technical problem in the prior art where new values ​​are generated when business data is updated and iterated, leading to misjudgments by other business parties.

[0004] On one hand, embodiments of this application provide a data processing method, which includes the following steps:

[0005] Based on preset business tracking points, collect business data from each target business terminal in the data platform;

[0006] Obtain the target business fields and field annotation information from the business data, and extract abnormal business data from the business data based on the target business fields and field annotation information;

[0007] Annotate abnormal business data to obtain updated business data from the target business end;

[0008] In response to a data distribution request, the updated business data will be distributed to the preset downstream data endpoints.

[0009] In one possible implementation of this application, the target business field and field annotation information in the business data are obtained, and abnormal business data in the business data are extracted based on the target business field and field annotation information, including:

[0010] Retrieve the target business field and field annotation information that correspond to the preset monitoring fields in the business data;

[0011] If the field annotation information is different from each of the preset annotation information in the preset monitoring fields, or if the field annotation information is empty, then the business data corresponding to the target business field will be set as abnormal business data.

[0012] In one possible implementation of this application, abnormal business data is marked to obtain updated business data from the target business side, including:

[0013] Send an update annotation request to the target business terminal and obtain the update annotation information returned by the target business terminal;

[0014] Based on the preset annotation information, update annotation information, and field annotation information of abnormal business data, update business data is generated.

[0015] In one possible implementation of this application, updated business data is generated based on preset annotation information, update annotation information, and field annotation information of abnormal business data, including:

[0016] Retrieve the updated annotation text and updated annotation values ​​from the updated annotation information;

[0017] Input the updated annotation text and updated annotation values ​​into the preset annotation information to generate the updated annotation information;

[0018] The updated business data is generated by updating the field annotations in the field annotation information corresponding to the abnormal business data based on the updated annotation text.

[0019] In one possible implementation of this application, updated business data is generated based on preset annotation information, update annotation information, and field annotation information of abnormal business data, including:

[0020] Retrieve the data type of abnormal business data;

[0021] If the data type of the abnormal business data is different from the preset data type of the target business field, then delete the abnormal business data;

[0022] If the data type of the abnormal business data is the same as the preset data type of the target business field, then the preset annotation information and field annotation information are updated according to the update comment information to obtain the updated business data.

[0023] In one possible implementation of this application, responding to a data distribution request and distributing normal business data and updated business data to a preset downstream data terminal includes:

[0024] Obtain the first field name of the normal business data and the second field name of the updated business data from the business data;

[0025] If the first field name and the second field name are the same, then the normal business data and the updated business data are merged into the database to obtain the first merged data, and the first merged data is distributed to the preset downstream data terminal.

[0026] In one possible implementation of this application, responding to a data distribution request and distributing updated business data to a preset downstream data endpoint includes:

[0027] Retrieve the first field name and first comment content of normal business data from the business data, and the second field name and second comment content of updated business data;

[0028] If the first field name and the second field name are different, and the first comment content and the second comment content are the same, then create a target mapping table;

[0029] Based on the target mapping table, normal business data and updated business data are merged into the database to obtain the second merged data, which is then distributed to the preset downstream data terminals.

[0030] On the other hand, this application provides a data processing apparatus, which includes:

[0031] The data acquisition module is configured to collect business data from each target business terminal in the data platform based on preset business tracking points;

[0032] The anomaly detection module is configured to acquire the target business fields and field annotation information in the business data, and extract the abnormal business data in the business data based on the target business fields and field annotation information.

[0033] The data annotation module is configured to annotate abnormal business data to obtain updated business data from the target business end.

[0034] The data distribution module is configured to respond to data distribution requests and distribute updated business data to preset downstream data terminals.

[0035] On the other hand, this application also provides a data processing device, which includes:

[0036] One or more processors;

[0037] Memory; and

[0038] One or more applications, wherein the applications are stored in memory and configured to be executed by a processor to implement steps of a data processing method.

[0039] On the other hand, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to execute steps in a data processing method.

[0040] This application collects business data from various target business terminals in a data platform based on preset business tracking points; obtains target business fields and field annotation information from the business data; extracts abnormal business data from the business data based on the target business fields and field annotation information; annotates the abnormal business data to obtain updated business data for the target business terminals; and responds to data distribution requests to distribute the updated business data to preset downstream data terminals. This achieves real-time detection and annotation of unannotated abnormal business data, improving the timeliness and accuracy of updating annotation information in the business data. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram illustrating a scenario of the data processing method in an embodiment of this application.

[0043] Figure 2 This is a flowchart illustrating one embodiment of the data processing method in this application.

[0044] Figure 3 A flowchart illustrating one embodiment of the data processing method for generating updated business data provided in this application;

[0045] Figure 4 A flowchart illustrating an embodiment of the data processing method for deduplication and data entry provided in this application;

[0046] Figure 5 A schematic diagram of the structure of one embodiment of the data processing apparatus provided in this application;

[0047] Figure 6 This is a schematic diagram of one embodiment of the data processing device provided in this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0050] In this application, the term "exemplary" is used to mean "serving as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0051] Currently, with the rapid development of the computer and AI fields, enterprises require an increasing number of data sources to conduct various business operations. Data platforms acquire business data from various business data sources, organize the data, and make it available to other business parties. However, as data sources iterate on business data, new values ​​may appear for the same fields. Other business parties may not be aware of these new values ​​and their meaning, leading to data analysis errors. This could cause other business parties to flag these values ​​as abnormal, impacting business operations.

[0052] Based on this, this application proposes a data processing method, apparatus, device, and computer-readable storage medium to solve the technical problem in the prior art where new values ​​are generated when business data is updated and iterated, leading to misjudgments by other business parties.

[0053] The data processing method in this embodiment of the invention is applied to a data processing device, which is disposed in a data processing equipment. The data processing equipment is provided with one or more processors, a memory, and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the data processing method. The data processing equipment can be a smart terminal, such as a mobile phone, tablet computer, network device, and smart computer. Optionally, the data processing equipment can also be a server or a service cluster composed of multiple servers.

[0054] like Figure 1 As shown, Figure 1 This is a schematic diagram illustrating a data processing method scenario according to an embodiment of this application. The data processing scenario in this embodiment includes a data processing device 100 (which integrates a data processing unit) and a target business terminal 200. The data processing device 100 runs a computer-readable storage medium corresponding to the data processing method to execute the steps of the data processing method. The target business terminal 200 is a business terminal that interfaces with the data processing device 100 to annotate business data, or receives business data distributed by the data processing terminal 100.

[0055] Understandable Figure 1 The data processing device in the data processing method scenario shown, or the device included in the data processing device, does not constitute a limitation on the embodiments of the present invention. That is, the number or type of data processing device in the data processing method scenario, or the number or type of device included in each device, does not affect the overall implementation of the technical solution in the embodiments of the present invention, and can all be considered as equivalent substitutions or derivatives of the technical solutions claimed in the embodiments of the present invention.

[0056] In this embodiment of the invention, the data processing device 100 is mainly used for: collecting business data from each target business terminal in the data platform based on preset business embedding points;

[0057] Obtain the target business fields and field annotation information from the business data, and extract abnormal business data from the business data based on the target business fields and field annotation information;

[0058] Annotate abnormal business data to obtain updated business data from the target business end;

[0059] In response to a data distribution request, the updated business data will be distributed to the preset downstream data endpoints.

[0060] The data processing device 100 in this embodiment of the invention can be an independent data processing device, such as a mobile phone, tablet computer, network device, server and smart computer, or a data processing network or data processing cluster composed of multiple data processing devices.

[0061] This application provides a data processing method, apparatus, device, and computer-readable storage medium, which will be described in detail below.

[0062] It will be understood by those skilled in the art that Figure 1 The application environment shown is only one application scenario related to the solution of this application and does not constitute a limitation on the application scenario of this application. Other application environments may include more than one application scenario. Figure 1 The number of data processing devices shown, or the data processing network connections, for example Figure 1 Only one data processing device is shown in the figure. It is understood that the scenario of this data processing method may also include one or more data processing devices, which are not limited here. The data processing device 100 may also include a memory for storing business data.

[0063] It should be noted that, Figure 1 The schematic diagram of the data processing method shown is merely an example. The scenarios of the data processing method described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided in the embodiments of the present invention.

[0064] Based on the scenarios described above, various embodiments of the data processing method disclosed in this invention are proposed.

[0065] like Figure 2 As shown, Figure 2 This is a flowchart illustrating one embodiment of the data processing method in this application. The data processing method includes the following steps 201 to 204:

[0066] 201. Collect business data from each target business terminal in the data platform based on preset business data points;

[0067] The data processing method in this embodiment is applied to a data processing device. The type and number of data processing devices are not specifically limited. That is, the data processing device can be one or more smart terminals or servers. In a specific embodiment, the data processing device is a smart computer.

[0068] Specifically, the data processing equipment pre-establishes a data platform for receiving and processing business data transmitted from upstream target business terminals. During operation, the equipment responds to business data collection requests. These requests do not drive the data processing equipment to retrieve business data from associated target business terminals based on preset business tracking points. The triggering method for these requests is not specifically limited; for example, they can be triggered by the user, such as by clicking a data collection button on the data processing equipment. Optionally, the requests can also be triggered automatically by the data processing equipment, for example, by setting up an automatic collection process within a preset time period.

[0069] Specifically, the data processing equipment pre-inserts preset business tracking points into the target business terminal. Upon receiving a business data collection request, it collects authorized business data from the target business terminal through these preset tracking points according to the corresponding data collection method. The business data is then transmitted to the data platform, and further processed by preset big data computing and data transmission tools to reach the data warehouse within the data platform. Finally, the data is stored in the big data storage and query table within the data warehouse. In one specific embodiment, the big data computing tool is the Flink computing engine, and the big data transmission tool is the Kafka messaging system. The business data consists of heterogeneous data generated by various business lines within the target business terminal. The data platform is configured to integrate data sources from multiple target business terminals, store data uniformly, and then layer the data according to the data processing method. Each business terminal can independently acquire data at different stages of data processing as needed, and finally perform personalized processing and output on a data management platform.

[0070] 202. Obtain the target business fields and field annotation information from the business data, and extract abnormal business data from the business data based on the target business fields and field annotation information;

[0071] Specifically, after acquiring business data, the data processing device also acquires the target business fields and field annotation information in the business data, and extracts abnormal business data from the business data based on the target business fields and associated field annotation information.

[0072] Specifically, the data processing equipment is pre-set with preset monitoring fields for judging business data. These preset monitoring fields determine whether the business data is abnormal, such as unclear or uncommented data, thereby avoiding analysis errors caused by abnormal business data at the downstream data end.

[0073] Specifically, the data processing device acquires the target business field and field annotation information corresponding to the preset monitoring field in the business data. Based on the preset monitoring field, target business field, and field annotation information, it performs data supervision on the business data to identify abnormal business data. The target business field is a data field carrying specific annotation information used for business data verification. Optionally, in one specific embodiment, the business data is audiovisual business data, and the target business field is a program type. The target business field includes a value of 1, corresponding to the field annotation information "TV series"; a value of 2, corresponding to the field annotation information "movie"; and a value of 3, corresponding to the field annotation information "animation," etc.

[0074] Specifically, after acquiring the preset monitoring fields, target business fields, and field annotation information, the data processing equipment compares the preset monitoring fields, target business fields, and field annotation information to determine abnormal business data.

[0075] Optionally, if the field value and corresponding field annotation information in the target business field of the business data are different from each preset annotation information in the preset monitoring field, that is, if there is business data with unknown annotation information in the business data, the data processing device sets the business data associated with the target business field as abnormal business data and sends the corresponding reminder information to the corresponding target business terminal.

[0076] In one specific embodiment, the value range of the preset monitoring field is an integer in the range [1, 3], and the preset annotation information corresponding to each integer is the program type of the audiovisual service data. If the field value of the target service field in the service data exceeds this value range, or if the field annotation information is different from the preset annotation information of the corresponding field value in the preset monitoring field, the data processing device determines that the service data corresponding to the target service field is abnormal service data. Optionally, the value range of the preset monitoring field can be customized according to the service type or data type of different service data.

[0077] Optionally, if the field labeling information of the target business field corresponding to the preset monitoring field in the business data is empty, that is, there is unlabeled business data in the business data, the data processing device sets the business data corresponding to the target business field as abnormal business data.

[0078] 203. Mark the abnormal business data to obtain the updated business data of the target business end;

[0079] Specifically, after identifying abnormal business data in the business data through preset monitoring fields, target business fields, and field annotation information, the data processing equipment also annotates the abnormal business data to obtain the updated business data from the target business end.

[0080] Optionally, after acquiring abnormal business data, the data processing device also performs error judgment on the abnormal business data to determine whether the abnormal business data is normal business data generated by the upstream target business version update that is not defined by the data processing device, or erroneous data generated by garbled characters, network errors, or test data being mistakenly entered.

[0081] Optionally, the data processing device obtains the data type of the abnormal business data; if the data type of the abnormal business data is different from the preset data type of the target business field, the data processing device determines that the abnormal business data is erroneous data and deletes the erroneous data.

[0082] If the data type of the abnormal business data is the same as the preset data type of the target business field, the data processing device determines that the abnormal business data is normal business data that was not defined by the data processing device due to the version update of the upstream target business terminal. The device then updates the preset annotation information and field annotation information according to the update annotation information to obtain the updated business data.

[0083] After determining that the abnormal business data is erroneous, the data processing equipment traces the erroneous data and deletes or corrects the erroneous data after the trace.

[0084] Specifically, after determining that abnormal business data is normal business data, the data processing device sends an update annotation request to the data source of the abnormal business data, i.e., the upstream target business end, and receives the update annotation information fed back by the target business end. The update annotation information is the business data annotation information generated after the data is updated.

[0085] Specifically, after obtaining the update annotation information, the data processing device performs annotation update operations based on the update annotation information, the corresponding preset annotation information in the preset monitoring fields, and the field annotation information of the abnormal business data, updates the abnormal annotation information in the abnormal business data, and generates updated business data.

[0086] 204. Respond to the data distribution request and distribute the updated business data to the preset downstream data terminal.

[0087] Specifically, after the data processing device obtains the updated business data from the target business end, it merges and deduplicates the normal business data and the updated business data in the business data and stores them in a preset big data cluster database. It then updates and overwrites the data according to a preset cycle and responds to the data distribution request from the preset downstream data end, distributing the updated business data to the preset downstream data end.

[0088] Specifically, after the data processing device deduplicatively stores the business data according to the preset field names and preset field contents, it also responds to the data distribution request sent by the preset downstream data terminal, parses the data distribution request, and sends the normal business data and updated business data specified in the data distribution request to the specified page front end. Each preset downstream data terminal can query or process the business data through the page front end.

[0089] In this embodiment, the data processing device collects business data from each target business terminal in the data platform based on preset business tracking points; it obtains target business fields and field annotation information from the business data; based on the target business fields and field annotation information, it extracts abnormal business data from the business data; it annotates the abnormal business data to obtain updated business data for the target business terminal; and it responds to a data distribution request by distributing the updated business data to preset downstream data terminals. This achieves real-time detection and annotation of unannotated abnormal business data in the business data, improving the timeliness and accuracy of updating annotation information in the business data.

[0090] like Figure 3 As shown, Figure 3 This is a flowchart illustrating an embodiment of the data processing method for generating updated business data provided in this application. Specifically, the data processing method includes steps S301 to S303:

[0091] 301. Retrieve the update annotation text and update annotation value from the update annotation information;

[0092] 302. Input the updated annotation text and updated annotation values ​​into the preset annotation information to generate updated annotation information;

[0093] 303. Update the field annotation content in the field annotation information corresponding to the abnormal business data according to the updated annotation text, and generate updated business data.

[0094] Based on the above embodiments, in this embodiment, after obtaining the update annotation information, the data processing device annotates and updates the abnormal business data according to the update annotation text and update annotation value in the update annotation information, and generates updated business data.

[0095] Specifically, the data processing device acquires the update annotation text and update annotation value from the update annotation information. The update annotation text is the annotation text used to update field annotation information in abnormal business data. The update annotation value is the business field value mapped to the update annotation text.

[0096] Specifically, after acquiring the updated annotation text and updated annotation value, the data processing device updates the preset annotation information based on the updated annotation text and updated annotation value. That is, the updated annotation text and updated annotation value are input into the preset annotation information. If the preset annotation information does not store the historical version annotation information associated with the updated annotation value and updated annotation text, a mapping of the updated annotation text and updated annotation value is added to the preset annotation information to generate updated annotation information. If the preset annotation information stores historical version annotation information of the updated annotation value and updated annotation text, the data processing device overwrites the historical version annotation information with the updated annotation value and updated annotation text to generate updated annotation information.

[0097] Specifically, the data processing device also performs text updates on abnormal business data based on the update annotation text and update annotation values. If the target business field and field annotation information in the abnormal business data do not match the update annotation text and update annotation values, the data processing device updates the field annotation content in the field annotation information corresponding to the abnormal business data using the update annotation text, and generates updated business data. The target business field and associated field annotation information in the updated business data are the same as the update annotation values ​​and update annotation text.

[0098] In this embodiment, the data processing device acquires the update annotation text and update annotation value from the update annotation information; inputs the update annotation text and update annotation value into preset annotation information to generate update annotation information; and updates the field annotation content in the field annotation information corresponding to the abnormal business data according to the update annotation text to generate updated business data. This enables timely acquisition of update annotation information and updates to preset annotation information and abnormal business data based on the update annotation information, improving the interpretability of business data and reducing the probability of data misjudgment.

[0099] like Figure 4 As shown, Figure 4 A flowchart illustrating an embodiment of the data processing method provided in this application for deduplication and data entry, specifically including steps 401 to 403:

[0100] 401. Obtain the first field name and first comment content of normal business data and the second field name and second comment content of updated business data;

[0101] 402. If the first field name and the second field name are different, and the first comment content and the second comment content are the same, then create a target mapping table;

[0102] 403. Based on the target mapping table, merge normal business data and updated business data into the database to obtain second merged data, and distribute the second merged data to the preset downstream data terminal.

[0103] Based on the above embodiments, in this embodiment, before responding to the data distribution request, the data processing device performs deduplication and data entry operations on the normal business data and the updated business data in the business data.

[0104] Specifically, in one embodiment, the data processing device acquires the first field name and first annotation content of each normal business data in the business data, and updates the second field name and second field content in the business data. The first field name is a preset field name used for deduplication identification in the normal business data that has not been updated. The second field name is a preset field name corresponding to the first field name in the updated business data. The first annotation content is the field annotation content associated with the field corresponding to the first field name. The second annotation content is the field annotation content associated with the field corresponding to the second field name.

[0105] Specifically, after obtaining the first field name and the second field name, as well as the first comment content and the second comment content, the data processing device performs deduplication and data entry processing on the business data based on the first field name, the second field name, the first comment content, and the second comment content.

[0106] Optionally, in one embodiment, the data processing device compares the first field name and the second field name, and performs deduplication and data entry processing based on the comparison result.

[0107] If the comparison result shows that the first field name and the second field name are the same, the data processing device will merge the normal business data and the updated business data into the database, merge fields with the same dimension, and make a note in the corresponding data table in the big data cluster database, so that the normal business data and the updated business data are merged and stored in a unified manner.

[0108] Optionally, in another embodiment, if the first field name and the second field name are different, the data processing device further compares the first annotation content and the second annotation content. If the first annotation content and the second annotation content are the same, that is, the first field name and the second field name are different, and the first annotation content and the second annotation content are the same, the data processing device creates a target mapping table corresponding to the normal business data and the updated business data, and transforms the normal business data and the updated business data based on the target mapping table, and stores the transformed normal business data and the updated business data in a unified manner into the big data cluster database.

[0109] Optionally, the data processing device can also compare the first field name and the first annotation content of different normal business data according to the above steps, and perform deduplication and data entry processing on the different normal business data based on the first field name and the first annotation content.

[0110] Optionally, the data processing device can also compare the second field name and the second annotation content of different updated business data according to the above steps, and perform deduplication and data entry processing on the different updated business data based on the second field name and the second traceability content.

[0111] In this embodiment, the data processing device obtains the first field name and first annotation content of normal business data and the second field name and second annotation content of updated business data. If the first field name and second field name are different, but the first annotation content and second annotation content are the same, a target mapping table is created. Based on the target mapping table, the normal business data and updated business data are merged into the database to obtain second merged data, which is then distributed to a preset downstream data terminal. This achieves unified storage of business data in the big data cluster database based on field names and annotation content, improving data storage efficiency and integration.

[0112] To better implement the data processing method in the embodiments of this application, based on the data processing method, the embodiments of this application also provide a data processing apparatus, such as... Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application. Specifically, the data processing apparatus 500 includes:

[0113] The data acquisition module 501 is configured to collect business data from each target business terminal in the data platform based on preset business tracking points;

[0114] The anomaly identification module 502 is configured to acquire the target business field and field annotation information in the business data, and extract the abnormal business data in the business data based on the target business field and field annotation information.

[0115] The data annotation module 503 is configured to annotate abnormal business data to obtain updated business data from the target business end;

[0116] The data distribution module 504 is configured to respond to data distribution requests and distribute updated business data to preset downstream data terminals.

[0117] In one possible implementation of this embodiment, the data processing device acquires target business fields and field annotation information from business data, and extracts abnormal business data from the business data based on the target business fields and field annotation information, including:

[0118] Retrieve the target business field and field annotation information that correspond to the preset monitoring fields in the business data;

[0119] If the field annotation information is different from each of the preset annotation information in the preset monitoring fields, or if the field annotation information is empty, then the business data corresponding to the target business field will be set as abnormal business data.

[0120] In one possible implementation of this embodiment, the data processing device marks abnormal business data to obtain updated business data from the target business end, including:

[0121] Send an update annotation request to the target business terminal and obtain the update annotation information returned by the target business terminal;

[0122] Based on the preset annotation information, update annotation information, and field annotation information of abnormal business data, update business data is generated.

[0123] In one possible implementation of this embodiment, the data processing device generates updated business data based on preset annotation information, update annotation information, and field annotation information of abnormal business data, including:

[0124] Retrieve the updated annotation text and updated annotation values ​​from the updated annotation information;

[0125] Input the updated annotation text and updated annotation values ​​into the preset annotation information to generate the updated annotation information;

[0126] The updated business data is generated by updating the field annotations in the field annotation information corresponding to the abnormal business data based on the updated annotation text.

[0127] In one possible implementation of this embodiment, the data processing device generates updated business data based on preset annotation information, update annotation information, and field annotation information of abnormal business data, including:

[0128] Retrieve the data type of abnormal business data;

[0129] If the data type of the abnormal business data is different from the preset data type of the target business field, then delete the abnormal business data;

[0130] If the data type of the abnormal business data is the same as the preset data type of the target business field, then the preset annotation information and field annotation information are updated according to the update comment information to obtain the updated business data.

[0131] In one possible implementation of this embodiment, the data processing device responds to a data distribution request and distributes updated business data to a preset downstream data terminal, including:

[0132] Obtain the first field name of the normal business data and the second field name of the updated business data from the business data;

[0133] If the first field name and the second field name are the same, then the normal business data and the updated business data are merged into the database to obtain the first merged data, and the first merged data is distributed to the preset downstream data terminal.

[0134] In one possible implementation of this embodiment, the data processing device responds to a data distribution request and distributes updated business data to a preset downstream data terminal, including:

[0135] Retrieve the first field name and first comment content of normal business data from the business data, and the second field name and second comment content of updated business data;

[0136] If the first field name and the second field name are different, and the first comment content and the second comment content are the same, then create a target mapping table;

[0137] Based on the target mapping table, normal business data and updated business data are merged into the database to obtain the second merged data, which is then distributed to the preset downstream data terminals.

[0138] In this embodiment, the data processing device collects business data from each target business terminal in the data platform based on preset business data points; obtains target business fields and field annotation information from the business data; extracts abnormal business data from the business data based on the target business fields and field annotation information; annotates the abnormal business data to obtain updated business data for the target business terminal; and responds to a data distribution request to distribute the updated business data to a preset downstream data terminal. This enables real-time detection of unannotated abnormal business data in the business data and the annotation of such abnormal business data, improving the timeliness and accuracy of updating annotation information in the business data.

[0139] This invention also provides a data processing device, such as... Figure 6 As shown, Figure 6 This is a schematic diagram of one embodiment of the data processing device provided in this application.

[0140] The data processing device integrates any of the data processing apparatuses provided in the embodiments of the present invention, and the data processing device includes:

[0141] One or more processors;

[0142] Memory; and

[0143] One or more applications, wherein the applications are stored in memory and configured to be executed by a processor from the steps of the data processing method in any of the embodiments described above.

[0144] Specifically, the data processing device may include components such as a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, a power supply 603, and an input unit 604. Those skilled in the art will understand that... Figure 6 The data processing device structure shown does not constitute a limitation on the data processing device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0145] The processor 601 is the control center of the data processing device. It connects various parts of the data processing device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 602, and by calling data stored in the memory 602, it performs various functions of the data processing device and processes data, thereby providing overall monitoring of the data processing device. Optionally, the processor 601 may include one or more processing cores; preferably, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 601.

[0146] The memory 602 can be used to store software programs and modules. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the data processing device, etc. In addition, the memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.

[0147] The data processing device also includes a power supply 603 that supplies power to the various components. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 603 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0148] The data processing device may also include an input unit 604, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0149] Although not shown, the data processing device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 601 in the data processing device loads the executable files corresponding to the processes of one or more applications into the memory 602 according to the following instructions, and the processor 601 runs the applications stored in the memory 602 to realize various functions, as follows:

[0150] Based on preset business tracking points, collect business data from each target business terminal in the data platform;

[0151] Obtain the target business fields and field annotation information from the business data, and extract abnormal business data from the business data based on the target business fields and field annotation information;

[0152] Annotate abnormal business data to obtain updated business data from the target business end;

[0153] In response to a data distribution request, the updated business data will be distributed to the preset downstream data endpoints.

[0154] Therefore, embodiments of the present invention provide a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any of the data processing methods provided in the embodiments of the present invention. For example, the computer program loaded by the processor may execute the following steps:

[0155] Based on preset business tracking points, collect business data from each target business terminal in the data platform;

[0156] Obtain the target business fields and field annotation information from the business data, and extract abnormal business data from the business data based on the target business fields and field annotation information;

[0157] Annotate abnormal business data to obtain updated business data from the target business end;

[0158] In response to a data distribution request, the updated business data will be distributed to the preset downstream data endpoints.

[0159] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed descriptions of other embodiments above, which will not be repeated here.

[0160] In practice, each of the above units or structures can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units or structures, please refer to the previous method embodiments, which will not be repeated here.

[0161] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0162] The data processing method provided by the embodiments of this application has been described in detail above. Specific embodiments have been used to illustrate the principles and implementation of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data processing method, characterized in that, The data processing method includes: Based on preset business tracking points, collect business data from each target business terminal in the data platform; Obtain the target business field and field annotation information corresponding to the preset monitoring field in the business data; If the field annotation information is different from each of the preset annotation information in the preset monitoring fields, or if the field annotation information is empty, then the business data corresponding to the target business field will be set as abnormal business data. The abnormal business data is labeled to obtain the updated business data of the target business terminal; In response to the data distribution request, the updated business data is distributed to a preset downstream data terminal.

2. The data processing method as described in claim 1, characterized in that, The step of annotating the abnormal business data to obtain the updated business data for the target business terminal includes: Send an update annotation request to the target service terminal and obtain the update annotation information returned by the target service terminal; Updated business data is generated based on preset annotation information, the update annotation information, and the field annotation information of the abnormal business data.

3. The data processing method as described in claim 2, characterized in that, The step of generating updated business data based on preset annotation information, the updated annotation information, and the field annotation information of the abnormal business data includes: Obtain the update annotation text and update annotation value from the update annotation information; Input the updated annotation text and the updated annotation value into the preset annotation information to generate updated annotation information; The updated business data is generated by updating the field annotation content in the field annotation information corresponding to the abnormal business data according to the updated annotation text.

4. The data processing method as described in claim 2, characterized in that, The step of generating updated business data based on preset annotation information, the updated annotation information, and the field annotation information of the abnormal business data includes: Obtain the data type of the abnormal business data; If the data type of the abnormal business data is different from the preset data type of the target business field, then the abnormal business data is deleted. If the data type of the abnormal business data is the same as the preset data type of the target business field, then the preset annotation information and the field annotation information are updated according to the update annotation information to obtain the updated business data.

5. The data processing method according to any one of claims 1-4, characterized in that, The response data distribution request, which distributes the updated business data to a preset downstream data terminal, includes: Obtain the first field name of the normal business data and the second field name of the updated business data from the business data; If the first field name and the second field name are the same, then the normal business data and the updated business data are merged into the database to obtain the first merged data, and the first merged data is distributed to the preset downstream data terminal.

6. The data processing method according to any one of claims 1-4, characterized in that, The response data distribution request, which distributes the updated business data to a preset downstream data terminal, includes: Obtain the first field name and first comment content of the normal business data in the business data, and the second field name and second comment content of the updated business data; If the first field name and the second field name are different, and the first comment content and the second comment content are the same, then create a target mapping table; Based on the target mapping table, the normal business data and the updated business data are merged into the database to obtain the second merged data, and the second merged data is distributed to the preset downstream data terminal.

7. A data processing apparatus, characterized in that, The data processing device includes: The data acquisition module is configured to collect business data from each target business terminal in the data platform based on preset business tracking points; The anomaly detection module is configured to obtain the target business field and field annotation information corresponding to the preset monitoring field in the business data; If the field annotation information is different from each of the preset annotation information in the preset monitoring fields, or if the field annotation information is empty, then the business data corresponding to the target business field will be set as abnormal business data. The data annotation module is configured to annotate the abnormal business data to obtain the updated business data of the target business terminal; The data distribution module is configured to respond to a data distribution request and distribute the updated business data to a preset downstream data terminal.

8. A data processing device, characterized in that, The data processing device includes: One or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the steps of the data processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to execute the steps of the data processing method according to any one of claims 1 to 6.

Images