Abnormal data processing method and device

Abstract

The embodiment of the invention provides an abnormal data processing method and device.The method comprises the steps that for abnormal data obtained through data detection, firstly, the abnormal data are analyzed to obtain service log data and a data exception type; secondly, writing the abnormal data into an assembly template corresponding to the data exception type to perform message assembly to obtain a data processing message, then calling a message sending platform interface to send the data processing message to a detection terminal, and if a data processing instruction for the abnormal data returned by interface calling is obtained, sending the data processing message to the detection terminal; and the abnormal data is processed according to the instruction type of the data processing instruction, so that the controllability and the flexibility of the data processing process are improved.

Description

Technical Field

[0001] This document relates to the field of data processing technology, and in particular to a method and device for processing abnormal data. Background Art

[0002] With the continuous development of data processing technology and the increasing number of application scenarios, it is becoming increasingly important to manage and analyze massive data through big data processing systems, and to provide efficient and accurate data services through multiple links such as data collection, data processing, and data storage.

[0003] At present, most big data processing systems rely mainly on preset rules and manual intervention when processing abnormal data. For example, for abnormal data that does not conform to the database field constraints, they will continuously restart and try to process these abnormal data, or directly discard the abnormal data; however, with the rapid growth of data volume and the increase in data processing complexity, on the one hand, constantly restarting to process abnormal data may cause delays in data processing, and on the other hand, directly discarding data may cause valuable data loss and affect the accuracy of data statistics. Therefore, how to effectively process abnormal data without affecting data processing efficiency has become an urgent problem to be solved in the field of data processing technology. Summary of the invention

[0004] An object of one embodiment of the present specification is to provide a method and device for processing abnormal data, so as to improve the controllability and flexibility of processing abnormal data.

[0005] To solve the above technical problems, an embodiment of this specification is implemented as follows: In a first aspect, an embodiment of the present specification provides a method for processing abnormal data, including: Analyze the abnormal data to obtain business log data and data abnormality type; Writing the abnormal data into an assembly template corresponding to the data abnormality type to perform message assembly to obtain a data processing message; Calling a message sending platform interface to send the data processing message to the detection terminal; If a data processing instruction for the abnormal data returned by the interface call is obtained, the abnormal data is processed according to the instruction type of the data processing instruction; the data processing instruction is submitted by the detection terminal.

[0006] In a second aspect, an embodiment of the present specification provides another abnormal data processing method, including: Obtain and display a data processing message of the abnormal data; the data processing message is obtained by writing the abnormal data into an assembly template corresponding to the data abnormality type for message assembly; generating a data processing instruction after a response control configured in the data processing message is triggered; The data processing instruction is sent and processed; the data processing instruction is used to perform data processing on the abnormal data based on the instruction type.

[0007] In a third aspect, an embodiment of the present specification provides an abnormal data processing device, including: A data analysis module is used to analyze abnormal data to obtain business log data and data abnormality types; A message assembly module, used for writing the abnormal data into an assembly template corresponding to the data abnormality type to perform message assembly to obtain a data processing message; A message sending module, used for calling a message sending platform interface to send the data processing message to a detection terminal; The data processing module is used to process the abnormal data according to the instruction type of the data processing instruction if a data processing instruction for the abnormal data returned by the interface call is obtained; the data processing instruction is submitted by the detection terminal.

[0008] In a fourth aspect, an embodiment of the present specification provides another abnormal data processing device, including: A message acquisition module, used to acquire and display data processing messages of abnormal data; the data processing messages are obtained by writing the abnormal data into an assembly template corresponding to the data abnormality type for message assembly; An instruction generation module, used for generating a data processing instruction after the response control configured in the data processing message is triggered; An instruction sending module is used to send the data processing instruction; the data processing instruction is used to perform data processing on the abnormal data based on the instruction type.

[0009] In the fifth aspect, another embodiment of the present specification provides an abnormal data processing device, comprising: a memory, a processor, and computer executable instructions stored on the memory and executable on the processor, wherein the computer executable instructions, when executed by the processor, implement the steps of the abnormal data processing method as described in the first aspect above.

[0010] In the sixth aspect, another embodiment of the present specification provides another abnormal data processing device, including: a memory, a processor, and computer executable instructions stored on the memory and executable on the processor, and when the computer executable instructions are executed by the processor, the steps of the abnormal data processing method as described in the second aspect above are implemented.

[0011] In a seventh aspect, another embodiment of the present specification provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the abnormal data processing method as described in the first aspect above are implemented.

[0012] In an eighth aspect, another embodiment of the present specification provides another computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the abnormal data processing method as described in the second aspect above are implemented.

[0013] In a ninth aspect, another embodiment of the present specification provides a computer program product, wherein the computer program product includes an exception data processing program, and the exception data processing program is executed by a processor to implement the steps of the exception data processing method as described in the first aspect above.

[0014] In a tenth aspect, another embodiment of the present specification provides another computer program product, wherein the computer program product includes an exception data processing program, and the exception data processing program is executed by a processor to implement the steps of the exception data processing method as described in the second aspect above.

[0015] The present embodiment provides an abnormal data processing method. For the abnormal data obtained by data detection, the abnormal data is first parsed to obtain business log data and data anomaly types. By determining the data anomaly type of the abnormal data, a basis for subsequent data processing is provided. Secondly, the abnormal data is written into an assembly template corresponding to the data anomaly type to assemble the message to obtain a data processing message. By assembling the message based on the assembly template, the standardization and consistency of the data processing message are ensured. Then, the message sending platform interface is called to send the data processing message to the detection terminal. If the data processing instruction for the abnormal data returned by the interface call is obtained, the abnormal data is processed according to the instruction type of the data processing instruction. By allowing the detection terminal to select an appropriate data processing method according to the specific situation, the controllability and flexibility of the data processing process are improved. At the same time, by processing the abnormal data according to the data processing instruction type, it is ensured that the abnormal data will not be directly discarded, thereby retaining the integrity of the data.

[0016] Another abnormal data processing method provided by the present embodiment obtains and displays a data processing message of the abnormal data during the abnormal data processing process, so that the detection personnel can see the data abnormality type of the abnormal data and make corresponding data processing decisions based on the data abnormality type. After the response control configured in the data processing message is triggered, a data processing instruction is generated, and the generated data processing instruction is sent for processing, so as to process the abnormal data based on the instruction type of the data processing instruction. By generating the corresponding data processing instruction based on the operation of the detection personnel, the flexibility and accuracy of the data processing process are improved, and the efficiency of processing abnormal data is also improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in one or more embodiments of the present specification, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 A processing flow chart of an abnormal data processing method provided by one or more embodiments of this specification; Figure 2 A schematic diagram of an implementation environment of an abnormal data processing method provided in one or more embodiments of this specification; Figure 3 A schematic diagram of a data processing message provided for one or more embodiments of this specification; Figure 4 A processing flow chart of an abnormal data processing method applied to a first abnormal data processing scenario provided by an embodiment of this specification; Figure 5 A processing flow chart of an abnormal data processing method applied to a second abnormal data processing scenario provided by an embodiment of this specification; Figure 6 A processing flow chart of another abnormal data processing method provided for one or more embodiments of this specification; Figure 7 A schematic diagram of an abnormal data processing device provided for one or more embodiments of this specification; Figure 8 A schematic diagram of another abnormal data processing device provided for one or more embodiments of this specification; Fig. 9 A schematic diagram of the structure of an abnormal data processing device provided for one or more embodiments of this specification; Fig.10A schematic diagram of the structure of another abnormal data processing device provided for one or more embodiments of this specification. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0020] This specification provides an abnormal data processing method embodiment: Reference Figure 1 , which shows a processing flow chart of an abnormal data processing method provided in this embodiment. The abnormal data processing method provided in this embodiment specifically includes the following steps S102 to S108.

[0021] Step S102: parse the abnormal data to obtain business log data and data abnormality type.

[0022] In this embodiment, abnormal data refers to business log data that does not conform to preset specifications, wherein the preset specifications may be predefined specifications used to detect whether the business log data conforms to the preset specifications, and the preset specifications may include data types, field lengths, required fields, data formats, and / or other constraint specifications for business log data. Accordingly, abnormal data may be business log data whose field value type does not conform to the preset data type, or business log data whose field value exceeds the preset field length, or business log data whose field value is empty, or business log data that does not conform to other constraint specifications. Optionally, abnormal data is obtained after data detection.

[0023] In practical applications, abnormal data is obtained by performing data detection on business log data. On the one hand, data detection can ensure the integrity and consistency of business log data and reduce errors and failures caused by business log data. On the other hand, abnormal data in the business log data obtained through data detection can be identified and recorded to provide a data basis for subsequent data processing. In an optional implementation manner provided in this embodiment, data detection is performed on business log data before parsing the abnormal data to obtain business log data and data abnormality types. Specifically, data detection includes: Obtain business log data, and generate a data object for business log data mapping based on a preset mapping relationship; Check whether the data object meets the preset specifications; If not, perform data assembly based on the business log data and data anomaly type to obtain abnormal data; If so, store and process the business log data.

[0024] For example, Figure 2 As shown, the Flink big data processing system can read business log data from the Kafka (distributed stream processing platform) queue, and can filter, clean and / or otherwise pre-process the read business log data. In order to correctly store the pre-processed business log data in the database, a data object mapped to the business log data can be generated based on a preset mapping relationship, and the data object to be stored in the warehouse is detected to determine whether it meets the preset specifications to determine whether the data object can be normally stored in the warehouse. If it is detected that the data object meets the preset specifications, the data object is stored in the warehouse; if it is detected that the data object does not meet the preset specifications, the metadata of the business log data is obtained, and the business log data and the metadata of the business log data are assembled to obtain abnormal data. The metadata of the business log data may include identifying the data abnormality type of the obtained business log data, the data business type of the business log data and / or the Kafka queue name corresponding to the business log data, and the obtained abnormal data is sent to the abnormal data processing system for subsequent data processing.

[0025] In a specific implementation, after obtaining abnormal data based on data detection on business log data, the abnormal data can be parsed to obtain business log data and metadata of the business log data, wherein the metadata includes identifying the data anomaly type of the obtained business log data, that is: parsing the abnormal data to obtain business log data and the data anomaly type; in addition, the metadata may also include the data business type of the business log data and / or the Kafka queue name corresponding to the business log data.

[0026] The data exception type may include a field value type mismatch, a field value that is too long, and / or a field value that is empty. In addition, the data exception type may also include a data exception type that does not meet other field constraint standards.

[0027] Step S104, writing the abnormal data into the assembly template corresponding to the data abnormality type to perform message assembly to obtain a data processing message.

[0028] The assembly template refers to a predefined template for assembling exception data, which is used to assemble the exception data into a data processing message. Correspondingly, the assembly template corresponding to the data exception type refers to a predefined template for exception data of a specific data exception type, which is used to assemble the exception data under the data exception type into a data processing message.

[0029] For example, taking the data anomaly type as field value being too long as an example, the abnormal data of field value being too long can be written into an assembly template for processing abnormal data of field value being too long to perform message assembly to obtain a data processing message, and the obtained data processing message can display the specific content of data anomaly type as field value being too long and / or field value being too long; Figure 3 The data processing message shown may display a data exception type: field value is too long.

[0030] During specific implementation, in order to improve the pertinence and effectiveness of data processing, an assembly template can be determined based on the data anomaly type of the abnormal data. At the same time, in order to enable the detection terminal to quickly understand the data anomaly type of the abnormal data and thus perform corresponding data processing on the abnormal data, the abnormal data can be written into the assembly template corresponding to the data anomaly type for message assembly to obtain a data processing message containing basic information of the abnormal data.

[0031] In actual applications, in order to improve the efficiency and speed of managing and responding to abnormal data, a processing module specifically used for sending data processing messages can be configured on the message sending platform. For example, a chat robot can be applied for on the message sending platform to send data processing messages. At the same time, in order to enhance the security and reliability of the data processing message transmission process, the chat robot can be audited and authenticated to ensure that only authorized chat robots can send data processing messages and prevent unauthorized chat robots from sending malicious messages. Furthermore, an assembly template can be preset for the chat robot, and the assembly template can be configured with corresponding data processing controls. Additional data processing controls can also be configured in the assembly template according to different abnormal data types to improve the flexibility of data processing.

[0032] Step S106, calling the message sending platform interface to send the data processing message to the detection terminal.

[0033] During specific implementation, in order to achieve timely notification of data processing messages and improve the efficiency and speed of managing and responding to abnormal data, the message sending platform interface can be called to send the data processing messages to the detection terminal for the obtained data processing messages.

[0034] Among them, the detection terminal refers to a device or system that receives and displays data processing messages to notify detection personnel to perform data processing on abnormal data; the data processing message can be configured with a response control, and the response control includes at least one data processing control and / or a text input control; optionally, the data processing message is configured with at least one data processing control and / or a text input control.

[0035] Correspondingly, after sending the data processing message, the detection terminal can generate corresponding data processing instructions by triggering the response control configured by the data processing message, so that corresponding data processing can be performed on the abnormal data based on the data processing instructions during the subsequent execution process.

[0036] For example, Figure 3 As shown, the data processing message may be configured with a response control, the response control includes at least one data processing control and / or a text input control, and the data processing control may be Figure 3 In the details control shown in 301, the delete control shown in 302, the reprocessing control shown in 303 and / or the truncation control shown in 304, the text input control can be as shown in FIG. Figure 3 As shown in Figure 305.

[0037] In the specific execution process, in order to prevent too many data processing messages from being sent within a certain period of time and avoid overloading the message sending platform and the detection terminal, the sending frequency of the data processing message can be controlled so that the load of the message sending platform and the detection terminal is within a controllable range; in an optional implementation manner provided by this embodiment, before calling the message sending platform interface to send the data processing message to the detection terminal, the frequency threshold of sending the data processing message can be detected; specifically, detecting the frequency threshold of sending the data processing message includes: Preset the frequency threshold for calling the message sending platform interface to send data processing messages; Detecting whether the frequency of sending data processing messages within a preset time exceeds a frequency threshold; If yes, suspend sending the data processing message; If not, call the message sending platform interface to send the data processing message to the detection terminal.

[0038] Specifically, the frequency threshold for sending data processing messages can be preset according to the load capacity of the message sending platform. After each data processing message is obtained, the number of data processing messages that have been sent in the past certain period of time can be checked first to detect whether the frequency of sending data processing messages exceeds the frequency threshold. If so, it indicates that the number of data processing messages sent has reached the frequency threshold, and the process of sending the data processing message is suspended until the count of the frequency of sending the data processing message is reset; if not, it indicates that the number of data processing messages sent has not reached the frequency threshold, and the message sending platform interface can be called to send the data processing message to the detection terminal.

[0039] For example, Figure 2As shown, the 5G message sending platform can be called to send the data processing message (downlink message) to the detection terminal; accordingly, after receiving the data processing message (downlink message), the detection terminal can generate the corresponding data processing instruction by triggering the response control configured by the data processing message, and call the 5G message sending platform to send and process the data processing instruction, or the detection terminal can also call the 5G message sending platform to send and process the uplink message containing the data processing instruction.

[0040] Step S108: If a data processing instruction for the abnormal data returned by the interface call is obtained, data processing is performed on the abnormal data according to the instruction type of the data processing instruction.

[0041] After calling the message sending platform interface to send the data processing message to the detection terminal, the detection terminal can display the data processing message, generate the corresponding data processing instruction after triggering the response control configured for the data processing message, and send the data processing instruction. Correspondingly, here, if the data processing instruction for the abnormal data returned by the interface call is obtained, the abnormal data is processed according to the instruction type of the data processing instruction.

[0042] This embodiment improves the response speed and the efficiency of processing abnormal data by performing data processing on abnormal data after obtaining data processing instructions. At the same time, by performing corresponding data processing on abnormal data based on data processing instructions, the possibility of misprocessing abnormal data is reduced. On this basis, by processing abnormal data in a timely manner in response to data processing instructions, the risk of failure and data loss caused by abnormal data is reduced.

[0043] As described above, after parsing the abnormal data to obtain the log data and the metadata of the log data, the log data and the metadata of the log data can be stored in the database, and the corresponding primary key identifier can be obtained; and, after calling the message sending platform interface to send the data processing message to the detection terminal, the message identifier returned by the interface call can be obtained, and the message identifier can be associated with the primary key identifier so that the corresponding abnormal data can be queried in the subsequent data processing process; for example, the message identifier returned by the interface call can be used as the key, and the primary key identifier generated when the abnormal data is entered into the database can be used as the key value and stored in Redis.

[0044] In specific implementation, if the data processing instruction returned by the interface call is obtained, the instruction type of the data processing instruction can be determined, and the abnormal data can be processed according to the instruction type of the data processing instruction; optionally, the data processing instruction is submitted by the detection terminal.

[0045] Optionally, the data processing instruction includes a control instruction and / or a text instruction; accordingly, the instruction type corresponding to the control instruction is a control instruction type, and the instruction type corresponding to the text instruction is a text instruction type; based on this, data processing is performed on the abnormal data according to the instruction type of the data processing instruction.

[0046] The control instruction refers to the data processing instruction generated by the detection terminal through the data processing control configured by triggering the data processing message, and each data processing control may correspond to a data processing method; the text instruction refers to the data processing instruction obtained by the detection terminal by identifying the text entered in the text input control configured based on the data processing message.

[0047] Specifically, when the data processing instruction is a control instruction type, the abnormal data can be processed according to the control instruction generated by the triggering data processing control, and when the data processing instruction is a text instruction type, the abnormal data can be processed according to the text instruction generated by the recognition text. The following specifically describes two processes of processing abnormal data according to the instruction type of the data processing instruction.

[0048] (1) Control instruction type During the specific execution process, in order to improve the convenience and accuracy of the detection terminal submitting the data processing instruction, the data processing message can be configured with a data processing control, and the detection terminal can submit the data processing instruction based on the triggering data processing control. Accordingly, the instruction type of the data processing instruction generated based on the triggering data processing control is a control instruction type; in an optional implementation manner provided by this embodiment, data processing is performed on the abnormal data according to the instruction type of the data processing instruction, including: If the instruction type is a control instruction type, data processing is performed on the abnormal data based on the data processing method corresponding to the data processing instruction.

[0049] Optionally, the data processing instruction is generated after the data processing control is triggered.

[0050] For example, Figure 3 As shown, the data processing message may be configured with at least one data processing control, which may be Figure 3 The details control shown in 301, the delete control shown in 302, the reprocessing control shown in 303 and / or the truncation control shown in 304, respectively, can generate corresponding control instructions after the control is triggered.

[0051] Specifically, under the control instruction type, the detection terminal can generate corresponding control instructions by triggering the following four data processing controls. Correspondingly, after obtaining the corresponding control instruction sent by the detection terminal, the abnormal data can be processed based on the data processing method corresponding to the control instruction. The following describes the process of processing the abnormal data based on the data processing methods corresponding to the four control instructions.

[0052] (a) Details acquisition instruction In a first optional implementation manner provided by this embodiment, data processing is performed on abnormal data based on a data processing method corresponding to a data processing instruction, including: If a detail acquisition instruction is obtained, the detail data of the abnormal data is acquired and returned based on the detail acquisition instruction.

[0053] Optionally, the details acquisition instruction is generated after the details control configured in the trigger data processing message is triggered.

[0054] For example, if the detection terminal triggers Figure 3 The details control shown in 301 can generate a details acquisition instruction for the abnormal data; accordingly, after obtaining the details acquisition instruction, the details data of the associated abnormal data can be obtained through the primary key identifier, and the obtained details data can be written into the corresponding assembly template for message assembly to obtain the details data message, and the message platform interface is called to send the details data message to return the details data of the abnormal data to the detection terminal.

[0055] (b) Data deletion instruction In a second optional implementation manner provided by this embodiment, data processing is performed on abnormal data based on a data processing method corresponding to a data processing instruction, including: If a data deletion instruction is obtained, the abnormal data is deleted based on the data deletion instruction.

[0056] Optionally, a data deletion instruction is generated after a deletion control configured in a data processing message is triggered.

[0057] For example, if the detection terminal triggers Figure 3 The deletion control shown in 302 can generate a data deletion instruction for abnormal data; accordingly, after obtaining the data deletion instruction, the associated abnormal data can be obtained through the primary key identifier, and the obtained abnormal data can be deleted.

[0058] (c) Reprocessing instructions In a third optional implementation manner provided by this embodiment, data processing is performed on abnormal data based on a data processing method corresponding to a data processing instruction, including: If a reprocessing instruction is obtained, the abnormal data is returned to the message queue for recalculation based on the reprocessing instruction.

[0059] Optionally, a reprocessing instruction is generated after a reprocessing control configured in a data processing message is triggered.

[0060] For example, if the detection terminal triggers Figure 3 The reprocessing control shown in 303 can generate a reprocessing instruction for the abnormal data; accordingly, after obtaining the reprocessing instruction, the associated abnormal data can be obtained through the primary key identifier, and the obtained abnormal data can be re-delivered to the Kafka queue so that the Flink big data processing system can reprocess the abnormal data; Figure 2 As shown, the abnormal data can be re-delivered to the Kafka queue, and the Flink big data processing system can read the abnormal data from the Kafka queue and re-clean and calculate the abnormal data.

[0061] (d) Truncation Instructions In a fourth optional implementation manner provided by this embodiment, data processing is performed on abnormal data based on a data processing method corresponding to a data processing instruction, including: If a truncation instruction is obtained, the fields in the abnormal data that exceed the preset field threshold are truncated according to the preset field threshold based on the truncation instruction.

[0062] Optionally, a truncation instruction is generated after a truncation control configured in a data processing message is triggered.

[0063] For example, if the detection terminal triggers Figure 3 The truncation control shown in 304 can generate a truncation instruction for abnormal data; accordingly, after obtaining the truncation instruction, the associated abnormal data can be obtained through the primary key identifier, the overlong field content can be truncated according to the preset field threshold, and the processed abnormal data can be stored again.

[0064] In addition, in addition to the data processing methods corresponding to the above four data processing instructions, additional data processing controls can be added to the data processing message according to actual needs, and after the added data processing controls are triggered, corresponding data processing instructions are generated to process the abnormal data based on the data processing instructions. For example, if abnormal data with a field being empty is detected, an assignment control can be added to the data processing message, and if the detection terminal triggers the assignment control, an assignment instruction for the abnormal data can be generated; accordingly, after obtaining the assignment instruction, a default value can be assigned to the field according to the preset configuration, so that the abnormal data complies with the field constraints of the database, so that the abnormal data can be stored.

[0065] (2) Text instruction type In the specific implementation process, in order to improve the flexibility of data processing to cope with diverse and complex data processing needs, and to realize batch processing of data based on data processing instructions, thereby improving data processing efficiency, the data processing message can be configured with a text input control, and the detection terminal can generate a data processing instruction by identifying the text input based on the text input control; accordingly, the abnormal data can be processed based on the data processing instruction generated by the identified text. Optionally, the data processing instruction is generated after identifying the text input based on the text input control.

[0066] For example, in Figure 3 The text input into the text input control shown in 305 may be count, and the detection terminal may generate a number counting instruction after recognizing the text; the text input into the text input control may also be count type=field_empty, and a number counting instruction of type may be generated after recognizing the text; the text input into the text input control may also be reHandle, and a reprocessing instruction may be generated after recognizing the text; the text input into the text input control may also be reHandle type=field_empty, and a reprocessing instruction of type may be generated after recognizing the text.

[0067] Specifically, during the data processing process, in order to improve the security of data processing, authorization detection may be performed on the detection terminal that submits the data processing instruction. On this basis, in order to reduce the risk of misoperation of abnormal data and improve the accuracy and reliability of data processing, the authorized detection terminal may be further confirmed for the submitted data processing instruction. In an optional implementation manner provided by this embodiment, data processing is performed on the abnormal data according to the instruction type of the data processing instruction, including: If the instruction type is a text instruction type, checking whether the detection terminal is a detection terminal authorized to process data; If so, send an instruction confirmation message of the configuration instruction confirmation control to the detection terminal, and after obtaining the confirmation instruction submitted by the detection terminal based on the instruction confirmation control, perform data processing on the abnormal data according to the data processing instruction; If not, no action is required.

[0068] Specifically, after the detection terminal obtains a text instruction based on the text input by the text input control, if it is detected that the detection terminal is a detection terminal authorized to perform data processing, a command confirmation message of the configuration command confirmation control can be sent to the detection terminal, and after obtaining the confirmation instruction submitted by the detection terminal, data processing is performed on the abnormal data according to the text instruction obtained by identification. The following describes the process of data processing of abnormal data according to six types of text instructions.

[0069] (a) Number counting instructions In a first optional implementation manner provided by this embodiment, after obtaining the confirmation instruction submitted by the detection terminal based on the instruction confirmation control, data processing is performed on the abnormal data according to the data processing instruction, including: If the data processing instruction is a number counting instruction, the number of abnormal data is obtained; Write the data number into the number assembly template to assemble the message to obtain the number message; Call the message sending platform interface to send the number message to the detection terminal.

[0070] For example, if the detection terminal is Figure 3 The text input in the text input control shown in 305 is count, and the text is recognized to obtain a number counting instruction. After the detection determines that the detection terminal is authorized, first send an instruction confirmation message of the configuration instruction confirmation control to the detection terminal, and after obtaining the confirmation instruction submitted by the detection terminal, query the data number of all abnormal data based on the number counting instruction, write the data number into the number assembly template for message assembly to obtain the number message, and then call the message sending platform interface to send the number message to the detection terminal to inform the data number of the abnormal data.

[0071] (b) Type number counting instructions In a second optional implementation manner provided by this embodiment, data processing is performed on abnormal data according to the data processing instruction, including: If the data processing instruction is a type number counting instruction based on the data exception type, the type data number of the target exception data corresponding to the target data exception type is obtained; Write the type data number into the number assembly template to assemble the message to obtain the type number message; Call the message sending platform interface to send the type number message to the detection terminal.

[0072] For example, if the text input into the detection terminal is count type=field_empty, the text is identified to obtain the type number counting instruction, and the corresponding data anomaly type is determined to be field empty. Then, all abnormal data with empty fields can be queried, and the number of abnormal data with empty fields can be counted. Based on the number of data, the message is assembled to obtain the type number message, and then the message sending platform interface is called to send the type number message to the detection terminal to inform the number of abnormal data with empty fields.

[0073] (c) Reprocessing instructions In a third optional implementation manner provided by this embodiment, data processing is performed on abnormal data according to the data processing instruction, including: If the data processing instruction is a reprocessing instruction, the abnormal data is returned to the message queue for recalculation based on the reprocessing instruction.

[0074] For example, if the text input by the detection terminal is reHandle, the text is identified to obtain a reprocessing instruction, and all abnormal data can be queried and obtained, and all the obtained abnormal data are re-delivered to the Kafka queue so that the Flink big data processing system can reprocess each abnormal data.

[0075] (d) Type reprocessing instructions In a fourth optional implementation manner provided by this embodiment, data processing is performed on abnormal data according to the data processing instruction, including: If the data processing instruction is a type reprocessing instruction, the target abnormal data corresponding to the target data abnormal type is returned to the message queue for recalculation based on the type reprocessing instruction.

[0076] For example, if the text input by the detection terminal is reHandle type=field_empty, the text is identified to obtain the type reprocessing instruction, and the corresponding data exception type is determined to be field empty. Then, all abnormal data with empty fields can be queried and obtained, and the obtained abnormal data with empty fields can be re-delivered to the Kafka queue, so that the Flink big data processing system can reprocess the abnormal data with empty fields.

[0077] (e) Data deletion order In a fifth optional implementation manner provided by this embodiment, data processing is performed on abnormal data according to the data processing instruction, including: If the data processing instruction is a data deletion instruction, the abnormal data is deleted based on the data deletion instruction.

[0078] Based on the type reprocessing instruction, the target abnormal data corresponding to the target data abnormal type is returned to the message queue for recalculation For example, if the text input into the detection terminal is "delete", the text is identified to obtain a data deletion instruction, and then all abnormal data can be queried and obtained, and all the obtained abnormal data can be deleted.

[0079] (f) Type data deletion instruction In a sixth optional implementation manner provided by this embodiment, data processing is performed on abnormal data according to the data processing instruction, including: If the data processing instruction is a type data deletion instruction, the target abnormal data corresponding to the target data abnormal type is queried and deleted based on the type data deletion instruction.

[0080] For example, if the text input into the detection terminal is delete type=field_empty, the text is identified to obtain the type data deletion instruction, and the corresponding data anomaly type is determined to be field empty. Then, all abnormal data with empty fields can be queried and obtained, and the obtained abnormal data with empty fields can be deleted.

[0081] In addition, in addition to the above-mentioned data processing of abnormal data according to the six types of text instructions, other texts can be input based on the text input control, and the corresponding text instructions can be obtained after the text is recognized. Accordingly, the abnormal data can be processed accordingly according to the text instructions, which will not be described one by one in this embodiment.

[0082] In actual applications, in order to timely discover changes in the number of abnormal data and avoid data errors caused by excessive accumulation of abnormal data, you can also configure a scheduled task to regularly query the total number of abnormal data in the database to ensure that the number of abnormal data is always within a controllable range. When the number of abnormal data reaches the preset threshold, a voice notification is issued so that the detection personnel can receive the notification of excessive abnormal data in time, so as to process the abnormal data in time.

[0083] To summarize, the one or more abnormal data processing methods provided in this embodiment, for the abnormal data obtained by data detection, first parse the abnormal data to obtain log data and data anomaly type, and provide a basis for subsequent data processing by determining the data anomaly type of the abnormal data. Secondly, write the abnormal data into the assembly template corresponding to the data anomaly type to assemble the message to obtain the data processing message. By assembling the message based on the assembly template, the standardization and consistency of the data processing message are ensured. Then, the message sending platform interface is called to send the data processing message to the detection terminal. If the data processing instruction for the abnormal data returned by the interface call is obtained, the abnormal data is processed according to the instruction type of the data processing instruction. By allowing the detection terminal to select the appropriate data processing method according to the specific situation, the controllability and flexibility of the data processing process are improved.

[0084] The following combination Figure 4 , taking the application of the abnormal data processing method provided in this embodiment in the first abnormal data processing scenario as an example, the abnormal data processing method provided in this embodiment is further described. Figure 4 The abnormal data processing method applied to the first abnormal data processing scenario specifically includes steps S402 to S442.

[0085] Step S402: Acquire business log data and perform data detection on the business log data to obtain abnormal data.

[0086] Step S404: parse the abnormal data to obtain business log data and data abnormality type.

[0087] Step S406, writing the abnormal data into the assembly template corresponding to the data abnormality type to perform message assembly to obtain a data processing message.

[0088] Step S408, detecting whether the frequency of sending data processing messages within a preset time exceeds the frequency threshold; if not, executing step S410, and if so, suspending sending data processing messages.

[0089] Step S410: If the frequency threshold is not exceeded, the message sending platform interface is called to send the data processing message to the detection terminal.

[0090] Step S418: If the detail acquisition instruction returned by the interface call is obtained, the detail data of the abnormal data is obtained according to the detail acquisition instruction.

[0091] Step S420, writing the detail data into the assembly template to perform message assembly to obtain a detail data message.

[0092] Step S422: calling the message sending platform interface to send the detailed data message to the detection terminal.

[0093] Step S430: If a data deletion instruction returned by the interface call is obtained, the abnormal data is deleted according to the data deletion instruction.

[0094] Step S436: If a reprocessing instruction returned by the interface call is obtained, the abnormal data is returned to the Kafka queue for reprocessing according to the reprocessing instruction.

[0095] Step S442: If a truncation instruction returned by the interface call is obtained, the abnormal data is truncated according to the truncation instruction.

[0096] It should be noted that the above steps S418 to S422 correspond to data processing of abnormal data according to the detail acquisition instruction, step S430 corresponds to data processing of abnormal data according to the data deletion instruction, step S436 corresponds to data processing of abnormal data according to the reprocessing instruction, and step S442 corresponds to data processing of abnormal data according to the truncation instruction; the data processing message is configured with at least one data processing control, and the data processing control can be any one or more of the details control, the deletion control, the reprocessing control and / or the truncation control. According to different data processing controls triggered by the detection terminal, the generated data processing instructions are also different. Correspondingly, the process of data processing of abnormal data according to the acquired data processing instructions is also different. Therefore, according to triggering any one or more of the details control, the deletion control, the reprocessing control and / or the truncation control, after executing steps S402 to S410, any one or more of the above steps S418 to S422, step S430, step S436, and step S442 may be executed.

[0097] It should be noted that any one of steps S402 to S410, steps S418 to S422, step S430, step S436, and step S442, or a combination of any multiple steps, can select any one of steps S102 to S108 or any multiple steps to be combined into a new implementation method according to the needs of implementation deployment; and any one or multiple technical features in the technical scheme composed of steps S402 to S410, steps S418 to S422, steps S430, step S436, and step S442 can also select any one or multiple technical features in the technical scheme composed of steps S102 to S108 to be combined into a new implementation method according to the needs of actual deployment, or select technical features in one or more optional implementation methods provided by steps S102 to S108 to be combined into a new implementation method, which will not be described one by one here.

[0098] The following combination Figure 5 , taking the application of the abnormal data processing method provided in this embodiment in the second abnormal data processing scenario as an example, the abnormal data processing method provided in this embodiment is further described. Figure 5 The abnormal data processing method applied to the second abnormal data processing scenario specifically includes steps S502 to S532.

[0099] Step S502: Acquire business log data and perform data detection on the business log data to obtain abnormal data.

[0100] Step S504: parse the abnormal data to obtain business log data and data abnormality type.

[0101] Step S506, writing the abnormal data into the assembly template corresponding to the data abnormality type to perform message assembly to obtain a data processing message.

[0102] Step S508, detecting whether the frequency of sending data processing messages within a preset time exceeds the frequency threshold; if not, executing step S510, and if so, suspending sending data processing messages.

[0103] Step S510: If the frequency threshold is not exceeded, the message sending platform interface is called to send the data processing message to the detection terminal.

[0104] Step S520: Obtain a data deletion instruction for abnormal data returned by the interface call.

[0105] Step S522: Detect whether the detection terminal is a detection terminal authorized to process data.

[0106] Step S524: If the detection terminal is authorized, the message sending platform interface is called to send an instruction confirmation message of the configuration instruction confirmation control to the detection terminal.

[0107] Step S532, obtain a confirmation instruction, and delete all abnormal data according to the data deletion instruction.

[0108] It should be noted that any one of steps S502 to step S510, steps S520 to step S524, and step S532, or any combination of multiple steps, can select any one of steps S102 to step S108 or any combination of multiple steps to form a new implementation method according to the needs of implementation deployment; and any one or multiple technical features in the technical scheme composed of steps S502 to step S510, steps S520 to step S524, and step S532 can also select any one or multiple technical features in the technical scheme composed of steps S102 to step S108 to form a new implementation method according to the needs of actual deployment, or select technical features in one or more optional implementation methods provided by steps S102 to step S108 to combine into a new implementation method, which will not be repeated here one by one.

[0109] Another abnormal data processing method embodiment provided in this specification is: Reference Figure 6 , which shows a processing flow chart of another abnormal data processing method provided by this embodiment. The another abnormal data processing method provided by this embodiment specifically includes the following steps S602 to S606.

[0110] Step S602, obtaining and displaying a data processing message of the abnormal data; the data processing message is obtained by writing the abnormal data into an assembly template corresponding to the data abnormality type for message assembly.

[0111] In this embodiment, abnormal data refers to business log data that does not conform to preset specifications, wherein the preset specifications may be predefined specifications used to detect whether the business log data conforms to the preset specifications, and the preset specifications may include data types, field lengths, required fields, data formats, and / or other constraint specifications for business log data. Accordingly, abnormal data may be business log data whose field value type does not conform to the preset data type, or business log data whose field value exceeds the preset field length, or business log data whose field value is empty, or business log data that does not conform to other constraint specifications. Optionally, abnormal data is obtained after data detection.

[0112] In practical applications, abnormal data is obtained by performing data detection on business log data. On the one hand, data detection can ensure the integrity and consistency of business log data and reduce errors and failures caused by business log data. On the other hand, abnormal data in the business log data obtained through data detection can be identified and recorded to provide a data basis for subsequent data processing. In an optional implementation manner provided in this embodiment, data detection is performed on business log data before parsing the abnormal data to obtain business log data and data abnormality types. Specifically, data detection includes: Obtain business log data, and generate a data object for business log data mapping based on a preset mapping relationship; Check whether the data object meets the preset specifications; If not, perform data assembly based on the business log data and data anomaly type to obtain abnormal data; If so, store and process the business log data.

[0113] The data exception type may include a field value type mismatch, a field value that is too long, and / or a field value that is empty. In addition, the data exception type may also include a data exception type that does not meet other field constraint standards.

[0114] For example, Figure 2As shown, the Flink big data processing system can read business log data from the Kafka (distributed stream processing platform) queue, filter, clean and / or otherwise pre-process the read business log data, and in order to correctly store the pre-processed business log data in the database, generate data objects mapped to the business log data based on the preset mapping relationship, and perform compliance detection on the data objects to be stored in the warehouse to determine whether the data objects can be normally stored in the warehouse. If it is detected that the data object meets the preset specifications, the data object is stored in the warehouse; if it is detected that the data object does not meet the preset specifications, the metadata of the business log data is obtained, and the business log data and the metadata of the business log data are assembled to obtain abnormal data. The metadata of the business log data may include identifying the data abnormality type of the obtained business log data, the data business type of the business log data and / or the Kafka queue name corresponding to the business log data, and the obtained abnormal data is sent to the abnormal data processing system for subsequent data processing.

[0115] Specifically, after obtaining abnormal data based on data detection on the business log data, the abnormal data processing system can parse the abnormal data to obtain business log data and metadata of the business log data, wherein the metadata includes identifying the data abnormality type of the obtained business log data, that is: the abnormal data processing system parses the abnormal data to obtain business log data and the data abnormality type; in addition, the metadata may also include the data business type of the business log data and / or the Kafka queue name corresponding to the business log data.

[0116] The data anomaly type may include a field value type mismatch, a field value that is too long, and / or a field value that is empty. In addition, the data anomaly type may also include a data anomaly type that does not meet other field constraint standards.

[0117] Furthermore, after parsing the abnormal data to obtain business log data and data anomaly types, in order to improve the pertinence and effectiveness of data processing, and at the same time to enable detection personnel to quickly understand the data anomaly types of abnormal data and thus perform corresponding data processing on the abnormal data, the abnormal data processing system can write the abnormal data into an assembly template corresponding to the data anomaly type to perform message assembly to obtain a data processing message.

[0118] Optionally, the data processing message is obtained by writing the abnormal data into an assembly template corresponding to the data abnormal type for message assembly.

[0119] The assembly template refers to a predefined template for assembling exception data, which is used to assemble the exception data into a data processing message. Correspondingly, the assembly template corresponding to the data exception type refers to a predefined template for exception data of a specific data exception type, which is used to assemble the exception data under the data exception type into a data processing message.

[0120] For example, taking the data exception type as field value being too long as an example, the abnormal data processing system can write the abnormal data of field value being too long into an assembly template for processing abnormal data of field value being too long to perform message assembly to obtain a data processing message, and the data processing message can display the specific content of data exception type as field value being too long and field value being too long; Figure 3 The data processing message shown may display a data exception type: field value is too long.

[0121] In actual applications, in order to improve the efficiency and speed of managing and responding to abnormal data, a processing module specifically used for sending data processing messages can be configured on the message sending platform. For example, a chat robot can be applied for on the message sending platform to send data processing messages. At the same time, in order to enhance the security and reliability of the data processing message transmission process, the chat robot can be audited and authenticated to ensure that only authorized chat robots can send data processing messages and prevent unauthorized chat robots from sending malicious messages. Furthermore, an assembly template can be preset for the chat robot, and the assembly template can be configured with corresponding data processing controls. Additional data processing controls can also be configured in the assembly template according to different abnormal data types to improve the flexibility of data processing.

[0122] During specific implementation, after the exception data is written into the assembly template corresponding to the data exception type for message assembly to obtain the data processing message, in order to achieve timely notification of the data processing message and improve the efficiency and speed of managing and responding to exception data, the exception data processing system can call the message sending platform interface to send and process the data processing message; accordingly, the data processing message can be obtained and displayed.

[0123] For example, a data processing message may be Figure 3 As shown, the data processing message may be configured with a response control, the response control includes at least one data processing control and / or a text input control, and the data processing control may be Figure 3 In the details control shown in 301, the delete control shown in 302, the reprocessing control shown in 303 and / or the truncation control shown in 304, the text input control can be as shown in FIG. Figure 3 As shown in 305.

[0124] In the specific execution process, in order to prevent too many data processing messages from being sent within a certain period of time and avoid overloading the message sending platform and the detection terminal, the sending frequency of the data processing message can be controlled so that the load of the message sending platform and the detection terminal is within a controllable range; in an optional implementation manner provided by this embodiment, before calling the message sending platform interface to send the data processing message to the detection terminal, the abnormal data processing system can detect the frequency threshold of sending the data processing message; specifically, detecting the frequency threshold of sending the data processing message includes: Preset the frequency threshold for calling the message sending platform interface to send data processing messages; Detecting whether the frequency of sending data processing messages within a preset time exceeds a frequency threshold; If yes, suspend sending the data processing message; If not, call the message sending platform interface to send the data processing message.

[0125] Specifically, the abnormal data processing system can determine the frequency threshold for sending data processing messages according to the load capacity of the message sending platform. After each data processing message is obtained, it can first check the number of data processing messages that have been sent in the past certain period of time, and detect whether the frequency of sending data processing messages exceeds the frequency threshold. If so, it indicates that the number of data processing messages sent has reached the frequency threshold, and the process of sending the data processing message is suspended until the count of the frequency of sending the data processing message is reset; if not, it indicates that the number of data processing messages sent has not reached the frequency threshold, and the message sending platform interface can be called to send and process the data processing message.

[0126] For example, Figure 2 As shown, the abnormal data processing system can call the 5G message sending platform to send and process the data processing message (downlink message).

[0127] In actual applications, after the exception data processing system parses the exception data to obtain the log data and the metadata of the log data, it can store the log data and the metadata of the log data in the database and obtain the corresponding primary key identifier; and, after the exception data processing system calls the message sending platform interface to send and process the data processing message, it can obtain the message identifier returned by the interface call, and associate the message identifier with the primary key identifier, so that the corresponding exception data can be queried in the subsequent data processing process; for example, the message identifier returned by the interface call can be used as the key, and the primary key identifier generated when the exception data is entered into the database can be used as the key value and stored in Redis.

[0128] Step S504, generating a data processing instruction after the response control configured in the data processing message is triggered.

[0129] As described above, the response control includes at least one data processing control and / or a text input control. After triggering at least one data processing control, the inspector can generate a data processing instruction corresponding to the data processing control, or, after inputting text in the text input control, the inspector can generate a corresponding data processing instruction based on the recognized text.

[0130] In specific implementation, if the response control configured by the data processing message is triggered, a corresponding data processing instruction can be generated based on the type of the triggered response control; optionally, the data processing instruction includes a control instruction and / or a text instruction; accordingly, triggering at least one data processing control can generate a control instruction, and the instruction type corresponding to the control instruction is a control instruction type, and triggering a text input control can generate a text instruction, and the instruction type corresponding to the text instruction is a text instruction type.

[0131] Among them, control instructions refer to data processing instructions generated by the detection terminal through the data processing control configured by triggering the data processing message, and each data processing control can correspond to a data processing method; text instructions refer to data processing instructions obtained by the detection terminal through identifying the text entered in the text input control configured based on the data processing message.

[0132] Specifically, when the triggered response control is at least one data processing control, a control instruction corresponding to the data processing control can be generated, and when the triggered response control is a text input control, a corresponding text instruction can be generated based on the text input into the recognized text input control. The following describes the process of generating data processing instructions by triggering two types of response controls.

[0133] (1) Trigger at least one data processing control During the specific execution process, in order to improve the convenience and accuracy of the detection personnel submitting data processing instructions, the data processing message can be configured with a data processing control, and the corresponding data processing instruction can be generated after the data processing control is triggered; in an optional implementation provided by this embodiment, the data processing instruction is generated after the response control configured in the data processing message is triggered, including: After at least one data processing control configured in the data processing message is triggered, a corresponding data processing instruction is generated.

[0134] For example, Figure 3 As shown, the data processing message may be configured with at least one data processing control, which may be Figure 3 The details control shown in 301, the delete control shown in 302, the reprocessing control shown in 303 and / or the truncation control shown in 304.

[0135] Specifically, different data processing controls configured by triggering data processing messages generate different data processing instructions. Accordingly, the abnormal data processing system can perform different data processing on abnormal data based on different data processing instructions. The following describes the process of triggering four types of data processing controls to generate corresponding data processing instructions.

[0136] (a) Trigger the details control to generate details acquisition instructions In a first optional implementation manner provided by this embodiment, after a response control configured in a data processing message is triggered, a data processing instruction is generated, including: Generates a detail acquisition instruction after the detail control configured in the data processing message is triggered.

[0137] For example, if the trigger Figure 3 The detail control shown in 301 can generate a detail acquisition instruction for abnormal data.

[0138] (b) Trigger the deletion control to generate data deletion instructions In a second optional implementation manner provided by this embodiment, after a response control configured in a data processing message is triggered, a data processing instruction is generated, including: Generates a data deletion instruction after the deletion control configured in the data processing message is triggered.

[0139] For example, if the trigger Figure 3 The delete control shown in 302 can generate a data deletion instruction for abnormal data.

[0140] (c) Trigger the reprocessing control to generate reprocessing instructions In a third optional implementation manner provided by this embodiment, after the response control configured in the data processing message is triggered, a data processing instruction is generated, including: Generates a reprocessing instruction after the reprocessing control of the data processing message configuration is triggered.

[0141] For example, if the trigger Figure 3 The reprocessing control shown in 303 can generate a reprocessing instruction for abnormal data.

[0142] (d) Trigger the truncation control to generate a truncation instruction In a fourth optional implementation manner provided by this embodiment, after the response control configured in the data processing message is triggered, a data processing instruction is generated, including: Generates a truncation instruction after the truncation control configured in the data processing message is triggered.

[0143] For example, if the trigger Figure 3 The truncation control shown in 304 can generate a truncation instruction for abnormal data.

[0144] In addition, in addition to triggering the four data processing controls mentioned above to generate corresponding data processing instructions, additional data processing controls can be configured in the data processing message according to actual needs. Accordingly, after triggering other data processing controls, corresponding data processing instructions can also be generated; for example, if the abnormal data processing system detects abnormal data in which a field is empty, an assignment control can be added to the data processing message. If the assignment control is triggered, an assignment instruction for the abnormal data can be generated.

[0145] (2) Triggering the text input control In the specific execution process, in order to improve the flexibility of data processing to cope with diverse and complex data processing requirements, and to realize batch processing of data based on data processing instructions, thereby improving data processing efficiency, the data processing message can be configured with a text input control, and a corresponding text instruction can be generated after the text input control is triggered; in an optional implementation manner provided by this embodiment, a data processing instruction is generated after a response control configured in the data processing message is triggered, including: The data processing instruction is generated after identifying the text input into the text input control configured based on the data processing message.

[0146] For example, Figure 3 As shown, data processing messages can be configured with Figure 3 If the text input control is triggered, the text of the data processing instruction to be executed can be entered in the text input control; for example, Figure 3 The text input into the text input control shown in 305 may be count, and after the text is recognized, a number counting instruction may be generated; the text input into the text input control may also be count type=field_empty, and after the text is recognized, a type number counting instruction may be generated; the text input into the text input control may also be reHandle, and after the text is recognized, a reprocessing instruction may be generated; the text input into the text input control may also be reHandle type=field_empty, and after the text is recognized, a type reprocessing instruction may be generated.

[0147] Step S606, performing the sending process of the data processing instruction.

[0148] In specific implementation, after generating the data processing instruction, the generated data processing instruction can be sent for processing so that the abnormal data processing system processes the abnormal data based on the data processing instruction; optionally, the data processing instruction is used to process the abnormal data based on the instruction type.

[0149] For example, Figure 2As shown, after generating the corresponding data processing instruction by triggering the response control of the data processing message configuration, the 5G message sending platform can be called to send and process the data processing instruction, or the 5G message sending platform can be called to send and process the uplink message containing the data processing instruction; accordingly, the abnormal data processing system can obtain the data processing instruction returned by the interface call.

[0150] In the specific execution process, after the data processing instruction is sent and processed, the abnormal data processing system can process the abnormal data based on the instruction type of the acquired data processing instruction. The following specifically describes two processes of processing the abnormal data based on the instruction type of the data processing instruction.

[0151] (1) Control instruction type As described above, after the data processing control configured by the data processing message is triggered, a control instruction may be generated. Accordingly, data processing may be performed on the abnormal data based on the control instruction. In an optional implementation manner provided by this embodiment, data processing is performed on the abnormal data according to the instruction type of the data processing instruction, including: If the instruction type is a control instruction type, data processing is performed on the abnormal data based on the data processing method corresponding to the data processing instruction.

[0152] Specifically, triggering the above four data processing controls can generate corresponding data processing instructions, and accordingly, the abnormal data processing system can process the abnormal data based on the data processing method corresponding to the data processing instruction. The following describes the process of processing the abnormal data based on the data processing methods corresponding to the four control instructions.

[0153] (a) Details acquisition instruction In a first optional implementation manner provided by this embodiment, data processing is performed on abnormal data based on a data processing method corresponding to a data processing instruction, including: If a detail acquisition instruction is obtained, the detail data of the abnormal data is acquired and returned based on the detail acquisition instruction.

[0154] For example, after sending and processing the details acquisition instruction, the exception data processing system can obtain the details acquisition instruction accordingly. In response to the details acquisition instruction, the details data of the associated exception data can be obtained through the primary key identifier, and the obtained details data can be written into the corresponding assembly template for message assembly to obtain the details data message, and the message platform interface is called to send the details data message. Accordingly, the details data message returned by the interface call can be obtained, and the details data of the exception data can be displayed.

[0155] (b) Data deletion instruction In a second optional implementation manner provided by this embodiment, data processing is performed on abnormal data based on a data processing method corresponding to a data processing instruction, including: If a data deletion instruction is obtained, the abnormal data is deleted based on the data deletion instruction.

[0156] For example, after sending and processing the data deletion instruction, the abnormal data processing system can obtain the data deletion instruction accordingly, and in response to the data deletion instruction, can obtain the associated abnormal data through the primary key identifier, and delete the obtained abnormal data.

[0157] (c) Reprocessing instructions In a third optional implementation manner provided by this embodiment, data processing is performed on abnormal data based on a data processing method corresponding to a data processing instruction, including: If a reprocessing instruction is obtained, the abnormal data is returned to the message queue for recalculation based on the reprocessing instruction.

[0158] For example, after the reprocessing instruction is sent and processed, the exception data processing system can obtain the reprocessing instruction accordingly, and in response to the reprocessing instruction, the associated exception data can be obtained through the primary key identifier, and the obtained exception data can be re-delivered to the Kafka queue so that the Flink big data processing system can reprocess the exception data; Figure 2 As shown, the abnormal data can be re-delivered to the Kafka queue, and the Flink big data processing system can read the abnormal data from the Kafka queue and re-clean and calculate the abnormal data.

[0159] (d) Truncation Instructions In a fourth optional implementation manner provided by this embodiment, data processing is performed on abnormal data based on a data processing method corresponding to a data processing instruction, including: If a truncation instruction is obtained, the fields in the abnormal data that exceed the preset field threshold are truncated according to the preset field threshold based on the truncation instruction.

[0160] For example, after the truncation instruction is sent and processed, the exception data processing system can obtain the truncation instruction accordingly. In response to the truncation instruction, the associated exception data can be obtained through the primary key identifier, the overlong field content can be truncated according to the preset field threshold, and the processed exception data can be re-stored.

[0161] In addition, in addition to the data processing methods corresponding to the above four data processing instructions, if the additional data processing controls added by the data processing message configuration are triggered, the corresponding data processing instructions can be generated and sent. Accordingly, the abnormal data processing system can perform corresponding data processing on the abnormal data based on the data processing instructions.

[0162] (2) Text instruction type As described above, after the text input control configured for the data processing message is triggered, text can be input, and after the text is recognized, a corresponding text instruction can be generated. Accordingly, the abnormal data processing system can process the abnormal data based on the text instruction.

[0163] During the specific execution process, in order to improve the security of data processing, after the data processing instruction is sent and processed, the abnormal data processing system can perform authorization detection on the detection terminal that sends the data processing instruction. On this basis, if the detection terminal is authorized, in order to reduce the risk of misoperation of abnormal data and improve the accuracy and reliability of data processing, the abnormal data processing system can send an instruction confirmation message to enable the detection terminal to further confirm the input text. In an optional implementation manner provided by this embodiment, data processing is performed on the abnormal data according to the instruction type of the data processing instruction, including: If the instruction type is a text instruction type, checking whether the detection terminal is a detection terminal authorized to process data; If so, send an instruction confirmation message of the configuration instruction confirmation control to the detection terminal, and after obtaining the confirmation instruction submitted by the detection terminal based on the instruction confirmation control, perform data processing on the abnormal data according to the data processing instruction; If not, no action is required.

[0164] Accordingly, the input text can be confirmed based on the instruction confirmation message sent by the abnormal data processing system to reduce the risk of text input errors or misoperations; in an optional implementation manner provided by this embodiment, after the data processing instruction is sent, it also includes: Receive and display the command confirmation message of the configuration command confirmation control; After the instruction confirmation control is triggered, a confirmation instruction is generated and submitted.

[0165] Specifically, after confirming the text input based on the text input control and submitting the confirmation instruction, the abnormal data processing system can start data processing on the abnormal data based on the confirmation instruction, and the data processing on the abnormal data can be performed according to the text instruction generated by the recognized text. The following describes the process of the abnormal data processing system processing the abnormal data based on six text instructions.

[0166] (a) Number counting instructions In a first optional implementation manner provided by this embodiment, after obtaining the confirmation instruction submitted by the detection terminal based on the instruction confirmation control, data processing is performed on the abnormal data according to the data processing instruction, including: If the data processing instruction is a number counting instruction, the number of abnormal data is obtained; Write the data number into the number assembly template to assemble the message to obtain the number message; Call the message sending platform interface to send the number message to the detection terminal.

[0167] For example, after identifying the count input based on the text input control, a number counting instruction is generated, and the number counting instruction is sent and processed. Accordingly, the abnormal data processing system can obtain the number counting instruction. After further obtaining the confirmation instruction for the number counting instruction, in response to the number counting instruction, the data number of all abnormal data can be queried, the data number is written into the number assembly template for message assembly to obtain the number message, and then the message sending platform interface is called to send and process the number message. Accordingly, the number message sent by the abnormal data processing system can be obtained, and the data number of the abnormal data can be displayed.

[0168] (b) Type number counting instructions In a second optional implementation manner provided by this embodiment, after obtaining the confirmation instruction submitted by the detection terminal based on the instruction confirmation control, data processing is performed on the abnormal data according to the data processing instruction, including: If the data processing instruction is a type number counting instruction based on the data exception type, the type data number of the target exception data corresponding to the target data exception type is obtained; Write the type data number into the number assembly template to assemble the message to obtain the type number message; Call the message sending platform interface to send the type number message to the detection terminal.

[0169] For example, after identifying count type=field_empty input based on a text input control, a type number counting instruction is generated, and the type number counting instruction is sent and processed. Accordingly, the abnormal data processing system can respond to the type number counting instruction, query all abnormal data with empty fields, and count the number of data of abnormal data with empty fields, assemble messages based on the number of data to obtain a type number message, and then call the message sending platform interface to send and process the type number message. Accordingly, the type number message sent by the abnormal data processing system can be obtained, and The number of abnormal data with empty fields is displayed.

[0170] (c) Reprocessing instructions In a third optional implementation manner provided by this embodiment, after obtaining the confirmation instruction submitted by the detection terminal based on the instruction confirmation control, data processing is performed on the abnormal data according to the data processing instruction, including: If the data processing instruction is a reprocessing instruction, the abnormal data is returned to the message queue for recalculation based on the reprocessing instruction.

[0171] For example, after identifying the reHandle input based on the text input control, a reprocessing instruction is generated, and the reprocessing instruction is sent for processing. Accordingly, the exception data processing system can respond to the reprocessing instruction, query and obtain all exception data, and re-deliver all the obtained exception data to the Kafka queue so that the Flink big data processing system can reprocess each exception data.

[0172] (d) Type reprocessing instructions In a fourth optional implementation manner provided by this embodiment, after obtaining the confirmation instruction submitted by the detection terminal based on the instruction confirmation control, data processing is performed on the abnormal data according to the data processing instruction, including: If the data processing instruction is a type reprocessing instruction, the target abnormal data corresponding to the target data abnormal type is returned to the message queue for recalculation based on the type reprocessing instruction.

[0173] For example, after identifying the reHandle type=field_empty input based on the text input control, a type reprocessing instruction is generated, and the type reprocessing instruction is sent for processing. Accordingly, the exception data processing system can respond to the type reprocessing instruction, query and obtain all the exception data with empty fields, and re-deliver the obtained exception data with empty fields to the Kafka queue, so that the Flink big data processing system can reprocess the exception data with empty fields.

[0174] (e) Data deletion order In a fifth optional implementation manner provided by this embodiment, after obtaining the confirmation instruction submitted by the detection terminal based on the instruction confirmation control, data processing is performed on the abnormal data according to the data processing instruction, including: If the data processing instruction is a data deletion instruction, the abnormal data is deleted based on the data deletion instruction.

[0175] Based on the type reprocessing instruction, the target abnormal data corresponding to the target data abnormal type is returned to the message queue for recalculation For example, after identifying the delete input based on the text input control, a data deletion instruction is generated, and the data deletion instruction is sent and processed. Accordingly, the abnormal data processing system can respond to the data deletion instruction, query and obtain all abnormal data, and delete all the obtained abnormal data.

[0176] (f) Type data deletion instruction In a sixth optional implementation manner provided by this embodiment, after obtaining the confirmation instruction submitted by the detection terminal based on the instruction confirmation control, data processing is performed on the abnormal data according to the data processing instruction, including: If the data processing instruction is a type data deletion instruction, the target abnormal data corresponding to the target data abnormal type is queried and deleted based on the type data deletion instruction.

[0177] For example, after identifying the delete type=field_empty input based on the text input control, a type data deletion instruction is generated, and the type data deletion instruction is sent and processed. Accordingly, the exception data processing system can respond to the type data deletion instruction, query and obtain all the exception data with empty fields, and delete the obtained exception data with empty fields.

[0178] In addition, in addition to the above-mentioned data processing of abnormal data according to the six types of text instructions, other texts can be input based on the text input control, and the corresponding text instructions can be obtained after the text is recognized. Accordingly, the abnormal data processing system can perform corresponding data processing based on other data processing instructions, which will not be described one by one in this embodiment.

[0179] In practical applications, in order to timely discover changes in the number of abnormal data and avoid data errors caused by excessive accumulation of abnormal data, the abnormal data processing system can also configure a scheduled task to regularly query the total number of abnormal data in the database to ensure that the number of abnormal data is always within a controllable range, and send a voice notification when the number of abnormal data reaches a preset threshold. Correspondingly, the voice notification sent by the abnormal data processing system can be received, so that the detection personnel can receive the notification of excessive abnormal data quantity in time, so as to process the abnormal data in time.

[0180] To summarize, the one or more abnormal data processing methods provided in this embodiment obtain and display data processing messages of abnormal data during the data processing process, so that the detection personnel can see the data abnormality type of the abnormal data and make corresponding data processing decisions based on the data abnormality type. After the response control configured in the data processing message is triggered, a data processing instruction is generated, and the generated data processing instruction is sent for processing, so that the abnormal data processing system processes the abnormal data based on the instruction type of the data processing instruction. By generating corresponding data processing instructions based on the operations of the detection personnel, the flexibility and accuracy of the data processing process are improved, and the efficiency of processing abnormal data is also improved.

[0181] The following combination Figure 4 , taking the application of the abnormal data processing method provided in this embodiment in the first abnormal data processing scenario as an example, the abnormal data processing method provided in this embodiment is further described. Figure 4 The abnormal data processing method applied to the first abnormal data processing scenario specifically includes steps S402 to S442.

[0182] Step S412, obtaining and displaying data processing information of abnormal data.

[0183] Step S414: Generate a detail acquisition instruction after the detail control configured in the data processing message is triggered.

[0184] Step S416: calling the message sending platform interface to send and process the details acquisition instruction.

[0185] Step S424, obtaining detailed data message and displaying it.

[0186] Step S426, generating a data deletion instruction after the deletion control configured in the data processing message is triggered.

[0187] Step S428, calling the message sending platform interface to send and process the data deletion instruction.

[0188] Step S432, generating a reprocessing instruction after the reprocessing control configured in the data processing message is triggered.

[0189] Step S434, calling the message sending platform interface to send the reprocessing instruction.

[0190] Step S438, generating a truncation instruction after the truncation control configured in the data processing message is triggered.

[0191] Step S440: calling the message sending platform interface to send and process the truncation instruction.

[0192] It should be noted that, the above steps S412 to S416, step S424 corresponds to generating a detail acquisition instruction and sending it after the detail control of the data processing message configuration is triggered, steps S426 to S428 correspond to generating a data deletion instruction and sending it after the delete control of the data processing message configuration is triggered, steps S432 to S434 correspond to generating a reprocessing instruction and sending it after the reprocessing control of the data processing message configuration is triggered, and steps S438 to S440 correspond to generating a truncation instruction and sending it after the truncation control of the data processing message configuration is triggered. Accordingly, the abnormal data processing system may generate a truncation instruction and send it according to the detail acquisition instruction, the data deletion instruction, the reprocessing instruction and / or The truncation instruction performs data processing on abnormal data; the data processing message is configured with at least one data processing control, which may be any one or more of a details control, a delete control, a reprocessing control and / or a truncation control. Depending on the data processing control that is triggered, the generated data processing instruction may also be different. Therefore, according to triggering any one or more of the details control, the delete control, the reprocessing control and / or the truncation control, after executing steps S412 to S416, any one or more of the above steps S412 to S416 and steps S424, S426 to S428, S432 to S434, and S438 to S440 may be executed.

[0193] It should be noted that any one or a combination of any multiple steps of the above steps S412 to S416, steps S412 to S416 and steps S424, steps S426 to S428, steps S432 to S434, and steps S438 to S440 can be combined into a new implementation method by selecting any one or a combination of any multiple steps of the above steps S602 to S606 according to the needs of implementation and deployment; and steps S412 to S416, steps S412 to S416 and steps Any one or more of the technical features in the technical scheme consisting of S424, steps S426 to S428, steps S432 to S434, and steps S438 to S440, and any one or more of the technical features in the technical scheme consisting of the above steps S602 to S606 can be selected according to the actual deployment needs to combine to form a new implementation method, or the technical features in one or more optional implementation methods provided by steps S602 to S606 can be selected to combine to form a new implementation method, which will not be described one by one here.

[0194] The following combination Figure 5, taking the application of the abnormal data processing method provided in this embodiment in the second abnormal data processing scenario as an example, the abnormal data processing method provided in this embodiment is further described. Figure 5 The abnormal data processing method applied to the second abnormal data processing scenario specifically includes steps S502 to S532.

[0195] Step S512, obtaining and displaying data processing information of abnormal data.

[0196] Step S514, acquiring and identifying the text delete inputted by the text input control configured based on the data processing message.

[0197] Step S516, generating a data deletion instruction based on the recognized text.

[0198] Step S518, calling the message sending platform interface to send the data deletion instruction.

[0199] Step S526: Receive and display the command confirmation message of the configuration command confirmation control returned by the interface call.

[0200] Step S528, generating a confirmation instruction after the instruction confirmation control is triggered.

[0201] Step S530, performing a sending process of a confirmation instruction.

[0202] It should be noted that any one of steps S512 to step S518, and any combination of any multiple steps from step S526 to step S530 can select any one of steps S602 to step S606 or any multiple steps from step S530 to form a new implementation method according to the needs of implementation deployment; and any one or multiple technical features in the technical scheme composed of steps S512 to step S518, and steps S526 to step S530 can also select any one or multiple technical features in the technical scheme composed of steps S602 to step S606 to form a new implementation method according to the needs of actual deployment, or select technical features in one or more optional implementation methods provided by steps S602 to step S606 to form a new implementation method, which will not be repeated here one by one.

[0203] Figure 7 A schematic diagram of an abnormal data processing device provided by an embodiment of the present invention is shown in FIG. Figure 7 As shown, the device comprises: The data analysis module 702 is used to analyze the abnormal data to obtain the business log data and the data abnormality type; A message assembly module 704, configured to write the abnormal data into an assembly template corresponding to the data abnormality type to perform message assembly to obtain a data processing message; A message sending module 706 is used to call a message sending platform interface to send the data processing message to a detection terminal; The data processing module 708 is used to process the abnormal data according to the instruction type of the data processing instruction if a data processing instruction for the abnormal data returned by the interface call is obtained; the data processing instruction is submitted by the detection terminal.

[0204] The present embodiment provides an abnormal data processing device. During the data processing process, the data parsing module 702 is first run to parse the abnormal data to obtain the business log data and the data abnormality type. By determining the data abnormality type of the abnormal data, a basis is provided for subsequent data processing. Then, the message assembling module 704 is run to write the abnormal data into the assembly template corresponding to the data abnormality type to assemble the message to obtain the data processing message. By assembling the message based on the assembly template, the standardization and consistency of the data processing message are ensured. Then, the message sending module 706 is run to call the message sending platform interface to send the data processing message to the detection terminal. Finally, the data processing module 708 is run. If the data processing instruction for the abnormal data returned by the interface call is obtained, the abnormal data is processed according to the instruction type of the data processing instruction. The data processing instruction is submitted by the detection terminal. By allowing the detection terminal to select the appropriate data processing method according to the specific situation, the flexibility of data processing is improved, and the processing efficiency and accuracy of the abnormal data are also improved.

[0205] The abnormal data processing device provided in an embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0206] Figure 8 A schematic diagram of another abnormal data processing device provided by an embodiment of the present invention is shown in FIG. Figure 8 As shown, the device comprises: The message acquisition module 802 is used to acquire and display a data processing message of the abnormal data; the data processing message is obtained by writing the abnormal data into an assembly template corresponding to the data abnormality type and assembling the message; An instruction generation module 804, configured to generate a data processing instruction after the response control configured in the data processing message is triggered; The instruction sending module 806 is used to send the data processing instruction; the data processing instruction is used to perform data processing on the abnormal data based on the instruction type.

[0207] Another abnormal data processing device provided in this embodiment first runs the message acquisition module 802 to obtain the data processing message of the abnormal data and display it, so that the detection personnel can see the data abnormality type of the abnormal data and make corresponding data processing decisions based on the data abnormality type. The data processing message is obtained by assembling the message by writing the abnormal data into the assembly template corresponding to the data abnormality type. Secondly, the instruction generation module 804 is run to generate the data processing instruction after the response control configured in the data processing message is triggered. Finally, the instruction sending module 806 is run to send the data processing instruction. The data processing instruction is used to perform data processing on the abnormal data based on the instruction type. By generating the corresponding data processing instruction based on the operation of the detection personnel, the flexibility and accuracy of the data processing process are improved, and the processing efficiency of the abnormal data is also improved.

[0208] Another abnormal data processing device provided in an embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0209] Furthermore, an embodiment of the present specification also provides an abnormal data processing device, Fig. 9 A schematic diagram of the structure of an abnormal data processing device provided in an embodiment of this specification is shown in FIG. Fig. 9 As shown, the device includes: a memory 901, a processor 902, a bus 903 and a communication interface 904. The memory 901, the processor 902 and the communication interface 904 communicate through the bus 903, and the communication interface 904 may include an input and output interface, which includes but is not limited to a keyboard, a mouse, a display, a microphone, a loudspeaker, etc.

[0210] Fig. 9 In the embodiment, the memory 901 stores computer executable instructions that can be run on the processor 902. When the computer executable instructions are executed by the processor 902, the following process is implemented: Analyze the abnormal data to obtain business log data and data abnormality type; Writing the abnormal data into an assembly template corresponding to the data abnormality type to perform message assembly to obtain a data processing message; Calling a message sending platform interface to send the data processing message to the detection terminal; If a data processing instruction for the abnormal data returned by the interface call is obtained, the abnormal data is processed according to the instruction type of the data processing instruction; the data processing instruction is submitted by the detection terminal.

[0211] The abnormal data processing device provided in this embodiment, through the cooperation of the memory 901, the processor 902, the bus 903 and the communication interface 904, for the abnormal data obtained by data detection, firstly parses the abnormal data to obtain the business log data and the data abnormality type, and provides a basis for subsequent data processing by determining the data abnormality type of the abnormal data. Secondly, the abnormal data is written into the assembly template corresponding to the data abnormality type for message assembly to obtain the data processing message. By assembling the message based on the assembly template, the standardization and consistency of the data processing message are ensured, and then the message sending platform interface is called to send the data processing message to the detection terminal. If the data processing instruction for the abnormal data returned by the interface call is obtained, the abnormal data is processed according to the instruction type of the data processing instruction. By allowing the detection terminal to select the appropriate data processing method according to the specific situation, the flexibility of data processing is improved, and the processing efficiency and accuracy of the abnormal data are also improved.

[0212] The abnormal data processing device provided in an embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0213] Furthermore, an embodiment of the present specification also provides another abnormal data processing device, Fig.10 A schematic diagram of the structure of another abnormal data processing device provided in an embodiment of this specification, such as Fig.10 As shown, the device includes: a memory 1001, a processor 1002, a bus 1003 and a communication interface 1004. The memory 1001, the processor 1002 and the communication interface 1004 communicate via the bus 1003, and the communication interface 1004 may include an input and output interface, which includes but is not limited to a keyboard, a mouse, a display, a microphone, a loudspeaker, etc.

[0214] Fig.10 In the embodiment, the memory 1001 stores computer executable instructions that can be run on the processor 1002, and when the computer executable instructions are executed by the processor 1002, the following process is implemented: Obtain and display a data processing message of the abnormal data; the data processing message is obtained by writing the abnormal data into an assembly template corresponding to the data abnormality type for message assembly; generating a data processing instruction after a response control configured in the data processing message is triggered; The data processing instruction is sent and processed; the data processing instruction is used to perform data processing on the abnormal data based on the instruction type.

[0215] Another abnormal data processing device provided in this embodiment, through the cooperation of the memory 1001, the processor 1002, the bus 1003 and the communication interface 1004, obtains and displays the data processing message of the abnormal data during the data processing process, so that the detection personnel can see the data abnormality type of the abnormal data and make corresponding data processing decisions based on the data abnormality type. After the response control configured in the data processing message is triggered, a data processing instruction is generated, and the generated data processing instruction is sent for processing, so as to process the abnormal data based on the instruction type of the data processing instruction. By generating the corresponding data processing instruction based on the operation of the detection personnel, the flexibility and accuracy of the data processing process are improved, and the processing efficiency of the abnormal data is also improved.

[0216] The abnormal data processing device provided in an embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0217] Furthermore, another embodiment of the present specification provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented: Analyze the abnormal data to obtain business log data and data abnormality type; Writing the abnormal data into an assembly template corresponding to the data abnormality type to perform message assembly to obtain a data processing message; Calling a message sending platform interface to send the data processing message to the detection terminal; If a data processing instruction for the abnormal data returned by the interface call is obtained, the abnormal data is processed according to the instruction type of the data processing instruction; the data processing instruction is submitted by the detection terminal.

[0218] The computer-readable storage medium provided in this embodiment first parses the abnormal data obtained by data detection to obtain business log data and data anomaly types, and provides a basis for subsequent data processing by determining the data anomaly type of the abnormal data. Secondly, the abnormal data is written into an assembly template corresponding to the data anomaly type to assemble the message to obtain a data processing message. By assembling the message based on the assembly template, the standardization and consistency of the data processing message are ensured. Then, the message sending platform interface is called to send the data processing message to the detection terminal. If the data processing instruction for the abnormal data returned by the interface call is obtained, the abnormal data is processed according to the instruction type of the data processing instruction. By allowing the detection terminal to select an appropriate data processing method according to the specific situation, the flexibility of data processing is improved, and the processing efficiency and accuracy of the abnormal data are also improved.

[0219] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0220] The computer-readable storage medium provided in an embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0221] Furthermore, another embodiment of the present specification provides another computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented: Obtain and display a data processing message of the abnormal data; the data processing message is obtained by writing the abnormal data into an assembly template corresponding to the data abnormality type for message assembly; generating a data processing instruction after a response control configured in the data processing message is triggered; The data processing instruction is sent and processed; the data processing instruction is used to perform data processing on the abnormal data based on the instruction type.

[0222] Another computer-readable storage medium provided by the present embodiment obtains and displays a data processing message of abnormal data during data processing, so that detection personnel can see the data anomaly type of the abnormal data and make corresponding data processing decisions based on the data anomaly type. After the response control configured in the data processing message is triggered, a data processing instruction is generated, and the generated data processing instruction is sent for processing, so as to process the abnormal data based on the instruction type of the data processing instruction. By generating corresponding data processing instructions based on the detection personnel's operations, the flexibility and accuracy of the data processing process are improved, and the efficiency of processing abnormal data is also improved.

[0223] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0224] The computer-readable storage medium provided in an embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0225] Furthermore, another embodiment of the present specification also provides a computer program product, which implements the following process when executed by a processor: Analyze the abnormal data to obtain business log data and data abnormality type; Writing the abnormal data into an assembly template corresponding to the data abnormality type to perform message assembly to obtain a data processing message; Calling a message sending platform interface to send the data processing message to the detection terminal; If a data processing instruction for the abnormal data returned by the interface call is obtained, the abnormal data is processed according to the instruction type of the data processing instruction; the data processing instruction is submitted by the detection terminal.

[0226] The computer program product provided in this embodiment, for the abnormal data obtained by data detection, first parses the abnormal data to obtain business log data and data anomaly type, and provides a basis for subsequent data processing by determining the data anomaly type of the abnormal data. Secondly, the abnormal data is written into the assembly template corresponding to the data anomaly type to assemble the message to obtain the data processing message. The message assembly based on the assembly template ensures the standardization and consistency of the data processing message, and then calls the message sending platform interface to send the data processing message to the detection terminal. If the data processing instruction for the abnormal data returned by the interface call is obtained, the abnormal data is processed according to the instruction type of the data processing instruction. By allowing the detection terminal to select the appropriate data processing method according to the specific situation, the flexibility of data processing is improved, and the processing efficiency and accuracy of the abnormal data are also improved.

[0227] The computer program product provided in one embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0228] Furthermore, another embodiment of the present specification provides another computer program product, which implements the following process when executed by a processor: Obtain and display a data processing message of the abnormal data; the data processing message is obtained by writing the abnormal data into an assembly template corresponding to the data abnormality type for message assembly; generating a data processing instruction after a response control configured in the data processing message is triggered; The data processing instruction is sent and processed; the data processing instruction is used to perform data processing on the abnormal data based on the instruction type.

[0229] Another computer program product provided by this embodiment obtains and displays data processing messages of abnormal data during data processing, so that detection personnel can see the data anomaly type of the abnormal data and make corresponding data processing decisions based on the data anomaly type. After the response control configured in the data processing message is triggered, a data processing instruction is generated, and the generated data processing instruction is sent for processing, so as to process the abnormal data based on the instruction type of the data processing instruction. By generating corresponding data processing instructions based on the operation of the detection personnel, the flexibility and accuracy of the data processing process are improved, and the efficiency of processing abnormal data is also improved.

[0230] Another computer program product provided in an embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0231] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0232] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0233] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0234] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0235] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0236] Memory may include non-permanent storage in a computer-readable storage medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable storage medium.

[0237] Computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0238] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0239] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0240] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for processing abnormal data, characterized in that: The method comprises: Analyze the abnormal data to obtain business log data and data abnormality type; Writing the abnormal data into an assembly template corresponding to the data abnormality type to perform message assembly to obtain a data processing message; Calling a message sending platform interface to send the data processing message to the detection terminal; If a data processing instruction for the abnormal data returned by the interface call is obtained, the abnormal data is processed according to the instruction type of the data processing instruction; the data processing instruction is submitted by the detection terminal.

2. The abnormal data processing method according to claim 1, wherein the abnormal data is obtained after data detection; The data detection includes: Acquire business log data, and generate a data object mapped to the business log data based on a preset mapping relationship; Detecting whether the data object complies with a preset specification; If not, data assembly is performed according to the business log data and the data anomaly type to obtain abnormal data.

3. The abnormal data processing method according to claim 1, wherein the processing of the abnormal data according to the instruction type of the data processing instruction comprises: If the instruction type is a control instruction type, data processing is performed on the abnormal data based on a data processing method corresponding to the data processing instruction.

4. According to the abnormal data processing method of claim 3, the data processing message is configured with at least one data processing control; and the data processing instruction is generated after the data processing control is triggered.

5. The abnormal data processing method according to claim 4, wherein the abnormal data is processed based on the data processing method corresponding to the data processing instruction, comprising: If a detail acquisition instruction is obtained, acquiring and returning detail data of the abnormal data based on the detail acquisition instruction; The detail acquisition instruction is generated after the detail control configured by the data processing message is triggered; If a data deletion instruction is obtained, deleting the abnormal data based on the data deletion instruction; If a reprocessing instruction is obtained, returning the abnormal data to the message queue for recalculation based on the reprocessing instruction; If a truncation instruction is obtained, the fields in the abnormal data that exceed the preset field threshold are truncated according to the preset field threshold based on the truncation instruction.

6. The abnormal data processing method according to claim 1, wherein the data processing message is configured with a text input control; and the data processing instruction is generated after recognizing the text input based on the text input control.

7. The abnormal data processing method according to claim 6, wherein the processing of the abnormal data according to the instruction type of the data processing instruction comprises: If the instruction type is a text instruction type, detecting whether the detection terminal is a detection terminal authorized to perform data processing; If so, an instruction confirmation message of the configuration instruction confirmation control is sent to the detection terminal, and after obtaining the confirmation instruction submitted by the detection terminal based on the instruction confirmation control, data processing is performed on the abnormal data according to the data processing instruction.

8. The abnormal data processing method according to claim 6, wherein after obtaining the confirmation instruction submitted by the detection terminal based on the instruction confirmation control, performing data processing on the abnormal data according to the data processing instruction comprises: If the data processing instruction is a type number counting instruction based on the data exception type, obtaining the type data number of the target exception data corresponding to the target data exception type; Writing the type data number into the number assembly template to perform message assembly to obtain a type number message; The message sending platform interface is called to send the type number message to the detection terminal.

9. The abnormal data processing method according to claim 1, after the step of writing the abnormal data into the assembly template corresponding to the data abnormality type to assemble the message to obtain the data processing message is executed, and before the step of calling the message sending platform interface to send the data processing message to the detection terminal is executed, it also includes: Presetting a frequency threshold for calling the message sending platform interface to send the data processing message; Detecting whether the frequency of sending the data processing message within a preset time exceeds the frequency threshold; If yes, suspend sending the data processing message; If not, execute the step of calling the message sending platform interface to send the data processing message to the detection terminal.

10. A method for processing abnormal data, characterized in that: The method comprises: Obtain and display a data processing message of the abnormal data; the data processing message is obtained by writing the abnormal data into an assembly template corresponding to the data abnormality type for message assembly; generating a data processing instruction after a response control configured in the data processing message is triggered; The data processing instruction is sent and processed; the data processing instruction is used to perform data processing on the abnormal data based on the instruction type.

11. The abnormal data processing method according to claim 10, wherein generating a data processing instruction after a response control configured in the data processing message is triggered comprises: generating a corresponding data processing instruction after at least one data processing control configured in the data processing message is triggered; Among them, after the details control of the data processing message configuration is triggered, a details acquisition instruction is generated; after the deletion control of the data processing message configuration is triggered, a data deletion instruction is generated; after the reprocessing control of the data processing message configuration is triggered, a reprocessing instruction is generated; after the truncation control of the data processing message configuration is triggered, a truncation instruction is generated.

12. The abnormal data processing method according to claim 10, wherein generating a data processing instruction after a response control configured in the data processing message is triggered comprises: The data processing instruction is generated after identifying the text input into the text input control configured based on the data processing message.

13. The abnormal data processing method according to claim 12, after executing the step of sending the data processing instruction, further comprising: Receive and display the command confirmation message of the configuration command confirmation control; After the instruction confirmation control is triggered, a confirmation instruction is generated and submitted.

14. An abnormal data processing device, characterized in that: The device comprises: A data analysis module is used to analyze abnormal data to obtain business log data and data abnormality types; A message assembly module, used for writing the abnormal data into an assembly template corresponding to the data abnormality type to perform message assembly to obtain a data processing message; A message sending module, used for calling a message sending platform interface to send the data processing message to a detection terminal; The data processing module is used to process the abnormal data according to the instruction type of the data processing instruction if a data processing instruction for the abnormal data returned by the interface call is obtained; the data processing instruction is submitted by the detection terminal.

15. An abnormal data processing device, characterized in that: The device comprises: A message acquisition module, used to acquire and display data processing messages of abnormal data; the data processing messages are obtained by writing the abnormal data into an assembly template corresponding to the data abnormality type for message assembly; An instruction generation module, used for generating a data processing instruction after the response control configured in the data processing message is triggered; An instruction sending module is used to send the data processing instruction; the data processing instruction is used to perform data processing on the abnormal data based on the instruction type.

16. An abnormal data processing device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer executable instructions, and when the computer executable instructions are executed on the processor, the steps of the method described in any one of claims 1 to 13 can be implemented.

17. A computer-readable storage medium having computer-executable instructions stored therein, characterized in that: When the computer executable instructions are executed by a processor, the steps of the method described in any one of claims 1 to 13 can be implemented.

18. A computer program product, characterized in that The computer program product comprises an abnormal data processing program, and when the abnormal data processing program is executed by a processor, the steps of the method described in any one of claims 1 to 13 can be implemented.

Images