Methods, devices, equipment and storage media for stream data error correction

Abstract

This invention relates to the field of data processing technology and discloses a method, apparatus, device, and storage medium for correcting errors in serial data. The method includes: preprocessing serial header data to be processed to obtain target serial header data; determining non-standard serial header data based on a preset standard header library and the target serial header data; comparing the non-standard serial header data with a preset error correction library to obtain successfully matched header data and unmatched header data; and correcting the non-standard serial header data based on the successfully matched header data and the unmatched header data. By correcting the non-standard serial header data based on the successfully matched header data and the unmatched header data, the accuracy of serial data can be effectively improved, and the format of serial data can be unified.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, equipment, and storage medium for correcting errors in streaming data. Background Technology

[0002] Bank transaction records are a crucial indicator of a customer's business performance, allowing for the assessment of their risk level. These records are typically printed and then scanned to create unformatted files such as PDFs or images. While OCR technology is currently a primary method for recognizing transaction records, it cannot achieve 100% accuracy. This necessitates manual correction after OCR recognition, which relies heavily on experience and suffers from low accuracy and efficiency. Furthermore, the format of transaction records varies significantly across thousands of banks nationwide, and even within the same bank, different channels may yield different results. The same information may correspond to different titles, leading to inconsistent data formats and ultimately causing mismatches in the title fields of data items during data analysis.

[0003] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this invention is to provide a method, apparatus, device, and storage medium for correcting errors in streaming data, aiming to solve the technical problems of low accuracy in correcting streaming data and the inability to standardize the format of streaming data in the prior art.

[0005] To achieve the above objectives, the present invention provides a method for correcting errors in streaming data, the method comprising the following steps:

[0006] Preprocess the header row data to be processed to obtain the target header row data;

[0007] Non-standard sequential header data are determined based on the preset standard header library and the target sequential header data;

[0008] The non-standard serial header data is compared with a preset error correction library to obtain header data that was successfully compared and header data that was not successfully compared.

[0009] The non-standard serial header data is corrected based on the successfully matched header data and the failed matched header data.

[0010] Optionally, the preprocessing of the pipeline header data to be processed to obtain the target pipeline header data includes:

[0011] The serial header data to be processed is identified to obtain character data and blank data;

[0012] The character data and the blank data are respectively processed to remove empty data;

[0013] Remove the blank data after removing null values ​​from the header row data to be processed;

[0014] After the removal is completed, the target serial header row data is generated based on the empty character data.

[0015] Optionally, determining non-standard sequential header line data based on a preset standard header library and the target sequential header line data includes:

[0016] The target serial header data is compared with the preset standard header library to obtain the serial header data that has been successfully matched.

[0017] The successfully matched serial header data is converted to obtain standard serial header data;

[0018] The standard serial header data is removed from the target serial header data to obtain non-standard serial header data.

[0019] Optionally, the step of comparing the non-standard serial header data with a preset error correction library to obtain successfully matched header data and unmatched header data includes:

[0020] Construct a serial header data error correction set based on the non-standard serial header data;

[0021] The corresponding serial title line character data is obtained based on the aforementioned serial title line data correction set;

[0022] The serial header character data is compared with a preset error correction library to obtain successfully matched header data and unmatched header data.

[0023] Optionally, the step of comparing the serial header character data with a preset error correction library to obtain successfully matched header data and unmatched header data includes:

[0024] Obtain the title data length of the serial title line character data;

[0025] When the title data length is the target length, determine whether there is error correction title data with the same length as the title data in the preset error correction library;

[0026] There is no error-correction title data with the same length as the title data in the preset error correction library;

[0027] The length of the title data is calculated by subtracting one from the length of the title data to obtain the length of the first title data;

[0028] There is no error correction title data with the same length as the first title data in the preset error correction library;

[0029] The second title data length is obtained by subtracting two from the title data length.

[0030] There is no error correction title data with the same length as the second title data in the preset error correction library;

[0031] Repeat the above steps to decrease the length of the title data at unit intervals until the length of the reduced title data is the target value, or the length of the reduced title data is less than the minimum title data length in the preset error correction library, or all the character data of the serial title line is in the target position state, and the preset error correction library contains error correction title data with the same length as the preset title data.

[0032] The serial header line character data and the error-correcting header data with the same length as the preset header data are traversed and compared to obtain the header line data that was successfully compared and the header line data that failed to be compared.

[0033] Optionally, the step of correcting the non-standard serial header data based on the successfully matched header data and the failed matched header data includes:

[0034] Extract the header row data that is not in the target position state from the non-standard pipeline header row data;

[0035] Replace the non-standard serial header data with the successfully matched header data, and rewrite the non-target position status of the header data to the target position status;

[0036] Based on the failed comparison header data, the non-target position state of the non-standard serial header data is rewritten to the target position state.

[0037] Optionally, after rewriting the non-target position state of the non-standard serial header data to the target position state based on the failed comparison header data, the method further includes:

[0038] Add the failed comparison header row data to the preset header update library;

[0039] The preset standard title library and preset error correction library are iteratively updated by adding the preset title update library;

[0040] Based on the iteratively updated preset standard title library and preset error correction library, continue to correct errors in other serial title line data.

[0041] Furthermore, to achieve the above objectives, the present invention also proposes a streaming data error correction device, the streaming data error correction device comprising:

[0042] The processing module is used to preprocess the header row data to be processed to obtain the target header row data;

[0043] The determination module is used to determine non-standard serial header data based on a preset standard header library and the target serial header data;

[0044] The comparison module is used to compare the non-standard serial header data with a preset error correction library to obtain header data that was successfully compared and header data that was not successfully compared.

[0045] The error correction module is used to correct the non-standard serial header data based on the successfully matched header data and the failed matched header data.

[0046] Furthermore, to achieve the above objectives, the present invention also proposes a pipelined data error correction device, which includes: a memory, a processor, and a pipelined data error correction program stored in the memory and executable on the processor, wherein the pipelined data error correction program is configured to implement the pipelined data error correction method described above.

[0047] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing a pipelined data error correction program, wherein the pipelined data error correction program, when executed by a processor, implements the pipelined data error correction method as described above.

[0048] The serial data error correction method proposed in this invention preprocesses the serial header data to be processed to obtain target serial header data; determines non-standard serial header data based on a preset standard header library and the target serial header data; compares the non-standard serial header data with a preset error correction library to obtain successfully matched header data and unmatched header data; and corrects the non-standard serial header data based on the successfully matched header data and the unmatched header data. Through this method, the error correction of non-standard serial header data based on the successfully matched header data and the unmatched header data can effectively improve the accuracy of serial data and achieve uniformity in serial data format. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the structure of the pipeline data error correction device in the hardware operating environment involved in the embodiments of the present invention;

[0050] Figure 2 This is a flowchart illustrating the first embodiment of the flow data error correction method of the present invention;

[0051] Figure 3 This is a flowchart illustrating the second embodiment of the flow data error correction method of the present invention;

[0052] Figure 4 This is a flowchart illustrating the third embodiment of the flow data error correction method of the present invention;

[0053] Figure 5 This is a schematic diagram of the functional modules of the first embodiment of the data error correction device of the present invention.

[0054] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0056] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of the pipeline data error correction device in the hardware operating environment involved in the embodiment of the present invention.

[0057] like Figure 1 As shown, the pipelined data error correction device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0058] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the pipeline data error correction device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0059] like Figure 1As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a pipelined data error correction program.

[0060] exist Figure 1 In the illustrated pipeline data correction device, the network interface 1004 is mainly used for data communication with the network integrated platform workstation; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the pipeline data correction device of the present invention can be set in the pipeline data correction device, and the pipeline data correction device calls the pipeline data correction program stored in the memory 1005 through the processor 1001 and executes the pipeline data correction method provided in the embodiment of the present invention.

[0061] Based on the above hardware structure, an embodiment of the streaming data error correction method of the present invention is proposed.

[0062] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the serial data error correction method of the present invention.

[0063] In the first embodiment, the streaming data error correction method includes the following steps:

[0064] Step S10: Preprocess the header row data to be processed to obtain the target header row data.

[0065] It should be noted that the execution subject in this embodiment is a streaming data error correction device, but it can also be other devices that can achieve the same or similar functions, such as a data error correction controller. This embodiment does not limit this, and in this embodiment, a data error correction controller is used as an example for explanation.

[0066] It should be understood that the serial header data to be processed refers to the data that needs to be corrected after being recognized by OCR (Optical Character Recognition) technology. The serial header data to be processed includes all serial header data after recognition and the number of header field data. The target serial header data refers to the serial header data after preprocessing, which includes removing null values ​​and eliminating null values.

[0067] Further, step S10 includes: identifying the serial header data to be processed to obtain character data and blank data; performing blank removal processing on the character data and the blank data respectively; removing the blank data after blank removal from the serial header data to be processed; and generating target serial header data based on the blank character data after blank removal after the removal is completed.

[0068] Understandably, character data refers to data containing non-empty characters, such as "transaction date," while blank data refers to data without non-empty characters, such as "". Specifically, the character data and blank data in the transaction header row data to be processed are identified. After obtaining the character data and blank data, they are processed to remove blanks. For example, the character data "transaction date" becomes "transaction date" after blanking, and the blank data "" becomes "" after blanking. Then, the blank data is removed from the transaction header row data to be processed. The remaining data in the transaction header row data is the character data after blanking, which is used as the target transaction header row data.

[0069] Step S20: Determine non-standard serial header data based on the preset standard header library and the target serial header data.

[0070] It is understood that the preset standard title library refers to a standard title library based on various collected bank statement templates. This preset standard title library is applicable to all types of bank statement data. The preset standard title library records the mapping relationship between the original title names and the standard titles. An example of the data recorded in the preset standard title library is shown in Table 1.

[0071] Standard title Original title Transaction date Transaction date Transaction date date Transaction date time Transaction date Trading day Transaction amount Transaction amount Transaction amount Income and expenditure Transaction amount Transfer Transaction amount Amount Transaction amount Amount incurred

[0072] Table 1

[0073] It should be understood that non-standard sequential title line data refers to title line data that does not match the preset standard title library, that is, it cannot completely match the original title content. Specifically, standard sequential title line data is obtained based on the preset standard title library and the target sequential title line data. Then, the standard sequential title line data is removed from the target sequential title line data, and the remaining sequential title line data is the non-standard sequential title line data.

[0074] Further, step S20 includes: comparing the target serial header data with the preset standard header library to obtain successfully matched serial header data; converting the successfully matched serial header data to obtain standard serial header data; and removing the standard serial header data from the target serial header data to obtain non-standard serial header data.

[0075] It should be understood that the successfully matched serial header data refers to the serial header data that matches the preset standard header library. After obtaining the target serial header data, the target serial header data is compared with the standard headers in the preset standard header library. Then, the successfully matched serial header data is converted into standard serial header data, and then the standard serial header data is removed from the target serial header data. After the removal is completed, non-standard serial header data is obtained.

[0076] Step S30: Compare the non-standard serial header data with a preset error correction library to obtain header data that was successfully matched and header data that was not successfully matched.

[0077] It should be understood that the preset error correction database refers to a database that corrects erroneous header row data. This preset error correction database records the mapping relationship between correct headers, incorrect headers, and header lengths. An example of the data recorded in the preset error correction database is shown in Table 2, specifically:

[0078] Correct title Error Title Title length channel Qidao 2 log Mouth 2 Trading hours Trading day in inches 5

[0079] Table 2

[0080] It is understandable that a successfully matched header row data refers to a header row data that has been successfully matched with the header data in the preset error correction library. Similarly, a failed match header row data refers to a header row data that has failed to be matched with the header data in the preset error correction library.

[0081] Step S40: Correct the non-standard serial header data based on the successfully matched header data and the failed matched header data.

[0082] Understandably, after obtaining the successfully matched header row data and the failed matched header row data, the non-standard serial header row data is corrected based on the successfully matched header row data and the failed matched header row data. The correction includes replacing the non-standard serial header row data and rewriting the non-target position status of the non-standard serial header row data. Specifically, the non-target position status is set to "0" and the target position status is set to "1".

[0083] This embodiment preprocesses the serial header data to be processed to obtain target serial header data; determines non-standard serial header data based on a preset standard header library and the target serial header data; compares the non-standard serial header data with a preset error correction library to obtain successfully matched header data and unmatched header data; and corrects the non-standard serial header data based on the successfully matched header data and the unmatched header data. Through the above method, by correcting the non-standard serial header data based on the successfully matched header data and the unmatched header data, the accuracy of the serial data can be effectively improved, and the format of the serial data can be unified.

[0084] In one embodiment, such as Figure 3 The second embodiment of the flow data error correction method of the present invention, based on the first embodiment, includes step S30, which includes:

[0085] Step S301: Construct a serial header data error correction set based on the non-standard serial header data.

[0086] It should be understood that the serial header data correction set refers to the set of correction results that correspond one-to-one with the non-standard serial header data. For example, the non-standard serial header data is {rtitle1, rtitle2, ..., rtitleN1}, and the constructed serial header data correction set is {result1, result2, ..., resultN1}.

[0087] Step S302: Obtain the corresponding serial title line character data based on the serial title line data error correction set.

[0088] It is understandable that the serial header line character data refers to the character data contained in the serial header line data. Since the error titles in the preset error correction library may completely match or only partially match the serial header line data error correction set, it is necessary to traverse and compare the characters. That is, after obtaining the serial header line data error correction set, the serial header line character data is obtained based on the serial header line character data.

[0089] Step S303: The serial header character data is compared with a preset error correction library to obtain successfully matched header data and unmatched header data.

[0090] It should be understood that after obtaining the serial header line character data, the serial header line character data is repeatedly compared with the error headers in the preset error correction library until the length of the reduced header data is the target value, or the length of the reduced header data is less than the minimum header data length in the preset error correction library, or all the serial header line character data are in the target position state. At this time, it indicates that the traversal is complete, that is, the successfully compared header line data and the unsuccessfully compared header line data are obtained.

[0091] Further, step S303 includes: obtaining the title data length of the serial title line character data; when the title data length is a target length, determining whether there is any error-correcting title data with the same title data length in a preset error correction library; if there is no error-correcting title data with the same title data length in the preset error correction library; subtracting one from the title data length to obtain a first title data length; if there is no error-correcting title data with the same title data length in the preset error correction library; subtracting two from the title data length to obtain a second title data length; and if there is no error-correcting title data with the same title data length in the preset error correction library... There is no error-correcting title data with the same length as the second title data; repeat the above steps, decreasing the title data length at unit intervals until the decreased title data length is the target value, or the decreased title data length is less than the minimum title data length in the preset error correction library, or all the character data of the serial title line is in the target position state, and the preset error correction library contains error-correcting title data with the same length as the preset title data; traverse and compare the character data of the serial title line and the error-correcting title data with the same length as the preset title data to obtain the successfully matched title line data and the unmatched title line data.

[0092] Understandably, the target value refers to the length of the title data after reduction. This target value can be 1. After reducing the title data length, it is necessary to determine whether there is any corrected title data with the same length in the preset error correction library. If not, the title data length needs to be reduced further. The reduction cutoff condition can be that the title data length is the target value, or the reduced title data length is less than the minimum title data length in the preset error correction library, or all the characters in the serial title line are in the target position state, or the preset error correction library contains any one of the following: the title data length is the target value, or the reduced title data length is less than the minimum title data length in the preset error correction library. The title data length being the target value means that the reduced title data length is 1, that is, all non-target position states are rewritten to the target position state. All the characters in the serial title line are in the target position state means that the reduced title data length is less than the minimum value of the corrected title data in the preset error correction library.

[0093] This embodiment constructs a serial title line data correction set based on the non-standard serial title line data; obtains corresponding serial title line character data based on the serial title line data correction set; compares the serial title line character data with a preset error correction library to obtain successfully compared title line data and unsuccessfully compared title line data; through the above method, a serial title line data correction set is constructed based on the non-standard serial title line data, and then the serial title line character data in the serial title line data correction set is compared with the error-corrected title data in the preset error correction library, thereby effectively improving the accuracy of the comparison of title data.

[0094] In one embodiment, such as Figure 4 The third embodiment of the flow data error correction method of the present invention, based on the first embodiment, includes step S40, which includes:

[0095] Step S401: Extract the header row data of non-target position states from the non-standard serial header row data.

[0096] It is understandable that header row data refers to the data in the non-target position state of the non-standard pipeline header row data, that is, the recorded result is "0". Specifically, the header row data is extracted from the non-standard pipeline header row data.

[0097] Step S402: Replace the non-standard serial header data with the successfully matched header data, and rewrite the non-target position status of the header data to the target position status.

[0098] It should be understood that after obtaining the successfully matched header row data, the non-standard serial header row data is replaced by the successfully matched header row data. At the same time, the position status of the header row data also needs to be rewritten to the target position status, i.e., 111…1 (the number of 1s is i).

[0099] Step S403: Based on the failed comparison header data, rewrite the non-target position state of the non-standard serial header data into the target position state.

[0100] It is understandable that since the header data that failed to match is the header data that failed to match with the header data in the preset error correction library, that is, there is no non-standard serial header data in the preset error correction library, this situation may be that the header data identified by OCR technology is correct. In this case, there is no need to correct the error. The non-target position state of the non-standard serial header data is directly rewritten to the target position state, that is, 111…1 (the number of 1s is i).

[0101] Understandably, the length of the title data in the serial header line data correction set generally does not exceed 8. The correction title data can be an incorrect title. After obtaining the serial header line character data, it is necessary to obtain the title data length corresponding to the serial header line character data. At this time, it is also necessary to determine whether the title data length i is the target length. This target length can be set to 1. If so, it is necessary to query in the preset correction library whether there is an incorrect title with a title data length of 1. If there is, the incorrect title with a title data length of 1 is directly compared with the serial header line character data. If the comparison is successful, the correct title corresponding to the incorrect title in the preset correction library is directly used to replace the non-standard serial header line data. If not, the non-target position state of the non-standard serial header line data is directly rewritten to the target position state, i.e., "1", to complete the traversal.

[0102] It should be understood that when the title data length is determined to be less than the target length (i.e., the title data length is greater than the target length), it is checked whether there is any corrected title data with the same length in the preset error correction library. If not, the title data length is reduced. If there is also no corrected title data with a shorter length in the preset error correction library, the non-target position state of the non-standard serial title line data is rewritten to the target position state, i.e., "111…1 (the number of 1s is i)", to complete the traversal. If there is corrected title data with the same length in the preset error correction library, the corrected title data is compared with the serial title line character data. If the comparison is successful, the correct title corresponding to the incorrect title in the preset error correction library is directly used to replace the non-standard serial title line data. If the comparison fails, the title data length is reduced.

[0103] Understandably, after determining whether the title data length *i* is the target length, the title data length is subtracted by one, i.e., *i-1*. At this point, it needs to compare with error-correcting title data of the same length as the first title data. If *i* = 2, then *i-1* = 1, indicating that the non-standard sequential title line data has no non-target position state, i.e., no "0". The traversal is then complete. If the preset error-correction library does not contain error-correcting title data of the same length as the first title data, the non-target position state of the non-standard sequential title line data is rewritten to the target position state, i.e., "11", to complete the traversal. If *i* > 2, then *i-1* > 1. Then, it is determined whether there is error-correcting title data in the preset error-correction library with a title data length less than *i-1*. If so, the non-target position state of the non-standard sequential title line data is rewritten to the target position state, i.e., "111...1 (the number of 1s is *i*)", to complete the traversal. If there is no title data in the preset error-correction library... If the error-correcting title data and non-standard serial title line data of length i-1 do not have i-1 consecutive non-target position states, then the title data length is decreased. If the error-correcting title data of length i-1 exists in the preset error-correction library, then the first to i-1 positions of the non-standard serial title line data are compared with the error-correcting title data in the preset error-correction library. If the comparison is successful, the non-standard serial title line data is modified by rewriting it as "111…10 (the number of 1s is the correct title length after error correction)". If the comparison fails, then the second to i positions of the non-standard serial title line data are compared with the error-correcting title data in the preset error-correction library. If the comparison is successful, the non-standard serial title line data is modified by rewriting it as "011…1 (the number of 1s is the correct title length after error correction)". If the comparison fails, then the title data length is decreased.

[0104] It should be understood that the length of the title data is reduced by two, i.e., i-2. At this point, it is necessary to compare it with the error-correcting title data that has the same length as the second title data. If i=3, then i-2=1. If there is no error-correcting title data with the same length as the second title data in the preset error-correcting library, the non-target position state of the non-standard sequential title line data is changed to the target position state, i.e., "1", to complete the traversal. If there is error-correcting title data with the same length as the second title data in the preset error-correcting library, then the characters in the non-target position state of the non-standard sequential title line data are compared one by one. If the comparison is successful, then... The non-standard serial header line data is directly replaced with the correct header corresponding to the erroneous header from the preset error correction library. If the comparison fails, the non-target position state of the non-standard serial header line data is rewritten to the target position state until all values ​​are 1 to complete the traversal. If i > 3, then i - 2 > 1. If there is no error correction header data with the same length as the second header data in the preset error correction library, the header data length is decreased. If there is no error correction header data with a header data length less than i - 2 in the preset error correction library, the non-target position state of the non-standard serial header line data is rewritten to the target position state, i.e., "111". ...1 (the number of 1s is i)” to complete the traversal. If the non-standard serial header data does not have i-1 consecutive non-target position states, the header data length is decreased. If there is a corrected header data in the preset error correction library with a header data length less than i-2, the first to i-2 positions of the non-standard serial header data are compared with the corrected header data in the preset error correction library. If the comparison is successful, the non-standard serial header data is modified, that is, the non-standard serial header data is rewritten as “11…100 (the number of 1s is the correct header length after correction)”, and continues from the i-1 position. Initially, the i-2 bits of the error-corrected title data are compared with the error-corrected title data in the preset error-correction library. If the comparison fails, the second to the (i-1)th bits of the non-standard serial title line data are compared with the error-corrected title data in the preset error-correction library. If the comparison succeeds, the non-standard serial title line data is modified by rewriting it as "011…10 (the number of 1s is the correct title length after correction)". Starting from the i-th bit, the i-2 bits of the error-corrected title data are compared with the error-corrected title data in the preset error-correction library. If the comparison fails, the title data length is continuously reduced.

[0105] Furthermore, after step S403, the method further includes: adding the failed header row data to the preset header update library; iteratively updating the preset standard header library and the preset error correction library using the added preset header update library; and continuing to correct other serial header row data based on the iteratively updated preset standard header library and the preset error correction library.

[0106] It should be understood that due to the limited scope of various bank statement templates collected, in practical applications, there may be situations where incomplete coverage leads to unsuccessful error correction. Therefore, this embodiment adds an iterative mechanism. After error correction is completed, the failed header row data is recorded and the correct header is analyzed. Simultaneously, both the failed header row data and the corresponding correct header are added to a preset header update library as a continuous update mechanism. The preset standard header library and preset error correction library are iteratively updated in a timely manner based on the added preset header update library. Then, based on the iteratively updated preset standard header library and preset error correction library, error correction of other bank statement header rows continues. An example of the preset header update library data is shown in Table 3.

[0107]

[0108] Table 3

[0109] It is understandable that, in order to ensure the effectiveness of the serial data error correction in this embodiment, the data in the preset standard title library, the preset error correction library, and the preset title update library should have certain logical verification relationships. Specifically, the preset standard title library should not contain data where the same original title corresponds to multiple different standard titles; the preset error correction library should not contain data where the same incorrect title corresponds to multiple different correct titles; to improve error correction efficiency, the preset error correction library should replace short characters with long characters as much as possible; and the preset title update library should not contain data where unidentified duplicate titles are present.

[0110] This embodiment extracts header row data in non-target position states from the non-standard serial header row data; replaces the non-standard serial header row data with successfully matched header row data, and rewrites the non-target position states of the header row data to the target position states; and rewrites the non-target position states of the non-standard serial header row data to the target position states based on the failed match header row data. Through this method, by extracting header row data in non-target position states from the extracted non-standard serial header row data, replacing the non-standard serial header row data with successfully matched header row data, and rewriting the non-target position states of the header row data and non-standard serial header row data to the target position states, the accuracy of rewriting to the target position states can be effectively improved.

[0111] Furthermore, this embodiment of the invention also proposes a storage medium storing a pipeline data error correction program, which, when executed by a processor, implements the steps of the pipeline data error correction method described above.

[0112] Since this storage medium adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought about by the technical solutions of the above embodiments, which will not be repeated here.

[0113] In addition, refer to Figure 5 The present invention also proposes a streaming data error correction device, the streaming data error correction device comprising:

[0114] Processing module 10 is used to preprocess the header row data to be processed to obtain the target header row data.

[0115] The determination module 20 is used to determine non-standard serial header data based on the preset standard header library and the target serial header data.

[0116] The comparison module 30 is used to compare the non-standard serial header data with a preset error correction library to obtain the header data that was successfully compared and the header data that failed to be compared.

[0117] The error correction module 40 is used to correct the non-standard serial header data based on the successfully matched header data and the failed matched header data.

[0118] This embodiment preprocesses the serial header data to be processed to obtain target serial header data; determines non-standard serial header data based on a preset standard header library and the target serial header data; compares the non-standard serial header data with a preset error correction library to obtain successfully matched header data and unmatched header data; and corrects the non-standard serial header data based on the successfully matched header data and the unmatched header data. Through the above method, by correcting the non-standard serial header data based on the successfully matched header data and the unmatched header data, the accuracy of the serial data can be effectively improved, and the format of the serial data can be unified.

[0119] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0120] In addition, for technical details not described in detail in this embodiment, please refer to the data error correction method provided in any embodiment of the present invention, which will not be repeated here.

[0121] In one embodiment, the processing module 10 is further configured to identify the serial header data to be processed to obtain character data and blank data; perform blank removal processing on the character data and the blank data respectively; remove the blank data after blank removal from the serial header data to be processed; and generate target serial header data based on the blank character data after blank removal after the removal is completed.

[0122] In one embodiment, the determining module 20 is further configured to compare the target serial header data with the preset standard header library to obtain successfully matched serial header data; convert the successfully matched serial header data to obtain standard serial header data; and remove the standard serial header data from the target serial header data to obtain non-standard serial header data.

[0123] In one embodiment, the comparison module 30 is further configured to construct a serial header data correction set based on the non-standard serial header data; obtain corresponding serial header character data based on the serial header data correction set; and perform a traversal comparison of the serial header character data with a preset correction library to obtain successfully compared header data and unsuccessfully compared header data.

[0124] In one embodiment, the comparison module 30 is further configured to: obtain the title data length of the serial title line character data; when the title data length is a target length, determine whether there is any error-correcting title data with the same title data length in a preset error correction library; if there is no error-correcting title data with the same title data length in the preset error correction library; subtract one from the title data length to obtain a first title data length; if there is no error-correcting title data with the same title data length in the preset error correction library; subtract two from the title data length to obtain a second title data length; and in the preset error correction library... The library does not contain error-correcting title data with the same length as the second title data; repeat the above steps, decreasing the title data length at unit intervals until the decreased title data length is the target value, or the decreased title data length is less than the minimum title data length in the preset error-correcting library, or all the character data of the serial title line is in the target position state, and the preset error-correcting library contains error-correcting title data with the same length as the preset title data; traverse and compare the character data of the serial title line and the error-correcting title data with the same length as the preset title data to obtain successfully matched title line data and unmatched title line data.

[0125] In one embodiment, the error correction module 40 is further configured to extract header data in non-target position states from the non-standard serial header data; replace the non-standard serial header data with successfully matched header data, and rewrite the non-target position state of the header data to the target position state; and rewrite the non-target position state of the non-standard serial header data to the target position state based on the failed match header data.

[0126] In one embodiment, the error correction module 40 is further configured to add the failed header row data to a preset header update library; iteratively update the preset standard header library and the preset error correction library using the added preset header update library; and continue to correct other serial header row data based on the iteratively updated preset standard header library and the preset error correction library.

[0127] Other embodiments or implementation methods of the streaming data error correction device of the present invention can be referred to the above-described method embodiments, and will not be repeated here.

[0128] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0129] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, all-in-one platform workstation, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0131] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for correcting errors in streaming data, characterized in that, The stream data error correction method includes the following steps: Preprocess the header row data to be processed to obtain the target header row data; Non-standard sequential header data are determined based on the preset standard header library and the target sequential header data; The non-standard serial header data is compared with a preset error correction library to obtain header data that was successfully compared and header data that was not successfully compared. The non-standard serial header data is corrected based on the successfully matched header data and the failed matched header data. The step of comparing the non-standard serial header data with a preset error correction library to obtain successfully matched header data and unmatched header data includes: Construct a serial header data error correction set based on the non-standard serial header data; The corresponding serial title line character data is obtained based on the aforementioned serial title line data correction set; The serial header character data is compared with a preset error correction library to obtain successfully matched header data and unmatched header data. The step of comparing the serial header line character data with a preset error correction library to obtain successfully matched header line data and unmatched header line data includes: Obtain the title data length of the serial title line character data; When the title data length is the target length, determine whether there is error correction title data with the same length as the title data in the preset error correction library; There is no error-correction title data with the same length as the title data in the preset error correction library; The length of the title data is calculated by subtracting one from the length of the title data to obtain the length of the first title data; There is no error correction title data with the same length as the first title data in the preset error correction library; The second title data length is obtained by subtracting two from the title data length. There is no error correction title data with the same length as the second title data in the preset error correction library; Repeat the above steps to decrease the length of the title data at unit intervals until the length of the reduced title data is the target value, or the length of the reduced title data is less than the minimum title data length in the preset error correction library, or all the character data of the serial title line is in the target position state, and the preset error correction library contains error correction title data with the same length as the preset title data. The serial header line character data and the error-correcting header data with the same length as the preset header data are traversed and compared to obtain the header line data that was successfully compared and the header line data that failed to be compared.

2. The method for correcting errors in streaming data as described in claim 1, characterized in that, The preprocessing of the pipeline header data to be processed to obtain the target pipeline header data includes: The serial header data to be processed is identified to obtain character data and blank data; The character data and the blank data are respectively processed to remove empty data; Remove the blank data after removing null values ​​from the header row data to be processed; After the removal is completed, the target serial header row data is generated based on the empty character data.

3. The method for correcting errors in streaming data as described in claim 1, characterized in that, The step of determining non-standard sequential header line data based on the preset standard header library and the target sequential header line data includes: The target serial header data is compared with the preset standard header library to obtain the serial header data that has been successfully matched. The successfully matched serial header data is converted to obtain standard serial header data; The standard serial header data is removed from the target serial header data to obtain non-standard serial header data.

4. The method for correcting errors in streaming data as described in any one of claims 1 to 3, characterized in that, The step of correcting the non-standard serial header data based on the successfully matched header data and the failed matched header data includes: Extract the header row data that is not in the target position state from the non-standard pipeline header row data; Replace the non-standard serial header data with the successfully matched header data, and rewrite the non-target position status of the header data to the target position status; Based on the failed comparison header data, the non-target position state of the non-standard serial header data is rewritten to the target position state.

5. The method for correcting errors in streaming data as described in claim 4, characterized in that, After rewriting the non-target position state of the non-standard serial header data to the target position state based on the failed comparison header data, the method further includes: Add the failed comparison header row data to the preset header update library; The preset standard title library and preset error correction library are iteratively updated by adding the preset title update library; Based on the iteratively updated preset standard title library and preset error correction library, continue to correct errors in other serial title line data.

6. A streaming data error correction apparatus for performing the streaming data error correction method as described in any one of claims 1 to 5, characterized in that, The streaming data error correction device includes: The processing module is used to preprocess the header row data to be processed to obtain the target header row data; The determination module is used to determine non-standard serial header data based on a preset standard header library and the target serial header data; The comparison module is used to compare the non-standard serial header data with a preset error correction library to obtain header data that was successfully compared and header data that was not successfully compared. The error correction module is used to correct the non-standard serial header data based on the successfully matched header data and the failed matched header data.

7. A data error correction device for streaming data, characterized in that, The pipeline data correction device includes: a memory, a processor, and a pipeline data correction program stored in the memory and executable on the processor, wherein the pipeline data correction program is configured to implement the pipeline data correction method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores a streaming data error correction program, which, when executed by the processor, implements the streaming data error correction method as described in any one of claims 1 to 5.

Images