Form data input mode judgment system

Abstract

The invention relates to a form data input mode judgment system, which is used for capturing a learning field information quantity and a learning time stamp quantity from each reference true phase form data of a basic true phase known as manual input or automatic input, and carrying out learning calculation according to the information quantity so as to generate a judgment calculation model; further capturing a judgment field information quantity and a judgment time stamp quantity from each piece of to-be-judged form data, and predicting a judgment result that each piece of to-be-judged form data is automatically input or manually input by utilizing the judgment calculation model according to the judgment field information quantity and the judgment time stamp quantity; and when a feedback reference true phase of each piece of to-be-judged form data does not accord with the judgment result, defining an abnormal judgment form data and a recognition reference true phase so as to carry out learning calculation again to modify the judgment calculation model.

Description

Technical Field

[0001] The present invention relates to a determination system, and more particularly to a determination system for determining whether the input method of form data is manual input or automatic input. Background Art

[0002] In response to the needs of organizational management, system management, and information integration management, more and more form data need to be sent to the data storage device (server) of the data management center for centralized management, so as to serve as the data basis for analyzing specific things. Some of these form data are generated by manual input methods such as typing, clicking options, or handwriting combined with image recognition technology; other parts may be generated by automatic input methods such as reading barcodes, chip sensing, label sensing, image recognition, or automatic system import or import.

[0003] In order to accurately judge and analyze the meaning of all the form data, accurate statistical analysis must be performed through big data calculations. The premise of this approach is that the content of the form data itself must have extremely high accuracy to avoid misjudging the true meaning of the form data. However, the error rate when entering data manually by typing, writing or clicking is often much higher than the error rate when entering data automatically by reading barcodes, chip sensing, label sensing, image recognition or system automatic import or import.

[0004] Due to the huge amount of form data, it is impractical for the staff of the data management center to check and confirm whether the form data is correct one by one. It can only be assisted by auxiliary verification tool software. Even so, due to the high diversity of form data, it is impossible to develop corresponding verification tool software for all form data to verify them one by one.

[0005] If form data can be automatically generated, the workload of the staff in the data management center can be greatly reduced. Therefore, it is imperative to promote the automatic input of form data. However, in response to various usage scenarios and usage needs, not all forms can be converted to automatic data input in a short period of time, so it is inevitable that some form data will still be generated by manual data input.

[0006] Since the form data stored in the data storage device (server) of the data management center is mixed with some form data generated by automatic input and some form data generated by manual input, it is necessary to develop a new judgment technology to judge which form data is generated by automatic input and which is generated by manual input. In this way, more inspection resources can be invested in inspecting the form data generated by manual input to improve the overall accuracy of the form data. Summary of the invention

[0007] In view of the fact that in the prior art, there is a lack of judgment technology for judging whether form data is automatically input or manually input, and therefore it is impossible to concentrate verification resources on verifying form data generated by manually input data, resulting in difficulty in improving the overall accuracy of form data; one of the necessary technical means adopted by the present invention to solve the problems of the prior art is to provide a form data input method judgment system (hereinafter referred to as "judgment system"), and the judgment system includes a data storage device and a judgment device.

[0008] The data storage device stores a plurality of reference truth form data and a plurality of to-be-judged form data, and the reference truth form data correspondingly have a plurality of initial reference truths defined as automatic input or manual input. The judgment device is communicatively connected to the data storage device to capture the reference truth form data and the to-be-judged form data, and generates a feature capture module, a supervised learning module, a judgment module, and a verification warning module after installing and executing a judgment program.

[0009] The feature extraction module extracts a learning field information quantity and a learning timestamp quantity reflecting the data disorder from each benchmark truth form data, so that the benchmark truth form data has a corresponding plurality of the above-mentioned learning field information quantities and a plurality of the above-mentioned learning timestamp quantities, and extracts a judgment field information quantity and a judgment timestamp quantity reflecting the data disorder from each form data to be judged.

[0010] The supervised learning module performs a learning calculation based on the learning field information amount and the learning time stamp amount corresponding to the benchmark truth form data and the initial benchmark truth to generate a judgment calculation model.

[0011] The judgment module judges each form data to be judged based on the judgment calculation model and the judgment field information amount and judgment timestamp amount corresponding to each form data to be judged to generate a judgment result for predicting whether each form data to be judged is automatically input or manually input, thereby generating multiple judgment results.

[0012] The verification alarm module receives a plurality of feedback reference truths used to define the form data to be judged as automatically input or manually input, and issues a warning message when the judgment result corresponding to the form data to be judged is verified to be inconsistent with the feedback reference truth, thereby defining a judgment abnormal form data and a confirmation reference truth. The verification alarm module further stores the judgment abnormal form data and the confirmation reference truth in a data storage device as the reference truth form data and the initial reference truth, so as to provide the supervised learning module with a new learning calculation to modify the judgment calculation model.

[0013] Based on the above necessary technical means, among the derived subsidiary technical means, preferably, the judgment system further includes a plurality of data input terminal devices, and the reference truth form data and the form data to be judged can be transmitted from the data input terminal device to the data storage device for storage. The data storage device can be a data storage server, and the judgment device can be a computing server. Each data input terminal device further includes a feedback operation interface, so that an operator of each data input terminal device can input feedback reference truth when verifying that the judgment result corresponding to the form data to be judged is wrong.

[0014] Preferably, the verification alarm module further includes a verification cycle setting interface for setting a verification cycle, so as to periodically judge abnormal form data and confirm the benchmark truth as the benchmark truth form data and the initial benchmark truth according to the verification cycle, so that the supervised learning module can periodically perform the learning calculation according to the verification cycle.

[0015] The feature extraction module may include a field information extraction unit, and the field information extraction unit is operated according to a field entropy algorithm to obtain the learning field information and the judgment field information, and the field entropy algorithm is for in k represents the number of data field types, indicating that there are k types of data field types in total. i represents the number of the i-th data field type among the k data field types, where i, k and a i All are natural numbers.

[0016] The feature extraction module may further include a timestamp quantity extraction unit, and the timestamp quantity extraction unit allows a user to specify q timestamp fields in p fields, and extracts learning timestamp quantities and judgment timestamp quantities according to a timestamp quantity algorithm, and the timestamp quantity algorithm is Max 1≤j≤q (v j ), where v j Indicates the number of different data content types contained in the r columns of data corresponding to the jth timestamp field in the q timestamp fields, where j, v j, p, q and r are all natural numbers, and p>q.

[0017] The determination module may further include a marking unit, and the marking unit assigns an automatic input mark or a manual input mark to each form data to be determined according to the determination result, and then stores it in the data storage device.

[0018] In addition, for the learning algorithm, preferably, the learning algorithm may include at least one basic training algorithm, and the basic training algorithm includes at least one of the K-nearest neighbor algorithm (KNN), the support vector machine (SVM) algorithm, the decision tree algorithm and the regression algorithm. More preferably, the learning algorithm may further include at least one fitting algorithm, and the fitting algorithm includes at least one of the random forest algorithm and the extreme gradient boosting (XGBoost).

[0019] In summary, in the form data input method judgment system provided by the present invention, based on the long-term observation results of the correlation and regularity between the form data being automatically input or manually input, the field information amount and the timestamp amount related to time and data disorder are specially selected as the important feature basis for subsequent learning training and judgment, and supervised learning training is performed based on this to establish a judgment calculation model with a higher confidence level and obtain a judgment result with a higher judgment accuracy rate in a short time.

[0020] Furthermore, by periodically conducting judgment, verification, warning, and generating the basic truth for verification, the erroneous judgment results can be corrected, and the judgment calculation model can be modified by re-learning and calculation. Thus, not only can the effect of automatic judgment input mode be achieved, but also the judgment accuracy can be greatly improved in a short time. After obtaining the judgment results with a high judgment accuracy rate, the inspection resources (including personnel, equipment, and / or tool software, etc.) can be further concentrated on the inspection of the form data generated by the manually input data, so as to further achieve the effect of improving the overall accuracy of the form data. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram showing the functional modules of a form data input mode determination system provided by a preferred embodiment of the present invention.

[0022] Explanation of symbols:

[0023] 100: Judgment system

[0024] 1: Data storage device

[0025] 2: Judgment device

[0026] 21: Feature Extraction Module

[0027] 211: Field information acquisition unit

[0028] 212:Time stamp quantity acquisition unit

[0029] 22:Supervised Learning Module

[0030] 23: Judgment module

[0031] 231: Judgment unit

[0032] 232: Marking unit

[0033] 24: Verify the alarm module

[0034] 241: Verification Unit

[0035] 242: Warning unit

[0036] 243: Verification cycle setting interface

[0037] 244: Judgment accuracy calculation unit

[0038] 3a~3c: Data input terminal device

[0039] 31a~31c: Data input interface

[0040] 32a~32c: Feedback operation interface

[0041] GF: Benchmark Truth Form Data

[0042] JF: Form data to be judged

[0043] JAP: Judgment Procedure DETAILED DESCRIPTION

[0044] Since the form data input method determination system provided by the present invention can be widely used to determine whether form data is generated by automatic input or manual input, its application level is quite broad, so it will not be described one by one here, and only a preferred embodiment is listed for specific description, and this embodiment is only used to conveniently and clearly assist in explaining the purpose and effect of the embodiment of the present invention.

[0045] See also Figure 1 , which is a schematic diagram showing the functional modules of the form data input mode determination system provided by the preferred embodiment of the present invention. Figure 1 As shown, a form data input mode determination system (hereinafter referred to as "determination system") 100 includes a data storage device 1, a determination device 2 and data input terminal devices 3a-3c.

[0046] The data storage device 1 can be a data storage server. The judgment device 2 can be a computing server. The data input terminal devices 3a-3c can be terminal devices for inputting form data such as a built-in computer in a work device, an industrial computer, a desktop computer, a notebook computer, a tablet computer or a smart phone, and each of them has a data input interface 31a-31c and a feedback operation interface 32a-32c. The data input interface 31a-31 and the feedback operation interface 32a-32 can be a program operation page after the data input terminal device 3a-3c executes a specific program, or can be a network operation page presented on the data input terminal device 3a-3c after connecting to a web server.

[0047] The data storage device 1 stores a plurality of reference truth form data GF and a plurality of form data to be judged JF, and the reference truth form data GF and the form data to be judged JF can be input by at least one operator of the data input terminal devices 3a-3c using at least one of the data input interfaces 31a-31c, and then transmitted to the data storage device 1 for storage. The reference truth form data GF has a plurality of initial reference truths correspondingly defined as automatic input (i.e., form data generated by automatic data input) or manual input (i.e., form data generated by manual data input). The so-called initial reference truth refers to the factual truth that has been verified and has a very high credibility before the form data to be judged JF is judged by the judgment system 100. The form data to be judged JF is the form data that is judged by the judgment system 100 as automatic input or manual input.

[0048] For example, as shown in Table 1, the data storage device 1 stores 10 reference truth form data, and the corresponding serial numbers are 0001 to 0010, respectively. The initial reference truth of the reference truth form data with serial numbers 0001 and 0005 is input by manual input, and the so-called manual input method can be input manually by typing, clicking on options, or handwriting with text recognition software. In order to facilitate personnel and software recognition and reading, the reference truth form data with serial numbers 0001 and 0005 can be given an "M" mark to represent manual input. The initial reference truth of the remaining 8 reference truth form data is input by automatic input, and the so-called automatic input method can be input by automatic input methods such as reading barcodes, chip sensing, label sensing, image recognition, or automatic import or import by the system. Similarly, the automatically input reference truth form data can be given an "A" mark.

[0049] Table 1: Benchmark truth form data list

[0050] Benchmark Truth Form Data Serial Number Initial Ground Truth mark 0001 Manual Input M 0002 Automatic Input A 0003 Automatic Input A 0004 Automatic Input A 0005 Manual Input M 0006 Automatic Input A 0007 Automatic Input A 0008 Automatic Input A 0009 Automatic Input A 0010 Automatic Input A

[0051] The judgment device 2 is communicatively connected to the data storage device 1 to capture the reference truth form data GF and the form data to be judged JF, and is installed with a judgment program JAP. After executing the judgment program JAP, a feature capture module 21, a supervised learning module 22, a judgment module 23 and a verification warning module 24 are generated.

[0052] After long-term observation, the inventors found that the common correlations and regularities between automatically input form data and manually input form data include:

[0053] 1. Automatic input can input a large amount of data in a short time, so those who input a large amount of data in a short time are mostly automatically input;

[0054] 2. Data with high similarity or high repetitiveness are mostly entered automatically, while data with high dissimilarity or low repetitiveness are mostly entered manually;

[0055] 3. Since form data is entered based on fields, there is a regularity that multiple rows of data in the same field often have the same type of attributes (such as time, text, and numbers, etc.); and

[0056] 4. The characteristics of data such as similarity, difference and repeatability can be reflected through parameters or indicators related to data disorder.

[0057] According to the above-mentioned correlations that the inventor has observed and summarized over a long period of time, in order to improve the judgment ability of the judgment device 2, it should be given priority to capture parameters or indicators related to time and data chaos as a basis for learning and training and subsequent judgment. Under the above premise, the feature capture module 21 includes a field information quantity capture unit 211 and a timestamp quantity capture unit 212.

[0058] During the learning phase, the field information acquisition unit 211 can acquire a learning field information amount reflecting the data disorder from each reference truth form data GF, and when determining whether each form data to be determined JF is automatically input or manually input (i.e., during the determination phase), acquire a judgment field information amount reflecting the data disorder from the form data to be determined JF.

[0059] The field information acquisition unit 212 can be operated according to a field entropy algorithm to obtain the learning field information and the judgment field information. The field entropy algorithm is: in k represents the number of data field types, indicating that there are k types of data field types in total. i represents the number of the i-th data field type among the k data field types, where i, k and a iAll are natural numbers. Since the field entropy algorithm is a logarithmic function of probability, the probability must be less than 1, and the logarithm of a value less than 1 must be negative, so a minus sign must be added to restore it to a positive value.

[0060] In addition to the above-mentioned field entropy algorithm, the field information acquisition unit 212 can also acquire other parameters related to data disorder, such as data repetition rate (the number of fields with the same data or the total number of data or the proportion) or data similarity (the proportion of the same data content) as field information for learning and field information for judgment.

[0061] The timestamp quantity acquisition unit 212 may provide a timestamp designation interface (not shown) for a user to designate q timestamp fields from p fields, and acquire learning timestamp quantities according to a timestamp quantity algorithm during the learning phase, and acquire judgment timestamp quantities according to the same timestamp quantity algorithm during the judgment phase. The timestamp quantity algorithm may be Max 1≤j≤q (v j ), where v j Indicates the number of different data content types contained in the r columns of data corresponding to the jth timestamp field in the q timestamp fields, where j, v j , p, q and r are all natural numbers, and p>q.

[0062] Although the data content of the timestamp field does not necessarily have to be the time itself, it is preferably related to time, such as a sequence number or a serial number that can reflect the time sequence. In addition, although in this embodiment, the timestamp quantity algorithm takes the maximum value of the number of different data content types in all timestamp fields, in actual application, the arithmetic mean, median or mode of the number of different data content types in all timestamp fields can also be used as the timestamp quantity.

[0063] Regarding the extraction of information from columns (for learning or judgment), for example, as shown in Table 2, a form contains 6 columns, namely, sales date, brand, model, quantity, unit price and sales amount. Among them, sales date belongs to the time column; brand and model belong to the text column; quantity, unit price and sales amount belong to the number column. Therefore, there are 3 columns, namely, time column, text column and number column, indicating that the above k is 3; the number of time columns is 1, indicating that a1 is 1; the number of text columns is 2, indicating that a2 is 2; and the number of number columns is 3, indicating that a 3 is 3.

[0064] The probability of the first column (time column) =1 / 6;

[0065] The probability of the second column (text column) is equal to 2 / 6;

[0066] Probability of the third column (number column) is equal to 3 / 6;

[0067] Bring in the field entropy algorithm The amount of column information that can be obtained is 0.4392473. When this form data is the above-mentioned reference truth form data GF, it means that the amount of column information for learning is 0.4392473; when this form data is the above-mentioned form data to be judged JF, it means that the amount of column information for judgment is 0.4392473.

[0068] Regarding the extraction of time stamp quantity (for learning or judgment), for example, as shown in Table 2, as mentioned above, there are 6 columns representing P equal to 6. Among these 6 columns, only the sales date is related to time. Therefore, the user can specify the "sales date" column as the time stamp column through a time stamp designation interface (not shown) provided by the time stamp quantity extraction unit 212, indicating that q is 1. The time stamp column ("sales date" column) in the table has 14 rows of data, indicating that r is 14. Among the 14 rows of data in the only time stamp column ("sales date" column), there are only 3 different data contents, namely "March 26", "March 27" and "March 28", which means that the number of different data content types is 3, that is, v j is 3. Because there is only one timestamp field, q is 1, so Max 1≤j≤q (v j ) is 3, indicating that the timestamp amount is 3. When this form data is the above-mentioned reference truth form data GF, it means that the learning time stamp amount is 3; when this form data is the above-mentioned to-be-judged form data JF, it means that the judgment time stamp amount is 3.

[0069] Table 2: Form Information

[0070]

[0071]

[0072] Although in the above example, the six columns of sales date, brand, model, quantity, unit price and sales amount are arranged horizontally, in practice, the above columns may also be arranged vertically. When the columns are arranged vertically, the above r columns (arranged horizontally) of data can be replaced by r rows (arranged vertically), and the feature extraction method adopted, including the extraction of the column information (for learning or judgment) and the time stamp (for learning or judgment) is similar to the above description, except that the vertical and horizontal columns are interchanged and the rows and columns are interchanged, which will not be described in detail below.

[0073] The supervised learning module 22 performs a learning calculation to generate a judgment calculation model based on the learning field information amount and the learning timestamp amount corresponding to the benchmark truth form data GF and the initial benchmark truth (which can be represented by the automatic input mark A or the manual input mark M). For the learning calculation part, preferably, the learning calculation can include at least one basic training algorithm, and the basic training algorithm includes at least one of the K-nearest neighbor algorithm (KNN), the support vector machine (SVM) algorithm, the decision tree algorithm and the regression algorithm. More preferably, in addition to using the basic training algorithm, the learning calculation can also include at least one fitting algorithm, and the fitting algorithm includes at least one of the random forest algorithm and the extreme gradient boosting (XGBoost). The judgment calculation model refers to a mathematical calculation model automatically derived by performing the above-mentioned learning calculation.

[0074] Since the above-mentioned learning algorithm techniques (including basic training algorithms and fitting algorithms) are currently quite mature algorithm techniques, anyone with general knowledge in the relevant field can use the above-mentioned algorithms or their combinations to construct the above-mentioned judgment algorithm model, and will not be elaborated below.

[0075] The judgment module 23 may include a judgment unit 231 and a marking unit 232. The judgment unit 231 judges each form data to be judged to generate a judgment result for predicting whether each form data to be judged is automatically input or manually input according to the judgment calculation model and the judgment field information amount and judgment time stamp amount corresponding to each form data to be judged, thereby generating a plurality of judgment results. The marking unit 232 assigns an automatic input mark A or a manual input mark M to each form data to be judged according to the judgment result, and then stores it in the data storage device 1.

[0076] Simply put, the judgment algorithm model is used to learn and train in the learning phase how to derive the initial reference truth (which can be represented by the automatic input mark A or the manual input mark M) based on the learning field information and the learning timestamp quantity, so as to derive and predict the judgment result (which can also be represented by the automatic input mark A or the manual input mark M) based on the judgment field information and the judgment timestamp quantity in the judgment phase.

[0077] For example, as shown in Table 3, the data storage device 1 also stores 10 to-be-judged form data JF, and the corresponding serial numbers are 1001 to 1010, respectively. Therefore, 10 judgment results will be generated after the judgment is completed. The judgment results of the to-be-judged form data with serial numbers 1001, 1003 and 0005 are input by manual input, so the marking unit 232 will assign the manual input mark M to the to-be-judged form data with serial numbers 1001, 1003 and 0005, and the judgment results of the remaining to-be-judged form data are automatically input, and the marking unit 232 will assign the automatic input mark A.

[0078] Table 3: Judgment results and marks of form data to be judged

[0079]

[0080]

[0081] After storing the judgment result in the data storage device 1, the judgment device 2 can send a push signal to the data input terminal device 3a~3c, asking the operator of the data input terminal device 3a~3c to verify whether the judgment result is correct. In the data input terminal device 3a~3c, when it is verified that the judgment result corresponding to the form data to be judged is wrong, the feedback reference truth is inputted correspondingly using the feedback operation interface 32a~32c. At this time, the corresponding form data to be judged will be defined as a judgment abnormal form data. When it is verified that the judgment result corresponding to the form data to be judged is correct, the judgment correct information can be inputted. The feedback reference truth and the judgment correct information will be transmitted to the judgment device 2.

[0082] The verification alarm module 24 may include a verification unit 241, an alarm unit 242, a verification cycle setting interface 243, and a judgment accuracy calculation unit 244. When the judgment device 2 receives the feedback reference truth, the verification unit 241 will define the corresponding form data to be judged as a judgment abnormal form data, and determine that the judgment result does not match the feedback reference truth. At this time, the verification unit 241 will list the feedback reference truth as the confirmation reference truth corresponding to the judgment abnormal form data, and the alarm unit 242 will issue a judgment abnormality prompt message.

[0083] On the contrary, when the judgment device 2 receives the correct judgment information, the verification unit 241 will define the corresponding form data to be judged as a correct judgment form data, and determine that the judgment result is consistent with the feedback reference truth. At this time, the verification unit 241 will directly list the judgment result as the confirmed reference truth corresponding to the correct judgment form data. Then, the verification unit 241 can transmit the abnormal judgment form data and the corresponding confirmed reference truth to the data storage device 1 for storage, as the newly added reference truth form data and the initial reference truth, respectively, so as to provide the supervised learning module 22 with a new learning calculation to modify the judgment calculation model.

[0084] The verification cycle setting interface 243 can be further used to set a verification cycle, so that the abnormal form data and the confirmed benchmark truth are periodically used as the benchmark truth form data and the initial benchmark truth according to the verification cycle, so that the supervised learning module can periodically perform learning calculations according to the verification cycle to periodically modify the judgment calculation model. The judgment accuracy calculation unit 244 will count the number of abnormal form data and correct form data defined in the judgment form data to calculate the judgment accuracy during each verification cycle. The verification cycle can be determined based on the statistical cycle requirements for obtaining the judgment accuracy, the number of form data or other requirements, and can be set to verify once a day, once a week, once a month, or once a quarter, etc.

[0085] For example, following the judgment result of Table 3, as shown in Table 4, after the operator of the data input terminal device 3a-3c verifies the judgment result, it is found that the judgment result of the form data to be judged with the serial number 1003 is manually input, but in fact, the form data to be judged with the serial number 1003 is automatically input, so one of the feedback operation interfaces 32a-32c can be used to input the feedback reference truth that should be "automatically input". At this time, the verification unit 241 will define the form data to be judged with the serial number 1003 as the judgment abnormal form data, and the feedback reference truth (automatically input) is listed as the confirmation reference truth of the judgment abnormal form data (i.e., the form data to be judged with the serial number 1003), and stored in the data storage device 1. The warning unit 242 will issue a judgment abnormality prompt message to prompt that the judgment result generated when the mathematical calculation model currently established is used to judge the form data to be judged with the serial number 1003 is wrong.

[0086] On the contrary, after the operator of the data input terminal device 3a~3c verifies the judgment result, if it is found that the judgment results of the remaining serial numbers of the form data to be judged are all correct, the operator can input the judgment correct information through the feedback operation interface 32a~32c. At this time, the verification unit 241 will directly define the remaining serial numbers of the form data to be judged as the correct form data, and directly list the corresponding judgment result as the confirmation reference truth, and also store it in the data storage device 1.

[0087] During each verification cycle, multiple verified form data JF to be judged and the corresponding confirmed benchmark truths can be accumulated, and when entering the next verification cycle, part or all of the accumulated verified form data JF to be judged (such as abnormal form data, or both abnormal form data and correct form data) and the corresponding confirmed benchmark truths will be used as newly added benchmark truth form data and the corresponding initial benchmark truth respectively.

[0088] The judgment accuracy calculation unit 244 will calculate that among the 10 forms to be judged, only 1 (i.e., the form data to be judged with serial number 1003) is judged as abnormal form data, and the remaining 9 forms to be judged are judged as correct form data. Based on this, the judgment accuracy calculation unit 244 can calculate that during this verification cycle, the judgment accuracy of the judgment system 100 in judging whether the form data to be judged JF is automatically input or manually input is 90%. At the same time, the judgment accuracy calculation unit 244 can also calculate the form data automation rate during this verification cycle. As shown in the results of the confirmation benchmark truth in Table 4, during this verification cycle, there are 8 automatically input form data and 2 manually input form data, which means that the form data automation rate during this verification cycle is 80%.

[0089] Table 4: Results of verification of judgment results

[0090]

[0091] After multiple verification cycles of judgment, verification, and re-learning calculation to modify the judgment calculation model, the judgment accuracy of the judgment system 100 in judging whether the form data JF to be judged is automatically input or manually input can be gradually improved, until the judgment accuracy is improved to above a target accuracy (such as 99.99%), indicating that the judgment ability of the judgment system 100 has reached a certain level of confidence. At this time, the verification cycle can be extended (for example, from verification once a quarter to verification once a year), and even the judgment result can be directly accepted, that is, the judgment result obtained by the judgment system 100 each time is directly regarded as the benchmark truth and accepted, without the need for subsequent verification.

[0092] Furthermore, inspection resources (including personnel, equipment and / or tool software, etc.) can be concentrated on inspecting form data generated by manually input data (i.e., form data with a manual input mark M) (especially the reference truth form data GF that has been verified to be manually input), and correcting errors in the form data generated by manually input data to improve the overall accuracy of the form data. In addition, the overall accuracy of the form data can be efficiently improved without increasing the total workload of the inspection (total inspection resources invested) by increasing the sampling rate of manually input form data and reducing the sampling rate of automatically input form data.

[0093] Since the above-mentioned feature extraction module 21, supervised learning module 22, judgment module 23 and verification warning module 24 are all generated by executing the judgment program JAP, the feature extraction module 21, supervised learning module 22, judgment module 23 and verification warning module 24 can essentially be (part of) the main program, sub-program of the judgment program JAP, or the program page or function interface generated after executing the judgment program JAP. Anyone with general knowledge in the relevant technical field (especially the field of artificial intelligence algorithms) can use appropriate programming languages ​​to write the judgment program JAP (including its main program or sub-program) with the functions of the above-mentioned feature extraction module 21, supervised learning module 22, judgment module 23 and verification warning module 24 according to the above learning and judgment logic, so as to realize the above-mentioned various technologies of the present invention.

[0094] In summary, in the form data input method judgment system 100 provided by the present invention, based on the long-term observation results of the correlation and regularity between the form data being automatically input or manually input, the field information amount and the timestamp amount related to time and data disorder are specially selected as the important feature basis for subsequent learning training and judgment, and supervised learning training is performed accordingly to establish a judgment calculation model with a higher confidence level and obtain a judgment result with a higher judgment accuracy rate in a short time.

[0095] Furthermore, the judgment device 2 can be used to periodically judge, verify, warn and generate the basic truth to correct the wrong judgment results, and the judgment calculation model can be modified by re-learning and calculation. In this way, not only can the effect of automatic judgment input mode be achieved, but also the judgment accuracy can be greatly improved in a short time. After obtaining the judgment result with a higher judgment accuracy, the inspection resources (including personnel, equipment and / or tool software, etc.) can be further concentrated on the inspection of the form data generated by the manually input data, so as to further achieve the effect of effectively improving the overall accuracy of the form data.

[0096] The above detailed description of the preferred embodiments is intended to more clearly describe the features and spirit of the present invention, but is not intended to limit the scope of the present invention by the preferred embodiments disclosed above. On the contrary, it is intended to cover various changes and arrangements with equivalents within the scope of the claims to be applied for by the present invention.

Claims

1. A form data input mode determination system, characterized in that: Include: A data storage device stores a plurality of reference truth form data and a plurality of to-be-judged form data, wherein the plurality of reference truth form data correspondingly have a plurality of initial reference truths defined as automatically input or manually input; and A judgment device is communicatively connected to the data storage device to retrieve the plurality of reference truth form data and the plurality of to-be-judged form data, and generates after installing and executing a judgment program: A feature extraction module extracts a learning field information amount and a learning time stamp amount reflecting data disorder from each of the plurality of reference truth form data, so that the plurality of reference truth form data have corresponding plurality of learning field information amounts and plurality of learning time stamp amounts, and extracts a judgment field information amount and a judgment time stamp amount reflecting data disorder from each of the plurality of to-be-judged form data; A supervised learning module performs a learning calculation based on the plurality of learning field information and the plurality of learning time stamps corresponding to the plurality of reference truth form data and one of the plurality of initial reference truths to generate a judgment calculation model; a judgment module, judging each of the plurality of form data to be judged according to the judgment calculation model and the judgment field information amount and the judgment time stamp amount corresponding to each of the plurality of form data to be judged, so as to generate a judgment result for predicting whether each of the plurality of form data to be judged is automatically input or manually input, thereby generating a plurality of the judgment results; as well as A verification alarm module receives a plurality of feedback reference truths used to define the plurality of form data to be judged as automatically input or manually input, and issues a warning message when it is verified that one of the plurality of judgment results corresponding to one of the plurality of form data to be judged is inconsistent with one of the plurality of feedback reference truths, thereby defining a judgment abnormal form data and a confirmation reference truth, and storing the judgment abnormal form data and the confirmation reference truth in the data storage device as one of the plurality of reference truth form data and one of the plurality of initial reference truths, so as to provide the supervised learning module with the learning calculation to modify the judgment calculation model.

2. The form data input mode determination system as claimed in claim 1, characterized in that: It further comprises a plurality of data input terminal devices, and the plurality of reference truth form data and the plurality of to-be-judged form data are transmitted from the plurality of data input terminal devices to the data storage device for storage.

3. The form data input mode determination system as claimed in claim 2, characterized in that: in, Each of the plurality of data input terminal devices further comprises a feedback operation interface for an operator of each of the plurality of data input terminal devices to input one of the plurality of feedback reference truths correspondingly when verifying that one of the plurality of judgment results corresponding to one of the plurality of form data to be judged is wrong.

4. The form data input mode determination system as claimed in claim 3, characterized in that: in, The verification alarm module further includes a verification cycle setting interface for setting a verification cycle, thereby periodically using the abnormal judgment form data and the corresponding confirmation benchmark truth as one of the above-mentioned multiple benchmark truth form data and one of the corresponding multiple initial benchmark truths according to the verification cycle, so that the supervised learning module can periodically perform the learning calculation according to the verification cycle.

5. The form data input mode determination system as claimed in claim 4, characterized in that: in, The verification warning module further includes a judgment accuracy calculation unit for counting the number of the plurality of form data to be judged that have completed judgment during the verification cycle and the number of form data defined as the judgment abnormality, so as to calculate a judgment accuracy.

6. The form data input mode determination system as claimed in claim 1, characterized in that: in, The data storage device is a data storage server, and the determination device is a computing server.

7. The form data input mode determination system as claimed in claim 1, characterized in that: in, The feature extraction module further includes a field information extraction unit, and the field information extraction unit is operated according to a field entropy algorithm to obtain the learning field information and the judgment field information, and the field entropy algorithm is in k represents the number of data field types, indicating that there are k types of data field types in total. i represents the number of the i-th data field type among the k data field types, where i, k and a i All are natural numbers.

8. The form data input mode determination system as claimed in claim 1, characterized in that: in, The feature extraction module further includes a timestamp quantity extraction unit, and the timestamp quantity extraction unit allows a user to specify q timestamp fields in p fields, and extracts the learning timestamp quantity and the judgment timestamp quantity according to a timestamp quantity algorithm, and the timestamp quantity algorithm is Max 1≤j≤q (v j ), where v j Indicates the number of different data content types contained in the r columns of data corresponding to the jth timestamp field in the q timestamp fields, where j, v j , p, q and r are all natural numbers, and p>q.

9. The form data input mode determination system as claimed in claim 1, characterized in that: in, The determination module further includes a marking unit, and the marking unit assigns an automatic input mark or a manual input mark to each of the plurality of form data to be determined according to the determination result, and then stores the result in the data storage device.

10. The form data input mode determination system according to claim 1, wherein: in, The learning algorithm performed by the supervised learning module includes at least one basic training algorithm and at least one fitting algorithm, the at least one basic training algorithm includes at least one of a K-nearest neighbor algorithm, a support vector machine algorithm, a decision tree algorithm, and a regression algorithm, and the at least one fitting algorithm includes at least one of a random forest algorithm and extreme gradient boosting.

Images