A method and system for improving and controlling the quality of report data

Abstract

The present invention relates to the technical field of data mining and data analysis, and specifically discloses a method and system for improving and controlling the quality of report data, including obtaining report data, the report data including text fields and numerical fields; establishing a data dictionary based on the text fields, and establishing a value domain based on the numerical fields; and analyzing the constraints and logical relationships of the text fields and numerical fields to establish specifications; managing the data dictionary, value domain and specifications in a coded manner to form a reporting specification library; and correcting the quality of the report data that has been collected and controlling the quality of the report data to be collected in the future based on the reporting specification library. The method improves the accuracy, completeness and reliability of report data by integrating the methods of data dictionary correction and reporting specification control, reduces the manpower cost and time cost of data quality correction and verification, and ensures the real-time and consistency of data processing; effectively improves the quality of data collected by the data reporting system.

Description

Technical Field

[0001] The present invention relates to the technical field of data mining and data analysis, and in particular to a method and system for improving and controlling the quality of report data. Background Art

[0002] As the main means for various departments, fields and industries to collect, investigate and survey the current situation, the data reporting system provides a basic and important working model for data collection; reporting data brings together the main achievements of the data reporting system, relying on computer systems for collection and storage, and covers a wealth of information on the objects of collection, investigation and census. Data quality is the key to the analysis and application of reporting data, and its quality is directly related to the accuracy of data analysis and the authority of external releases by industries or departments. In the process of large-scale data collection, investigation and census in the industry, due to the involvement of many departments, high data complexity and large differences in the level of reporting personnel, the uncertainty of data quality is prominent. How to correct the collected data and control the quality of future collected data has become an urgent problem to be solved.

[0003] The types of report data are mainly text and numerical field types. The current technology is mainly aimed at the quality detection, evaluation, assessment, and analysis of report data, including the following aspects:

[0004] (1) Before data is reported, the system fill-in value preset or threshold control method is adopted;

[0005] The preset or system threshold control method mainly relies on the computer system's automatic verification and screening of data. Before data collection, rules and thresholds are preset based on the experience of relevant personnel to automatically judge the input data; for example, for numerical data, a reasonable value range can be set in the collection system, and data beyond this range will be regarded as abnormal data and require further processing. For text data, the system can set keywords or format requirements, and data that does not meet the requirements will be rejected or prompted for modification. Although the preset or system threshold control method can improve the quality of some data processing, its shortcomings are also obvious; first, the preset rules and thresholds are often based on past experience or fixed standards, and it is difficult to adapt to complex and changing data environments; second, this method lacks in-depth judgment on the authenticity and rationality behind the data, and there may be misjudgments or omissions; finally, this method cannot handle some special or complex data situations, and manual intervention is required for further processing.

[0006] (2) During or after data reporting, manual verification and correction methods are used;

[0007] The manual verification and correction method mainly relies on the experience and knowledge of professionals to further verify and correct the data records reported by the reporting object in the system; professionals will conduct a comprehensive inspection of the source, content, format, etc. of the data to ensure the authenticity and rationality of the data. For abnormal data or data that does not meet the requirements, professionals will conduct in-depth investigations and verifications, and will contact the reporting personnel to make corrections if necessary; for example, in my country's ship energy consumption data reporting system, maritime law enforcement personnel board the ship to verify the authenticity of the data at a rate of one thousandth, and use methods such as reviewing fuel filling records to ensure the accuracy of the data. Although the manual verification and correction method can make up for the shortcomings of the preset or system threshold control method, its shortcomings are also obvious; first, manual verification requires a lot of manpower and time costs, and is less efficient for large-scale data processing; second, due to differences in personnel experience and knowledge levels, the verification results may be subjective and inconsistent; finally, manual verification is difficult to cope with data collection scenarios with large data volumes and high update frequencies, and it is difficult to ensure the accuracy of all data.

[0008] (3) After the data are reported, correction methods such as discarding, filling or replacing are adopted;

[0009] After the data is reported, for the problem of missing fields in the report data, according to the importance of the fields in the data records, the existing technology mostly adopts three methods: discarding the records where the missing data is located, filling the missing data, and replacing the missing values. For extremely important fields (such as the number of earthquakes in a county in a disaster census, and the total fuel consumption of a ship in a ship energy consumption report), the method of discarding missing data is used, which is to delete the entire data record where the missing field is located. This method can improve the integrity of the retained data records, but reduces the overall amount of data. For generally important fields, the method of filling missing data is used, which is to fill the missing fields in the data records according to numerical rules or text features. For unimportant fields, the method of replacing missing values ​​is used, which is to replace text or numeric missing fields with NaN or -9999.

[0010] In summary, the three main methods used to improve the quality of existing report data have their own shortcomings; first, although the preset or system threshold control method can improve the efficiency of data processing, it is often unable to flexibly respond to complex and changeable data environments due to its rigid rule restrictions and lack of in-depth judgment, which may lead to misjudgment or missed judgment; secondly, although the manual verification and correction method can make up for the shortcomings of the preset method, it faces problems such as high labor costs, subjectivity and inconsistency, which affects the accuracy and real-time performance of data processing; finally, although the correction method of discarding, filling or replacing can deal with the problem of missing some data fields, it may cause data loss, inaccurate filling and other problems; therefore, in practical applications, it is necessary to comprehensively consider various factors and adopt appropriate data quality control strategies based on specific circumstances to ensure the accuracy, completeness and reliability of the data. Summary of the invention

[0011] In view of the above problems, an object of the present invention is to provide a method for improving and controlling the quality of report data. The method improves the accuracy, completeness and reliability of report data by integrating data dictionary correction and reporting specification control, reduces the manpower and time costs caused by data quality correction and verification, and ensures the real-time and consistency of data processing; effectively improves the quality of data collected by the data reporting system, and provides more accurate and reliable support for data analysis and application.

[0012] A second object of the present invention is to provide a system for improving and controlling the quality of report type data.

[0013] The first technical solution adopted by the present invention is: a method for improving and controlling the quality of report data, comprising the following steps:

[0014] S100: Acquire report data, the report data including text fields and numerical fields; establish a data dictionary based on the text fields, establish a value range based on the numerical fields; and analyze constraints and logical relationships between the text fields and the numerical fields to establish specifications;

[0015] S200: managing the data dictionary, value range and specification according to the coding method to form a reporting specification library;

[0016] S300: Correct the quality of the collected report data based on the reporting specification library.

[0017] Preferably, the step S100 of establishing a data dictionary includes:

[0018] S111: Filter out fields that appear repeatedly from the text fields, sort them, and obtain sorted text fields;

[0019] S112: Screening out correct field text, incorrect field text, and unrecognizable field text from the sorted text fields;

[0020] S113: Recording the correspondence between the incorrect field text and the correct field text;

[0021] S114: A data dictionary is established based on the correct field text and the incorrect field text corresponding thereto; for the unrecognizable field text, the data dictionary marks it as unrecognizable.

[0022] Preferably, establishing the value range in step S100 includes:

[0023] S121: Determine whether any numerical field conforms to normal distribution. If it conforms to normal distribution, use the 3σ principle of normal distribution to select the interval from μ-3σ to μ+3σ as the reference value range;

[0024] If it does not conform to the normal distribution, the cumulative frequency selection method is adopted, and the 1%~99% value of the numerical field distribution is used as the reference value range according to the frequency accumulation method;

[0025] S122: Determine a final value range based on the reference value range combined with professional knowledge, thereby obtaining a value range.

[0026] Preferably, the establishing of the specification in step S100 includes establishing a specification based on a constraint relationship, and the establishing of the specification based on the constraint relationship includes:

[0027] S131: analyzing whether any text field and numeric field must be filled in, thereby obtaining a non-empty constraint specification for each text field and numeric field;

[0028] S132: screening multiple key fields for identifying and distinguishing data records, and determining whether the values ​​of the multiple key fields are reported multiple times, thereby obtaining a uniqueness constraint specification for the data records;

[0029] S133: Based on the report data, the data type and length of the text field and the numerical field are analyzed one by one to determine whether they meet the storage requirements or whether there is a waste of storage space, thereby obtaining the data type and length constraint specification of each text field and numerical field;

[0030] S134: Obtain a specification based on a constraint relationship based on the non-null constraint specification, the uniqueness constraint specification, the data type and the length constraint specification.

[0031] Preferably, the establishing of the specification in step S100 includes establishing a specification based on a logical relationship, and the establishing of the specification based on the logical relationship includes:

[0032] According to business requirements and rules, the business logic relationship between fields is analyzed to obtain specifications based on logical relationships.

[0033] Preferably, the step S300 includes correcting the text field based on the reporting specification library, including:

[0034] S311: Create a field copy for storing corrected data;

[0035] S312: Match all the contents of a field in the collected report data with the data dictionary, and when the correct field text is matched, directly copy the correct field text to the field copy;

[0036] When incorrect field text is matched, the correct field text Chinese corresponding to the incorrect field text is copied to the field copy according to the data dictionary correspondence;

[0037] When the field fails to match text, the field in the field copy is filled in as unrecognizable.

[0038] Preferably, the step S300 includes correcting the numerical field based on the reporting specification library, including:

[0039] S321: Create a field copy for storing corrected data;

[0040] S322: compare all the numerical contents of a field in the collected report data with the value domain, and when the numerical value is within the range of the value domain, directly copy it to the copy of the field;

[0041] When a value exceeds the range of the value domain, the value is marked as out of range in the field copy.

[0042] Preferably, the step S300 includes correcting the constraints and logical relationships of the fields based on the reporting specification library, including:

[0043] S331: Create a field copy for storing corrected data;

[0044] S332: The fields with non-null constraints, unique constraints, data type and length constraints in the collected report data are judged one by one according to the coding order in the reporting specification library. For those that do not violate the constraints, the field content is directly copied to the field copy;

[0045] For those that violate the constraint, they are marked in the copy of the field;

[0046] S333: The fields with logical relationships in the collected report data are judged one by one according to the coding sequence in the reporting specification library. For those that do not violate the logical specification, the field content is directly copied to the field copy;

[0047] Any logical violations are marked in the field copy.

[0048] Preferably, the step S300 further includes controlling the quality of the report data to be collected based on the reporting specification library, including:

[0049] S31: Embed the reporting specification library into the data collection system through computer development;

[0050] S32: providing the user with a correct text field based on the data dictionary in the reporting specification library;

[0051] S33: judging in real time whether the value reported by the user exceeds the value range based on the value range in the reporting specification library; if it does not exceed the value range, the value is allowed to be reported; if it exceeds the value range, the data collection system reminds the user;

[0052] S34: Based on the specifications in the reporting specification library, determine whether the data to be collected violates the constraints and logical relationships. If the data to be collected does not violate the constraints and logical relationships, reporting is allowed; if the data to be collected violates the constraints and logical relationships, the data collection system reminds the user.

[0053] The second technical solution adopted by the present invention is: a system for improving and controlling the quality of report data, including a data establishment module, a filling specification library construction module and a correction module;

[0054] The data establishment module is used to obtain report data, the report data includes a text field and a numerical field; establish a data dictionary based on the text field, establish a value range based on the numerical field; and analyze the constraints and logical relationships of the text field and the numerical field to establish specifications;

[0055] The reporting specification library construction model is used to manage the data dictionary, value range and specification in a coding manner to form a reporting specification library;

[0056] The correction module is used to correct the quality of the collected report data based on the reporting specification library.

[0057] Beneficial effects of the above technical solution:

[0058] (1) The present invention discloses a method for improving and controlling the quality of report data, which solves the shortcomings of existing methods for improving the quality of report data. The present invention improves the accuracy, completeness and reliability of report data by integrating data dictionary correction and reporting specification control, that is, combining automatic verification based on the data dictionary with manual verification and correction, and reduces the manpower and time costs of data quality correction and verification, while ensuring the real-time and consistency of data processing; effectively improves the quality of data collected by data reporting systems in various industries, departments and fields, realizes the quality optimization of report data, and provides more accurate and reliable support for data analysis and application.

[0059] (2) The present invention establishes a data dictionary to standardize and correct text fields, effectively solving the problem of irregular and incorrect reporting. At the same time, by establishing a value range for numeric fields, the numeric data is screened and verified, thus avoiding the existence of unreasonable data. In addition, by analyzing the constraints and logical relationships between fields, the accuracy and consistency of the data are ensured. These measures together improve the overall quality of report data.

[0060] (3) The present invention optimizes the data collection process. By embedding the reporting specification library into the data collection system, real-time verification and quality control of data are achieved. In the initial stage of data reporting, the system can verify according to the specifications, and immediately give prompts and require corrections for data that does not meet the specifications. This not only reduces the workload of data cleaning and correction, but also improves the efficiency and accuracy of data collection.

[0061] (4) The present invention enhances the availability and credibility of data. By implementing dynamic data monitoring and real-time feedback, the system can promptly detect and handle reporting errors or anomalies, thus avoiding misleading and misusing data. This enables data users to trust and use the data more, providing a more reliable basis for subsequent data analysis and decision-making.

[0062] (5) The present invention also has good scalability and adaptability; by establishing a dynamic update and maintenance mechanism, the reporting specification library can be adjusted and optimized accordingly as the business develops and changes; this enables the present invention to continuously adapt to new business needs and rule changes, and maintain its long-term practicality and effectiveness.

[0063] (6) By integrating data dictionary correction and reporting specification control, the present invention has significantly improved the quality of report data, optimized the data collection process, enhanced the availability and credibility of data, and has good scalability and adaptability. This has important practical significance and application value for improving data quality and promoting the effective use of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A flowchart of a method for improving and controlling the quality of report data provided by an embodiment of the present invention;

[0065] Figure 2 A schematic diagram of a reporting specification library provided for one embodiment of the present invention;

[0066] Figure 3 A schematic diagram of the structure of a system for improving and controlling the quality of report data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following detailed description of the embodiments of the present invention is further described in detail in conjunction with the accompanying drawings and examples. The detailed description of the following embodiments and the accompanying drawings are used to exemplarily illustrate the principles of the present invention, but cannot be used to limit the scope of the present invention, that is, the present invention is not limited to the preferred embodiments described, and the scope of the present invention is defined by the claims.

[0068] In the description of the present invention, it should be noted that, unless otherwise specified, “plurality” means two or more than two; the terms “first”, “second”, etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance; for ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0069] Embodiment 1

[0070] like Figure 1 As shown, an embodiment of the present invention provides a method for improving and controlling the quality of report data, comprising the following steps:

[0071] S100: Acquire report data, the report data including text fields and numeric fields; establish a data dictionary based on the text fields, establish a value range based on the numeric fields; and analyze the constraints and logical relationships between the text fields and the numeric fields to establish specifications.

[0072] Relevant data tables and records are exported from databases formed by various surveys, censuses or collection systems to obtain report data; wherein, the report data includes text fields and numeric fields, and the numeric fields include integer, floating point and double precision types.

[0073] (1) Establish a data dictionary based on the text fields in the report data, including the following sub-steps:

[0074] S111: Filter out fields that appear repeatedly from the text fields, sort them, and obtain sorted text fields;

[0075] Text fields are divided into fields that appear multiple times and fields that do not appear multiple times for processing. For fields that appear multiple times, such as place names and port names, the text is sorted in ascending order based on the first letter of the pinyin to prevent some field information from being missed during the processing. For fields that do not appear multiple times, such as remarks and instructions, since there are large differences in the data filled in, it is not considered to establish a data dictionary for them.

[0076] S112: Filter out correct field text, incorrect field text, and unrecognizable field text in the sorted text fields;

[0077] Correct field text refers to field text that is filled in correctly and without ambiguity, incorrect field text refers to field text that is filled in irregularly or incorrectly, and unrecognizable field text refers to field text whose content is unrecognizable, such as symbols, spaces, etc.

[0078] S113: Recording the correspondence between incorrect field text and correct field text;

[0079] The correspondence between incorrect field text and correct field text is analyzed and recorded, that is, the correct field text and the incorrect field text corresponding to the correct field text are recorded, so as to improve the quality of incorrect field text.

[0080] For example, in the port field of the ship energy consumption report data, "Dalian" is the correct field text, while "Liaonong Dalian", "Dalian Liaoyu Wharf Berth 9", etc. are incorrect field texts that correspond to "Dalian".

[0081] S114: A data dictionary is established based on the correct field texts and the incorrect field texts corresponding thereto; for the unrecognizable field texts, the data dictionary marks them as unrecognizable.

[0082] The field texts that are filled correctly and without ambiguity (i.e., correct field texts) are directly included in the data dictionary; and the field texts that are filled irregularly or incorrectly (i.e., incorrect field texts) that have a corresponding relationship with the correct field texts are included in the data dictionary; field texts whose reported content cannot be identified are marked as unrecognizable in the data dictionary.

[0083] like Figure 2 As shown, for example, the correct field text "Dalian" is recorded, and the incorrect field texts "Liaonong Dalian", "Dalian Liaoyu Wharf 9 Berths" etc. corresponding to the correct field text "Dalian" are recorded in the data dictionary, that is, in the data dictionary, the correct field text "Dalian" and the incorrect field texts "Liaonong Dalian" and "Dalian Liaoyu Wharf 9 Berths" have a corresponding relationship.

[0084] (2) Establish value ranges based on numeric fields in report data;

[0085] The present invention adopts the 3σ principle of normal distribution, cumulative frequency and other methods to select and establish a reasonable value range, determine the upper limit, lower limit and tolerance of the value range, and numerical fields such as total economic volume, fuel usage and other fields.

[0086] Creating a value range based on the numeric fields in the report data includes the following sub-steps:

[0087] S121: Determine whether any numerical field conforms to normal distribution. If it conforms to normal distribution, use the 3σ principle of normal distribution to select the interval from μ-3σ to μ+3σ as the reference value range;

[0088] If it does not conform to the normal distribution, the cumulative frequency selection method is used, and the 1%~99% value of the numerical field distribution is used as the reference value range according to the frequency accumulation method.

[0089] S122: adjusting and determining a final range of the value based on the reference value combined with professional knowledge, that is, establishing the value range;

[0090] For example, in the data collection of ship energy consumption voyage reports from * to * in 20**, the fuel consumption (numeric field) is taken as the reference value range of [0-32234.2] according to the 1%~99% value of its data distribution; combined with professional knowledge, the world's largest transport ship Hanyuan Algeciras has a tank capacity of 12,000 cubic meters, and its maximum voyage fuel consumption is about 10,440 tons (calculated at 0.87 tons per cubic meter of fuel weight). Taking into account other energy-consuming equipment (such as the energy consumption of ship-borne lifting equipment), the upper limit of the fuel consumption range is reasonably set to 11,000 tons. Therefore, the final value range of the fuel consumption is [0-11000].

[0091] (3) Analyze the constraints and logical relationships between text fields and numeric fields, between text fields and text fields, and between numeric fields and numeric fields, and establish specifications;

[0092] Based on the collected report data, analyze the constraint and logical relationships between text fields and numeric fields, between text fields and text fields, and between numeric fields, so as to establish specifications. That is, understand the meaning, purpose and importance of each field, and establish specifications through field relationship analysis. The specifications include specifications based on constraint relationships and specifications based on logical relationships.

[0093] 1) Establishing specifications based on constraint relationships;

[0094] Constraint relationships include non-null constraints, unique constraints, and data type and length constraints. By analyzing the constraint relationships of non-null constraints, unique constraints, and data type and length constraints, specifications based on constraint relationships are obtained, which specifically includes the following sub-steps:

[0095] S131: analyzing whether any text field and numeric field must be filled in, thereby obtaining a non-empty constraint specification for each text field and numeric field;

[0096] Analyzing non-empty constraints means: determining the importance of the field according to the purpose of the census or survey, analyzing whether any text field and numeric field must be filled in, and if so, recording the specification as required; if not, recording the specification as non-required, thereby obtaining the non-empty constraint specifications for each text field and numeric field.

[0097] S132: screening multiple key fields for identifying and distinguishing data records, and determining whether the values ​​of the multiple key fields are reported multiple times, thereby obtaining a uniqueness constraint specification for the data records;

[0098] Analyzing the uniqueness constraint means: for the data records of collected data, determine whether multiple key field information in the data records are reported multiple times. If multiple key field information is reported multiple times, it is determined to be duplicate data. After being determined to be duplicate data, retain the record with the closest reporting time to the current time and delete other records; thereby avoiding multiple reporting caused by operational errors of the data collection system, network and reporting personnel, avoiding record duplication, and obtaining the uniqueness constraint specification of the data record.

[0099] A key field refers to a combination of one or more columns in a table formed for the purpose of investigation or census, which is used to ensure the uniqueness of data records, prevent duplicate reporting, and avoid data redundancy. In other words, a key field refers to a field used to identify and distinguish specific records in a database. It can be a combination of one or more columns in a table and is used to uniquely identify a record. Key fields usually contain sufficient information to ensure the uniqueness of data records and avoid data redundancy and confusion.

[0100] For example, in the ship energy consumption data report, the unique constraint instance is generated: if the key fields IMO number, last port of call number, current port of call number and reporting date of two reporting records are the same, they are judged as duplicate data. After being judged as duplicate data, the record with the reporting time closest to the current time is retained and the other records are deleted.

[0101] S133: Based on the report data, the data type and length of the text field and the numeric field are analyzed one by one to see whether they meet the storage requirements or whether there is a waste of storage space, so as to obtain the data type and length constraint specification of each text field and numeric field, where the data type and length constraint specification is the data type (length value), for example, text(50), the data type is text type, and the length value is 50;

[0102] Based on the report data, the data type and length of the text field and the numerical field are analyzed one by one to see whether they meet the storage requirements or whether there is a waste of storage space. If the storage requirements are not met or there is a waste of storage space, the text length, the numerical length and the type need to be adjusted in time, so as to obtain the data type and length constraint specifications of each text field and numerical field;

[0103] Analyzing the data type and length constraint means: checking whether the data type and length of any text field and numeric field meet the storage requirements or whether there is a waste of storage space through the report data. For example, in the ship energy consumption data report, the "IMO number" field consists of 7 digits by analyzing the existing reported data (i.e., the obtained report data). Setting its type and length constraint specification to int(7) can meet the storage requirements and reduce the waste of storage space. The constraint can ensure the accuracy and consistency of the data, and give a reminder when the reported relevant data does not meet the relevant data type or exceeds the range, thereby reducing human errors in data reporting. At the same time, the constraint specification can reduce the waste of storage space while ensuring accuracy.

[0104] S134: Establishing a specification based on constraint relationships based on the non-null constraint specification, uniqueness constraint specification, data type and length constraint specification of the text field and the numeric field.

[0105] 2) Establishing norms based on logical relationships;

[0106] Logical relationships are mainly business logic relationships between fields. By analyzing the business logic relationships between fields, specifications based on logical relationships are obtained, including:

[0107] According to business requirements and rules, the business logic relationship between the fields is analyzed and obtained. For example, in the data collected by the ship energy consumption report, there is a business logic relationship between the two fields "whether to use SCR exhaust treatment device" and "SCR use time", which can be summarized into the following two specifications:

[0108] (1) When the "Whether to use SCR exhaust treatment device" field is filled in "Yes", the "SCR use time" field cannot be empty;

[0109] (2) When the “Whether to use SCR exhaust gas treatment device” field is filled in with “No”, the “SCR usage time” field must be 0.

[0110] S200: managing the data dictionary, value range and specification according to the coding method to form a reporting specification library;

[0111] (1) The organizational form of the reporting specification database;

[0112] The reporting specification library manages the data dictionary, value range and specifications in a coded manner, such as Figure 2 The reporting specification library includes a data dictionary managed according to the coding method, and a reporting specification formed by managing value domains and specifications according to the coding method.

[0113] like Figure 2 As shown, taking ship energy consumption data as an example, the data dictionary managed in a coding manner includes field name, code, correct field text in Chinese, correct field text in English, and incorrect field text. The field name and code can ensure that the data dictionary matches and corresponds to the relevant fields. The correct field text in Chinese (i.e., data dictionary item in Chinese) and the correct field text in Chinese (i.e., data dictionary item in English) record the correct content of the data filled in. The incorrect field text (i.e., wrong filling) corresponds to the data dictionary item in Chinese and the data dictionary item in English, and records the incorrect content of the data filled in.

[0114] The value range in the reporting specification formed by managing the value range and specification in the coding method is taken as an item in the reporting specification library and is included in the reporting specification for management;

[0115] The reporting specifications include field name, code and specification content; the field name is the name of a certain field; the code is the unique identifier of the reporting specifications, which includes three parts, namely the table code, field sequence code and specification sequence code. Taking the specification code HC.05.01 of the "Last Port of Call" field in the voyage report form of the ship energy consumption report as an example, HC is the table code, 05 is the field sequence code, and 01 is the specification sequence code. The reporting specifications composed of this coding method are unique and convenient for subsequent addition and deletion management; the specification content describes the reporting requirements in detail.

[0116] (2) Dynamic update and maintenance of the reporting specification database:

[0117] The reporting specification library manages relevant specifications in coded form, and the specifications are unique. When there is a need to update the reporting specifications, new specifications are added according to the coding rules to ensure that the specifications remain unique after addition and deletion, which facilitates subsequent data correction and quality control.

[0118] (3) Report the use feedback of the specification library and update the specifications;

[0119] The feedback effect refers to the use of the reporting specification library to correct and control data quality, combining specific business work scenarios and feedback from reporting personnel to modify, add or delete unreasonable specifications to achieve the purpose of updating specifications; updating specifications is mainly to solve the problem of mismatch between the reporting specification library and actual applications, in order to adapt to changes and optimizations in reporting specifications. For example, if the set value range is too large or too small, the corresponding specifications need to be updated.

[0120] The present invention establishes a clear, practical and easy-to-manage reporting specification library, which can be embedded in the data acquisition system by technical personnel and effectively used for data correction and quality control.

[0121] S300: Correct the quality of the collected report data based on the reporting specification library.

[0122] (1) Text field data correction;

[0123] For the text fields of the collected report data, they are corrected based on the data dictionary in the reporting specification library, including:

[0124] S311: Create a field copy for storing corrected data;

[0125] Filter out text fields with established data dictionaries from report data. To ensure that the original data is retained after correction, create copies of the relevant data one by one and then perform corrections. The copy of "XX field" is named "XX field_after correction".

[0126] S312: Match all the contents of a certain field with the data dictionary, and when the correct field text (including the correct field text in Chinese and the correct field text in English) is matched, directly copy the correct field text to the field copy (for example, "XX field_after correction");

[0127] When incorrect field text is matched, the correct field text Chinese corresponding to the incorrect field text is copied to the field copy according to the data dictionary correspondence;

[0128] When the field fails to match text, the field in the field copy is filled in as unrecognizable, for example, a field "XX field_after correction" is filled in as unrecognizable.

[0129] The present invention can correct the irregular and incorrect filling contents of the existing text field data through text field data correction; at the same time, the unrecognizable filling contents are marked to facilitate the selection by data users.

[0130] (2) Correction of numerical field data;

[0131] For the numerical data of the collected report data, the numerical data in the report data is corrected based on the value range in the reporting specification library, including:

[0132] S321: Create a field copy for storing corrected data.

[0133] Filter out the numeric fields with value domains from the report data. To ensure that the original data is retained after correction, create copies of the relevant data one by one and then perform correction. The copy of "XX field" is named "XX field_after correction".

[0134] S322: compare all the numerical contents of a certain field with the value range, and when the numerical value is within the range of the value range, directly copy it to the field copy (e.g., "XX field_after correction");

[0135] When a value exceeds the range of the value domain, the value will be marked as out of range in the field copy. For example, "XX field_after correction" will be marked as -9999 to facilitate data users to make choices.

[0136] (3) Data correction based on constraints and logical relationships;

[0137] S331: Create a field copy for storing corrected data.

[0138] Filter out fields with constraints and logical specifications from report data. To ensure that the original data is retained after correction, create copies of the relevant fields one by one and then perform corrections. The copy of "XX field" is named "XX field_after correction".

[0139] S332: Fields with non-null constraints, uniqueness constraints, data type and length constraints are judged one by one according to the coding order in the reporting specification library. For those that do not violate the constraints, the field content is directly copied to the field copy (for example, "XX field_after correction"); for those that violate the constraints, they are marked in the field copy. For example, the copy of the text field "XX field_after correction" is marked as NaN, and the numeric field is marked as -999. The purpose of marking is to remove the impact of the data that violates the constraints during subsequent statistical analysis.

[0140] S333: The fields that establish logical relationships are judged one by one according to the coding order in the reporting specification library. For those that do not violate the logical specifications, the field content is directly copied to the field copy (for example, "XX field_after correction"); for those that violate the logic, they are marked in the field copy. For example, the copy of the text field "XX field_after correction" is marked as NaN, and the numeric field is marked as -999. The purpose of marking is to remove the impact of data that violates the logical relationship during subsequent statistical analysis.

[0141] Furthermore, in one embodiment, the step S300 also includes: controlling the quality of the report data to be collected based on the reporting specification library.

[0142] For the report data to be collected in the future, the main method is to embed the reporting specification library into the data collection system to achieve data quality control, including:

[0143] S31: Embed the reporting specification library into the data collection system through computer development;

[0144] S32: providing the user with a correct text field based on the data dictionary in the reporting specification library;

[0145] Provide users with correct text fields for checking and filling in: When the data collection system collects data, the "correct field Chinese" and "correct field English" in the data dictionary are provided to users for checking to avoid incorrect and non-standard collected text data; at the same time, users are allowed to add data dictionary content to ensure that the data dictionary can meet the reporting needs.

[0146] S33: Based on the value domain in the reporting specification library, it is determined in real time whether the value reported by the user exceeds the value domain range. If it does not exceed the value domain range, the value is allowed to be reported; if it exceeds the value domain range, the data collection system reminds the user.

[0147] Provide a value domain for numeric fields to control the range of reported data: When the data collection system collects data, it determines in real time whether the values ​​reported by the user meet the value domain requirements. If they meet the requirements, they are allowed to report. For values ​​that are out of range, the user is reminded.

[0148] S34: Based on the specifications in the reporting specification library, determine whether the data to be collected violates the constraints and logical relationships. If the data to be collected does not violate the constraints and logical relationships, reporting is allowed; if the data to be collected violates the constraints and logical relationships, the data collection system reminds the user.

[0149] Control the reporting content based on constraints and logical relationships: Use constraints and logical relationships to determine whether the data to be collected violates the constraints and logical relationships. If not, reporting is allowed; if the reporting violates the constraints and logical relationships, the data collection system will immediately give a reminder and require the user to make corrections. After the corrections are completed and in compliance with the constraints and logical relationships, the user is allowed to report.

[0150] The present invention embeds a reporting specification library into a data collection system to ensure that verification and quality control can be implemented in the initial stage of data reporting; and by setting a pre-verification mechanism on the data reporting interface, pre-verification of the reporting specifications is implemented. When the user starts to input data, the system verifies it in real time according to the reporting specification library; for data that does not meet the specifications, the system immediately gives a prompt and requires the user to make corrections.

[0151] The present invention establishes existing data analysis and reporting specification sorting to achieve real-time control of the rationality, completeness and consistency of the reporting content by the data collection system. It can intervene in time for possible reporting errors or anomalies to avoid the generation of invalid data and improve data quality.

[0152] Embodiment 2

[0153] like Figure 3 As shown, an embodiment of the present invention provides a system for improving and controlling the quality of report data, including a data establishment module, a reporting specification library construction module and a correction module;

[0154] The data establishment module is used to obtain report data, the report data includes a text field and a numerical field; establish a data dictionary based on the text field, establish a value range based on the numerical field; and analyze the constraints and logical relationships of the text field and the numerical field to establish specifications;

[0155] The reporting specification library construction model is used to manage the data dictionary, value range and specification in a coding manner to form a reporting specification library;

[0156] The correction module is used to correct the quality of the collected report data based on the reporting specification library.

[0157] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0158] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0160] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0161] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for improving and controlling the quality of report data, characterized in that: The following steps are involved: S100: Acquire report data, the report data including a text field and a numerical field; establish a data dictionary based on the text field, and establish a value range based on the numerical field; and analyzing the constraints and logical relationships of the text fields and the numerical fields to establish specifications; wherein establishing a data dictionary includes: S111: Filter out fields that appear repeatedly from the text fields, sort them, and obtain sorted text fields; S112: Screening out correct field text, incorrect field text, and unrecognizable field text from the sorted text fields; S113: Recording the correspondence between the incorrect field text and the correct field text; S114: establishing a data dictionary based on the correct field text and the incorrect field text corresponding thereto; for unrecognizable field text, marking it as unrecognizable in the data dictionary; The range is established using the 3σ principle of normal distribution and the cumulative frequency method; S200: managing the data dictionary, value range and specification according to the coding method to form a reporting specification library; S300: Correcting the quality of the collected report data based on the reporting specification library, including respectively correcting the text field, the numerical field, and the constraints and logical relationships of the fields based on the reporting specification library, wherein correcting the text field based on the reporting specification library includes: S311: Create a field copy for storing corrected data; S312: Match all the contents of a field in the collected report data with the data dictionary, and when the correct field text is matched, directly copy the correct field text to the field copy; When incorrect field text is matched, the correct field text Chinese corresponding to the incorrect field text is copied to the field copy according to the data dictionary correspondence; When the field fails to match text, the field in the field copy is filled in as unrecognizable.

2. The method for improving and controlling the quality of report data according to claim 1, characterized in that: The step S100 of establishing the value range includes: S121: Determine whether any numerical field conforms to normal distribution. If it conforms to normal distribution, use the 3σ principle of normal distribution to select the interval from μ-3σ to μ+3σ as the reference value range; If it does not conform to the normal distribution, the cumulative frequency selection method is adopted, and the 1% to 99% value of the numerical field distribution is used as the reference value range according to the frequency accumulation method; S122: Determine a final value range based on the reference value range in combination with professional knowledge, thereby obtaining a value range.

3. The method for improving and controlling the quality of report data according to claim 1, characterized in that: The step S100 of establishing the specification includes establishing the specification based on the constraint relationship, and the establishing the specification based on the constraint relationship includes: S131: analyzing whether any text field and numeric field must be filled in, thereby obtaining a non-empty constraint specification for each text field and numeric field; S132: screening multiple key fields for identifying and distinguishing data records, and determining whether the values ​​of the multiple key fields are reported multiple times, thereby obtaining a uniqueness constraint specification for the data records; S133: Based on the report data, the data type and length of the text field and the numerical field are analyzed one by one to determine whether they meet the storage requirements or whether there is a waste of storage space, thereby obtaining the data type and length constraint specification of each text field and numerical field; S134: Obtain a specification based on a constraint relationship based on the non-null constraint specification, the uniqueness constraint specification, the data type and the length constraint specification.

4. The method for improving and controlling the quality of report data according to claim 1, characterized in that: The establishing of the specification in step S100 includes establishing a specification based on a logical relationship, and the establishing of the specification based on a logical relationship includes: According to business requirements and rules, the business logic relationship between fields is analyzed to obtain specifications based on logical relationships.

5. The method for improving and controlling the quality of report data according to claim 1, characterized in that: The step S300 includes correcting the numerical field based on the reporting specification library, including: S321: Create a field copy for storing corrected data; S322: compare all the numerical contents of a field in the collected report data with the value domain, and when the numerical value is within the range of the value domain, directly copy it to the copy of the field; When a value exceeds the range of the value domain, the value is marked as out of range in the field copy.

6. The method for improving and controlling the quality of report data according to claim 1, characterized in that: The step S300 includes correcting the constraints and logical relationships of the fields based on the reporting specification library, including: S331: Create a field copy for storing corrected data; S332: The fields with non-null constraints, unique constraints, data type and length constraints in the collected report data are judged one by one according to the coding order in the reporting specification library. For those that do not violate the constraints, the field content is directly copied to the field copy; For those that violate the constraint, they are marked in the copy of the field; S333: The fields with logical relationships in the collected report data are judged one by one according to the coding sequence in the reporting specification library. For those that do not violate the logical specification, the field content is directly copied to the field copy; Any logical violations are marked in the field copy.

7. The method for improving and controlling the quality of report data according to claim 1, characterized in that: The step S300 also includes controlling the quality of the report data to be collected based on the reporting specification library, including: S31: Embed the reporting specification library into the data collection system through computer development; S32: providing the user with a correct text field based on the data dictionary in the reporting specification library; S33: judging in real time whether the value reported by the user exceeds the value range based on the value range in the reporting specification library; if it does not exceed the value range, the value is allowed to be reported; if it exceeds the value range, the data collection system reminds the user; S34: Based on the specifications in the reporting specification library, determine whether the data to be collected violates the constraints and logical relationships. If the data to be collected does not violate the constraints and logical relationships, reporting is allowed; if the data to be collected violates the constraints and logical relationships, the data collection system reminds the user.

8. A system for improving and controlling the quality of report data, characterized in that: It includes data establishment module, reporting specification library construction module and correction module; The data establishment module is used to obtain report data, which includes text fields and numerical fields; establish a data dictionary based on the text fields, and establish a value domain based on the numerical fields; and analyzing the constraints and logical relationships of the text fields and the numerical fields to establish specifications; wherein establishing a data dictionary includes: S111: Filter out fields that appear repeatedly from the text fields, sort them, and obtain sorted text fields; S112: Screening out correct field text, incorrect field text, and unrecognizable field text from the sorted text fields; S113: Recording the correspondence between the incorrect field text and the correct field text; S114: establishing a data dictionary based on the correct field text and the incorrect field text corresponding thereto; for unrecognizable field text, marking it as unrecognizable in the data dictionary; The range is established using the 3σ principle of normal distribution and the cumulative frequency method; The reporting specification library construction model is used to manage the data dictionary, value range and specification in a coding manner to form a reporting specification library; The correction module is used to correct the quality of the collected report data based on the reporting specification library, including correcting the text field, the numerical field, and the constraints and logical relationships of the fields based on the reporting specification library, wherein the correction of the text field based on the reporting specification library includes: S311: Create a field copy for storing corrected data; S312: Match all the contents of a field in the collected report data with the data dictionary, and when the correct field text is matched, directly copy the correct field text to the field copy; When incorrect field text is matched, the correct field text Chinese corresponding to the incorrect field text is copied to the field copy according to the data dictionary correspondence; When the field fails to match text, the field in the field copy is filled in as unrecognizable.

Images