Multi-source data processing method and device

Abstract

The invention discloses a multi-source data processing method and device. The method comprises the steps of obtaining multi-source data; classifying the multi-source data according to the data source type of the multi-source data, and performing data encoding on the classified data to obtain each type of encoded data; the data coding enables the classified data and the code to form a unique corresponding relation; and performing quality inspection on the coded data to obtain final data. According to the application, automation of the data acquisition process can be realized, and real-time acquisition and instant processing of the data are realized, so that real-time decision and emergency response are supported. By optimizing a data processing flow and adopting a high-performance computing platform, the delay of data acquisition and processing is greatly reduced, and the efficiency and timeliness of information transmission are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a multi-source data processing method and device. BACKGROUND

[0002] At present, in order to meet the application requirements of comprehensive query of meteorological and hydrological data, intelligent prediction of meteorological and hydrological environment, and auxiliary analysis of influence of meteorological and hydrological environment, a meteorological and hydrological big data comprehensive service platform is developed by using multi-source data connection collection, data correlation fusion, GIS data analysis and display, etc. The platform realizes comprehensive query and display of meteorological and hydrological elements from different sources (including military, local and observation equipment) on one network and one map. The platform has the service capabilities of statistical analysis of meteorological and hydrological historical data, statistical analysis of climate background, short-term climate prediction and extended period prediction, and realizes accurate prediction of conventional meteorological and hydrological elements in a predetermined area.

[0003] In weather forecasting, the accuracy and timeliness of data are crucial to the forecasting results. Therefore, how to improve the accuracy and timeliness of data is a technical problem to be solved at present. SUMMARY

[0004] The present disclosure provides a multi-source data processing method and device to at least solve the above technical problems in the prior art.

[0005] According to a first aspect of the present application, a multi-source data processing method is provided, the method comprising:

[0006] acquiring multi-source data;

[0007] classifying the multi-source data according to the data source types of the multi-source data, and data encoding the classified data to obtain encoded data of each type; the data encoding forms a unique correspondence between the classified data and the code;

[0008] quality testing the encoded data to obtain final data.

[0009] In an implementable manner, the acquiring multi-source data comprises:

[0010] setting parameter indicators and task information according to task requirements, and setting the time of acquiring data;

[0011] generating a data receiving task sheet based on the parameter indicators, task information and time;

[0012] acquiring data of the corresponding data source according to the data receiving task sheet.

[0013] In an implementable manner, after acquiring multi-source data, the method further comprises:

[0014] Monitoring the receiving state of the multi-source data; specifically including:

[0015] Monitoring the log information of the data according to the parameter index; the log information includes file name, storage location, receiving time, transfer output time and output state;

[0016] According to the log information, it is judged whether the acquisition state of the parameter index data is normal, if normal, the log information is stored, otherwise, an alarm is given.

[0017] In an implementable manner, the classified data includes:

[0018] Automatic weather station data, weather radar data, cloud radar data and wind profile radar data.

[0019] In an implementable manner, the classified data is encoded, including:

[0020] The classified data is checked for format;

[0021] The data with format error is deleted, and the data with correct format is assigned with code one by one, so that each item of data and code form a unique corresponding relationship.

[0022] In an implementable manner, the encoded data is quality inspected, including:

[0023] The encoded data is checked for integrity, climatology limit value, data table header consistency, element consistency, change range and missing data, to obtain the checked data;

[0024] The checked data is quality marked; the quality mark includes data correct, data suspicious, data error and data without quality control.

[0025] In an implementable manner, the quality mark is represented by code;

[0026] Among them, 0 represents data correct, 1 represents data suspicious, 2 represents data error, and 3 represents data without quality control.

[0027] In an implementable manner, after the quality inspection of the encoded data, it further includes

[0028] The data after quality inspection is preprocessed and standardized.

[0029] In an implementable manner, the data after quality inspection is preprocessed and standardized, including:

[0030] The data after quality inspection is data deduplication, data sorting and format checking, to obtain the processed data;

[0031] standardizing the processing data to obtain final data.

[0032] According to a second aspect of the present application, a multi-source data processing apparatus is provided, the apparatus comprising:

[0033] an acquisition module configured to acquire multi-source data;

[0034] an encoding module configured to classify the multi-source data according to the data source type of the multi-source data, and to perform data encoding on the classified data to obtain encoded data of each type; the data encoding causes the classified data and the code to form a unique corresponding relationship;

[0035] a quality inspection module configured to perform quality inspection on the encoded data to obtain final data.

[0036] According to a third aspect of the present application, an electronic device is provided, comprising:

[0037] at least one processor; and

[0038] a memory in communication connection with the at least one processor; wherein,

[0039] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present application.

[0040] According to a fourth aspect of the present application, a non-transitory computer readable storage medium storing computer instructions is provided, the computer instructions being used to enable the computer to perform the method described in the present application.

[0041] According to a fifth aspect of the present application, a computer program product is provided, comprising computer programs or instructions, which, when executed by a processor, implement the method described in the present application.

[0042] By using the technical solutions of the present application, different types of data sources are classified, the classified data is encoded and quality inspected, the accuracy of the collected data is ensured, and thus the accuracy of the weather forecast result is improved. In addition, encoding the data can enhance the confidentiality of the data information.

[0043] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor to limit the scope of the present application. Other features of the present application will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0044] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description read in conjunction with the accompanying drawings, in which:

[0045] In the drawings, identical or corresponding components are denoted by identical or corresponding reference numerals.

[0046] Figure 1 An implementation flowchart of a multi-source data processing method in an embodiment of the present application is shown Figure 1 ;

[0047] Figure 2 An implementation flowchart of a multi-source data processing method in an embodiment of the present application is shown Figure 2 ;

[0048] Figure 3 An implementation flowchart of a multi-source data processing method in an embodiment of the present application is shown Figure 3 ;

[0049] Figure 4 An implementation flowchart of a multi-source data processing method in an embodiment of the present application is shown Figure 4 ;

[0050] Figure 5 An implementation flowchart of a multi-source data processing method in an embodiment of the present application is shown Figure 5 ;

[0051] Figure 6 An implementation flowchart of a multi-source data processing method in an embodiment of the present application is shown

[0052] Figure 7 An implementation flowchart of a multi-source data processing method in an embodiment of the present application is shown DETAILED DESCRIPTION

[0053] In order to make the purposes, features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0054] In the prior art, when data is acquired, the data is acquired in a multi-source manner, and the following technical problem of resource waste exists: a scheduled task is executed at a preset time regardless of whether the data changes, which can cause waste of system resources. The workflow engine system can more effectively utilize computing resources and execute a task only when necessary through event driving and dependency management.

[0055] Maintenance complexity: The management and maintenance of timed tasks usually require manual modification and updating of scripts, and frequent adjustments when task logic changes or new tasks are added. The workflow engine system simplifies the addition and modification of tasks through abstraction layers and modular design, improving the maintainability of the system.

[0056] Poor scalability: Timed task systems have difficulty in dynamic expansion and adjustment when facing complex and variable business requirements. The workflow engine system can flexibly respond to different business requirements through modularization and configuration management, achieving dynamic expansion and optimization.

[0057] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be described in further detail below with reference to the accompanying drawings, and the described embodiments should not be considered as limiting the present application. All other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0058] As Figure 1 shown, the present application provides a multi-source data processing method, which comprises:

[0059] S101, acquiring multi-source data;

[0060] It can be understood that the multi-source data is collected by multiple collection devices, which can include jurisdictional automatic observation equipment, station observation equipment, HJ special line data collection equipment, satellite data collection system, CMAcast data collection system, etc. It can be understood that the multi-source data processing method provided by the present application can be implemented in a data processing platform or a server.

[0061] For example, in a server, a data collection unit is provided in the server, and the data collection unit is provided with a data receiving unit corresponding to each collection device, for acquiring multi-source data collected by each collection device through a matching interface.

[0062] In some embodiments, the multi-source data is acquired, comprising:

[0063] Setting parameter indicators and task information according to task requirements, and setting data acquisition time;

[0064] Generating a data receiving task sheet based on the parameter indicators, task information and time;

[0065] According to the data receiving task sheet, the data of the corresponding data source is acquired.

[0066] In the present application, each device unit sets parameter indicators and task information according to task requirements, and sets data acquisition time, so as to acquire collection data of each collection device at a preset time.

[0067] For example, the CMAcast data receiving unit corresponding to the CMAcast data acquisition system regularly receives global meteorological data broadcast by the China Meteorological Administration via satellite broadcast, enabling storage, query browsing, and monitoring management of the received files. The CMAcast data receiving unit can obtain live data, satellite data, weather radar data, and domestic and international numerical forecast products. After acquiring the data, the CMAcast data receiving unit stores the data, where relevant transmission information such as the reception time, number of received bytes, number of received files, and current file name is formed into a log. This application also allows for recording and querying of log information.

[0068] The CMAcast data receiving unit in this application can be configured based on task-related parameters and task information. A scheduled reception schedule can be set, and the server can automatically search for messages within the 72 hours before the program startup time to avoid data loss caused by unexpected server program interruptions. The input and output of the CMAcast data receiving unit are shown in Table 1.

[0069] Table 1 CMACast data receiving unit input and output table

[0070]

[0071] In summary, if Figure 2 As shown in the figure, the CMAcast data receiving unit performs the receiving task according to the configured parameter information. When the task time arrives, it receives live data, satellite data, weather radar products, domestic and foreign numerical forecast products and other satellite broadcast meteorological products, stores the received data files, and can query and browse related transmission information such as the receiving time, number of received bytes, number of received files, current file name, etc., and can record and query log information.

[0072] In some embodiments, after acquiring multi-source data, the process further includes:

[0073] Monitor the status of receiving data from multiple sources; specifically including:

[0074] Monitor the log information of data obtained according to parameter indicators; the log information includes file name, storage location, reception time, ferry output time and output status;

[0075] It is determined whether the acquisition status of the parameter indicator acquisition data is normal according to the log information. If it is normal, the log information is stored; otherwise, an alarm is issued.

[0076] It can be understood that the server also has a data monitoring unit for monitoring the collected data. For example, monitoring whether various parameter indicators are normal. Monitor the data file receiving situation, file name, storage location, receiving time, ferry output time, output state and other data receiving conditions, and provide cache data directory management for temporarily stored received data.

[0077] As shown in Figure 3 The data monitoring unit in the present application can obtain log information of various parameter indicator states, monitor the obtained data situation through the log information, judge whether the obtained data state is normal, such as whether the data can be obtained, whether the data is repeated, etc. If normal, store the obtained data and record the log, otherwise, in the case of data exception, perform exception alarm and record the exception alarm to the log.

[0078] Among them, the monitored information includes: log information of the collection state, log information of various parameter indicators, local and domestic observation real-time data information, foreign city weather real-time data file data information, satellite data file data information, weather radar data file data information, wind profile radar data file data information, numerical prediction product file data information, etc. The above monitoring information can be cached to the data directory management, and when it is needed to be queried, it can be viewed in the data directory management.

[0079] S102, according to the data source type of the multi-source data, the multi-source data is classified, and the classified data is data encoded, and the encoded data of each type is obtained; the data encoding makes the classified data and the code form a unique corresponding relationship;

[0080] It can be understood that the data source type includes automatic weather station, weather radar, cloud measuring radar and wind profile radar, so that the multi-source data can be classified into automatic weather station data, weather radar data, cloud measuring radar data and wind profile radar data. After classifying the multi-source data, one code is assigned to each item of specific information, so that each item of specific information and the code form a unique corresponding relationship, providing a short and convenient symbol structure for data recording, access and retrieval, thereby facilitating information processing and information exchange, improving the efficiency and accuracy of data processing, and enhancing the confidentiality of information.

[0081] Specifically, the coding of the meteorological measured data in the present application should comply with the provisions of GB7027-88, and be coded and classified according to the face line combination mode. The meteorological data content attribute classification coding is the main face, and other attribute classification is the auxiliary face.

[0082] In some embodiments, the data encoding of the classified data includes:

[0083] The formatted data is checked.

[0084] The data with format error is deleted, and the data with correct format is assigned with code, so that each item of data forms a unique corresponding relationship with the code.

[0085] Specifically, as shown in the figure, when data encoding is performed, the data is first classified, then the data is subjected to format detection and corresponding error correction processing to obtain data materials with correct format, arrangement rules and being able to be correctly encoded; the meteorological data content attribute classification encoding is mainly performed to encode the measured data, so that each specific information forms a unique corresponding relationship with the code; and the measured data encoding is completed. Figure 4

[0086] S103, quality inspection is performed on the encoded data to obtain final data

[0087] In some embodiments, the quality inspection on the encoded data comprises:

[0088] The encoded data is subjected to integrity check, climatology limit value check, data table header consistency check, element consistency check, change range check and missing data check to obtain the checked data.

[0089] Quality identification is performed on the checked data; the quality identification comprises data correct, data suspicious, data error and data without quality control.

[0090] It should be noted that the quality inspection on the encoded data mainly performs rationality and consistency check on the received and collected meteorological and hydrological data, and through giving quality control score, the objective marking of the data quality state is realized. The score information can be stored in the meteorological and hydrological database together with related data.

[0091] The integrity check checks the file naming, format and data completeness of the observation data. The integrity and non-missingness of the data file, and the product file name and format compliance with the specification requirements are checked in sequence. Through checking whether the file classification code is accurate, whether the file is empty, whether the data storage format is unified and standard, etc., the data format is ensured to be correct to meet the needs of subsequent data set production. When the data is missing, the missing information is written into the data management log, and the replaceable data related information is provided.

[0092] ​Climatological limit value checks examine meteorological records for exceeding climatic extremes. Climatic extremes are defined as those occurring with a very low probability at a fixed meteorological station within a certain timeframe. Climatological limit value checks examine the possible extreme limits of climatological variation within a given element. Data outside these limits are considered erroneous. For example, the wind direction range of 0 to 360° can be automatically controlled by a computer. Monthly extreme value checks examine meteorological and hydrological elements at different locations and months. Any data outside these ranges is considered erroneous.

[0093] The consistency check of the data header is that the record type, ocean station code, geographical location and other information of the data should be consistent; otherwise, an error message will be output and the corresponding element will be marked with the quality control symbol "2".

[0094] Element consistency checks leverage the inherent connections between meteorological and oceanographic elements to verify the compatibility of elements within each observation record. For example, ocean temperature differences generally should not exceed a certain range; strong winds are associated with high waves. If inconsistencies arise, causal relationships are used to determine the appropriate balance. Because surface wind speed and wave height are key elements in compiling meteorological and oceanographic data from the three oceans, they are also prioritized in this consistency check. Elements that fail this check are marked with a "5" quality control symbol.

[0095] The range check is responsible for checking the range of data within the specified sea area and time domain according to the elements. Data that exceeds the range of element changes is suspicious and should be further checked to determine whether the data is correct. The range check methods include: extreme value control method and Rhineda test method ( 3 δ test method).

[0096] Among them, the extreme value control method is to determine the maximum and minimum values ​​of each factor based on the physical properties of each factor and statistical experience. Extreme value control is a commonly used, simple and effective method that can effectively detect extremely abnormal data. However, the extreme value control method cannot effectively detect abnormal data with relatively small differences, which requires more precise control methods. Extreme value control methods mainly include:

[0097] Based on the basic characteristics of the feature, its range of occurrence is judged. There are two types of extreme values ​​used in range checking: one is an outlier (u or U), and the other is a value that is physically impossible to occur (i or I).

[0098] Assuming that the historical minimum and maximum values ​​of factor X are r and R respectively, generally speaking:

[0099] i m =r m -Δ<r m <um m m m +Δ=I m

[0100] m is the month variable. The determination of (i, I) and (u, U) needs to be made according to the probability distribution characteristics of the element X.

[0101] The principle of this method is to determine whether the data is within the extreme value range of the site. The average value (M) plus or minus the standard deviation (σ) of the single-month data is used for quality control.

[0102] The multiple of the standard deviation is determined by experience: if the observation data is not between M±4σ, but is between M±8σ, it is considered to be suspicious data; if it is outside M±8σ, it is considered to be erroneous data.

[0103] where, 3 The δ test method is also known as the Lindley test method. According to error theory, random error δ generally follows a normal distribution. δ is the standard deviation, which is generally unknown, and is usually replaced by S calculated by the Bessel formula, and x is replaced by the true value.

[0104]

[0105] x i is the observation value, i=1,2,K,N

[0106]

[0107] For a certain observation data x i , if its residual error v i satisfies v i =|x i -x|>3S, i=1,2,K,N, then x i is suspicious and should be marked as an abnormal value.

[0108] Calculate the extreme value using the Lindley criterion. Remove the absolute value from the formula |x i -x|>3S.

[0109] x i >x+3S(x i -x≥0)

[0110] x i >x-3S(x i -x<0)

[0111] When a certain observation data x i satisfies the above conditions, the observation value x i is erroneous data.

[0112] ​​​The prerequisite for using the "Rhineda criterion" is to have a large enough number of observations. When N < 10, it is easy to fail and is generally not used. Otherwise, the probability of "discarding a true hypothesis" is high. When N > 300, the probability of "discarding a true hypothesis" α is stable at 0.003. Table 2 shows the relationship between the number of observations N and the probability of "discarding a true hypothesis" α for the "Rhineda criterion":

[0113] Table 2. Rejection rate of the Rhineda criterion

[0114] N 11 16 61 121 333 α 0.019 0.011 0.005 0.004 0.003

[0115] The missing check mainly checks for some data that are missing in the message. All elements must be checked for missing and unknown elements. If an element is missing or unknown, the identifier "9" is added to the quality control character corresponding to the element, and the missing element is replaced by "9999".

[0116] For a suspected element value, the continuity check compares it with the values ​​of the same station at the same time on the days before and after the element value to determine if the difference exceeds the permissible daily variation range. Alternatively, the difference between the two observations before and after the element value is considered correct. Observations that do not meet this condition are marked with a "4" quality control symbol. The difference between the two observations before and after the tide level must be ≤ 200.0 cm, the difference between two scheduled pressure observations must be ≤ 50.0 hPa, the difference between two scheduled temperature observations must be ≤ 10.0°C, and the difference between two scheduled dew point observations must be ≤ 7.0°C.

[0117] Specifically, such as Figure 5 As shown, this application obtains classified coded data, then selects an appropriate data quality inspection method based on the classified coded data. After the data undergoes unified quality control processing such as integrity check, limit value check, variation range check, spatial consistency check, temporal consistency check, and data quality identification, quality control scoring is performed on the data, and quality control identification is performed based on the quality control score. The quality control identification mainly provides information about the quality of the data to the user of the data. The identification of data quality includes: correct, suspicious, incorrect, and not quality controlled, which are represented by the following four codes: 0: data is correct; 1: data is suspicious; 2: data is incorrect; 3: data is not quality controlled.

[0118] In some embodiments, after the quality inspection of the coded data is performed, the method further includes:

[0119] The data after quality inspection is preprocessed and standardized.

[0120] In some embodiments, the data after quality inspection is preprocessed and standardized, including:

[0121] After quality inspection, the data is deduplicated, sorted, and formatted to obtain processed data;

[0122] standardizing the processing data to obtain final data.

[0123] Specifically, the data preprocessing mainly includes data deduplication, data sorting, and format detection, which is mainly for preprocessing various meteorological and hydrological data; the data standardization processing includes term standard, data element standard, and information classification and coding standard to obtain final data.

[0124] The multi-source data processing method provided by the application introduces machine learning and artificial intelligence technology to realize the automation of the data collection process. For example, the optimal data source and collection strategy are automatically selected, abnormal data are automatically identified and processed, and the collection efficiency and data quality are optimized through learning. Real-time data collection and immediate processing are realized to support real-time decision-making and emergency response. Through optimizing the data processing flow and using high-performance computing platforms, the delay of data collection and processing is greatly reduced, and the efficiency and timeliness of information transmission are improved. Different types and sources of data are integrated and analyzed, such as the fusion of structured data, semi-structured data, and unstructured data. Through intelligent algorithms and data mining technology, the relevance and patterns in the data are extracted and analyzed to improve the value and depth of insights. An intelligent data quality management system is developed to detect and repair data quality problems through automated tools and algorithms. This includes identifying and processing noise, missing values, outliers, and other data to ensure data consistency, integrity, and accuracy. Intelligent data collection technology is applied to different fields and industries, such as medical health, finance, and logistics. Through adaptable technology and platforms, customized needs of different industries and responses to complex data environments are supported. Efficient privacy protection technology is developed and applied to ensure data security during data collection, transmission, and processing. This includes the application of data encryption, access control, and identity verification technologies to meet data security and compliance requirements.

[0125] As shown in Figure 6 The embodiment of the application provides a multi-source data processing device, which comprises:

[0126] The acquisition module 601 is configured to acquire multi-source data.

[0127] The encoding module 602 is configured to classify the multi-source data according to the data source type of the multi-source data, and to encode the classified data to obtain encoded data of each type; the data encoding forms a unique correspondence between the classified data and the code.

[0128] The quality inspection module 603 is configured to perform quality inspection on the encoded data to obtain final data.

[0129] The multi-source data processing device provided in the application obtains multi-source data through an obtaining module 601; an encoding module 602 classifies the multi-source data according to the data source types of the multi-source data, and performs data encoding on the classified data to obtain encoded data of each type; the data encoding causes the classified data and the code to form a unique corresponding relationship; a quality inspection module 603 performs quality inspection on the encoded data to obtain final data.

[0130] According to the embodiments of the application, the application further provides an electronic device and a readable storage medium.

[0131] The electronic device includes at least one processor, and a memory connected with the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multi-source data processing method of the application. The computer instructions are used to enable the computer to perform the multi-source data processing method of the application.

[0132] The application further provides a computer program product, including computer programs / instructions, which, when executed by a processor, implement the multi-source data processing method of the application.

[0133] Figure 7 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the application described and / or claimed in this document.

[0134] As shown in Figure 7 The device 800 includes a computing unit 801 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0135] A plurality of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0136] The computing unit 801 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as the multi-source data processing method. For example, in some embodiments, the multi-source data processing method can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded onto the RAM 803 and executed by the computing unit 801, one or more steps of the multi-source data processing method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the multi-source data processing method by any other appropriate means, such as by means of firmware.

[0137] Various implementations of the systems and techniques described above herein can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0138] Program code for carrying out methods of the present application can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be retrieved from a machine-readable medium or device and executed by a processor to produce a machine for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed as a stand-alone program, or in combination with other program codes, on the machine to produce a machine that implements the functions / acts specified in the flowcharts and / or block diagrams.

[0139] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage media can include, without limitation, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include, but are not limited to, an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0140] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0141] The systems and techniques described herein can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0142] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0143] The above description is provided as an enabling teaching of the application and is not intended to limit the scope of the application. Any modification of the application in keeping with the spirit of the application falls within the scope of the application. The scope of the application is defined by the appended claims.

Claims

1. A multi-source data processing method, characterized in that: The method comprises: Acquire data from multiple sources; Classifying the multi-source data according to the data source type of the multi-source data, and performing data encoding on the classified data to obtain coded data of each type; the data encoding forms a unique correspondence between the classified data and the code; Performing a quality check on the encoded data to obtain final data.

2. The method according to claim 1, characterized in that The acquiring of multi-source data includes: Set parameter indicators and task information according to task requirements, and set the time for obtaining data; Generate data based on the parameter indicators, task information and time to receive the task order; The data of the corresponding data source is obtained according to the data receiving task list.

3. The method according to claim 2, characterized in that After acquiring multi-source data, it also includes: Monitor the status of receiving data from multiple sources; specifically including: Monitor the log information of data obtained according to parameter indicators; the log information includes file name, storage location, reception time, ferry output time and output status; It is determined whether the acquisition status of the parameter indicator acquisition data is normal according to the log information. If it is normal, the log information is stored; otherwise, an alarm is issued.

4. The method according to claim 1, wherein The classified data includes: Automatic weather station data, weather radar data, cloud detection radar data and wind profiler radar data.

5. The method according to claim 1, wherein Data encoding is performed on the classified data, including: Perform format check on the classified data; Delete the data with incorrect format and assign codes to the data with correct format one by one, so that each data and code form a unique correspondence.

6. The method according to claim 1, wherein The performing quality inspection on the encoded data includes: Performing integrity checks, climatological limit value checks, data header consistency checks, element consistency checks, variation range checks, and missing measurement checks on the coded data to obtain checked data; The checked data is quality-labeled; the quality labels include correct data, suspicious data, wrong data, and data that has not undergone quality control.

7. The method according to claim 6, characterized in that Use codes to indicate quality marks; Among them, 0 represents correct data, 1 represents suspicious data, 2 represents incorrect data, and 3 represents data without quality control.

8. The method according to claim 1, characterized in that After the quality inspection of the coded data, the method further includes: The data after quality inspection is preprocessed and standardized.

9. The method according to claim 8, characterized in that Preprocess and standardize the data after quality inspection, including: After quality inspection, the data is deduplicated, sorted, and formatted to obtain processed data; The processed data are standardized to obtain final data.

10. A multi-source data processing device, characterized in that: The device comprises: Acquisition module, used to acquire multi-source data; An encoding module, configured to classify the multi-source data according to the data source type of the multi-source data, and perform data encoding on the classified data to obtain coded data of each type; the data encoding forms a unique correspondence between the classified data and the code; The quality inspection module is used to perform quality inspection on the encoded data to obtain final data.

Images