Data processing method and device, equipment and medium
Through the combination of graphical data extraction templates, data set configuration information, labeling model library and target task conditions, efficient and automatic processing of data is achieved, solving the problems of high labor costs and low compatibility of traditional data processing methods, and meeting the needs of complex and large-scale data processing.
Patent Information
- Application Number
- CN202411630322.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-14
- Publication Date
- 2025-02-28
AI Technical Summary
Traditional data processing methods rely on manual operations, resulting in high labor costs, single data sources, low compatibility and flexibility, which cannot meet the needs of complex and large-scale data processing.
Data extraction is performed on multiple data sources based on the graphical data extraction template, data division is performed based on the data set configuration information, data annotation model library is used to perform data annotation, and data statistics are performed based on the target task conditions to generate data statistics results.
It realizes efficient and automatic data processing, reduces labor costs, improves data processing compatibility and flexibility, and meets the needs of complex and large-scale data processing.
Smart Images
Figure CN119537663A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a data processing method, device, equipment and medium. Background Art
[0002] With the advent of the Internet age and the mobile age, the cost of data generation and dissemination has dropped sharply, resulting in an explosive growth in the amount of data. In this environment, the requirements for data processing are also getting higher and higher.
[0003] At present, traditional data processing methods often rely on technicians to access and process data according to fixed rules. However, traditional data processing methods have high labor costs, relatively single data sources, low compatibility and flexibility of data processing, and thus cannot meet complex and large-scale data processing needs. Summary of the invention
[0004] The present invention provides a data processing method, device, equipment and medium to achieve efficient and automatic data processing, which can reduce labor costs, improve the compatibility and flexibility of data processing, and thus meet complex and large-scale data processing needs.
[0005] In a first aspect, an embodiment of the present invention provides a data processing method, including:
[0006] Based on the graphical data extraction template, data is extracted from multiple data sources corresponding to the target object to determine the original data corresponding to the target object;
[0007] Based on the data set configuration information, the original data is divided into data to determine the original data set corresponding to the target object;
[0008] Based on the annotation model library, annotate the unannotated items in the original data set to obtain a target data set corresponding to the target object;
[0009] Data statistics are performed based on the data statistical conditions corresponding to the target task and the target data set corresponding to the target object to generate the data statistical results corresponding to the target task.
[0010] In a second aspect, an embodiment of the present invention further provides a data processing device, including:
[0011] The original data determination module is used to extract data from multiple data sources corresponding to the target object based on the graphical data extraction template to determine the original data corresponding to the target object;
[0012] A data partitioning module, used to partition the original data based on the data set configuration information, and determine the original data set corresponding to the target object;
[0013] A data annotation module is used to annotate the unannotated items in the original data set based on the annotation model library to obtain a target data set corresponding to the target object;
[0014] The data statistics module is used to perform data statistics based on the data statistics conditions corresponding to the target task and the target data set corresponding to the target object, and generate the data statistics results corresponding to the target task.
[0015] In a third aspect, an embodiment of the present invention further provides an electronic device, characterized in that the electronic device comprises: at least one processor; and
[0016] a memory communicatively connected to the at least one processor; wherein,
[0017] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can execute the data processing method provided by any embodiment of the present invention.
[0018] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the data processing method provided by any embodiment of the present invention when executing.
[0019] The technical solution of the embodiment of the present invention extracts data from multiple data sources corresponding to the target object based on the graphical data extraction template, determines the original data corresponding to the target object, and improves the accuracy and efficiency of data extraction. Based on the data set configuration information, the original data is divided to determine the original data set corresponding to the target object, thereby organizing the data into a more orderly and manageable structure. Based on the annotation model library, the unannotated items in the original data set are annotated to obtain the target data set corresponding to the target object, thereby improving the accuracy and efficiency of data annotation. Based on the data statistical conditions corresponding to the target task and the target data set corresponding to the target object, data statistics are performed to generate the data statistical results corresponding to the target task, and the data performance of the target object under different dimensions and indicators is displayed. The graphical interface makes the data extraction process more intuitive and easy to understand, improves the accuracy and efficiency of data extraction, divides the data into original data sets to help organize the data into a more orderly and manageable structure, introduces the annotation model library to improve the accuracy and efficiency of data annotation, and the task data statistics provide a comprehensive and in-depth understanding of the data performance of the target object, improves the compatibility and flexibility of data processing, and thus meets the complex and large-scale data processing requirements.
[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0022] Figure 1 is a flow chart of a data processing method provided according to Embodiment 1 of the present invention;
[0023] Figure 2 is a flow chart of a data processing method provided according to Embodiment 2 of the present invention;
[0024] Figure 3 is a structural schematic diagram of a data processing device provided according to Embodiment 3 of the present invention;
[0025] Figure 4 It is a structural schematic diagram of an electronic device for implementing the data processing method of an embodiment of the present invention. DETAILED DESCRIPTION
[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0027] It should be noted that the terms "target", "current", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0028] Embodiment 1
[0029] Figure 1 A flowchart of a data processing method is provided for the first embodiment of the present invention. This embodiment is applicable to the case where data is automatically processed. Figure 1 As shown, the method can be executed by a data processing device, which can be implemented in the form of hardware and / or software, and the data processing device can be configured in an electronic device. Figure 1 As shown, the method specifically comprises the following steps:
[0030] S110 . Based on the graphical data extraction template, extract data from multiple data sources corresponding to the target object to determine the original data corresponding to the target object.
[0031] Among them, the data extraction template may refer to a tool or framework for guiding how to extract data from a data source, which may be based on a process. The target object may refer to the main entity or object that the data extraction task focuses on or analyzes, which may be anything with analytical value, such as an enterprise, project, user group, product, etc. The data source may refer to an entity or system that stores or provides data. For example, a data source may be a database, file, API interface, third-party data service, etc. Raw data may refer to data extracted from a data source without being processed or analyzed.
[0032] Specifically, the data extraction template is processed into a graphic so that the template can intuitively display the data extraction process and the required information, and the graphical data extraction template is used to extract data from multiple data sources (such as databases, files, network data, etc.) corresponding to the target object (such as a certain enterprise, project or individual). During the extraction process, according to the rules and logic preset in the template, the original data related to the target object is screened from the data source, which improves the accuracy and efficiency of data extraction.
[0033] S120: Based on the data set configuration information, divide the original data into data sets to determine the original data set corresponding to the target object.
[0034] The dataset configuration information may refer to various parameters and settings used to describe, manage, and operate the dataset. Such information is usually used to guide the creation, loading, processing, and analysis of the dataset. The raw dataset may refer to a set of data directly obtained from the data source, without processing or only after preliminary processing, divided according to the data and configuration information.
[0035] Specifically, the original data is divided according to the data set configuration information (such as data type, format, storage location, etc.). During the division process, the data can be cleaned, converted, and integrated to ensure the accuracy and consistency of the data. Finally, the original data set corresponding to the target object is determined based on the division results. Data division helps to organize the data into a more orderly and manageable structure.
[0036] Exemplarily, S120 may include: performing data cleaning and normalization preprocessing on the original data to obtain preprocessed target data; performing data partitioning on the target data based on data set configuration information to determine the original data set corresponding to the target object, wherein the data set configuration information includes: at least one of: data type, data format, data version and storage location.
[0037] Among them, target data may refer to data extracted from raw data after pre-processing steps such as data cleaning and normalization. Data type may refer to the way or category of storing, processing and representing data in a computer. Data format may refer to the format of data arrangement in a file or record, including the arrangement of data, separators, data type identifiers, etc. Data version may refer to different versions of data generated after multiple modifications to the same data object in a database. Storage location may refer to the specific location or path where data is stored in a computer system.
[0038] Specifically, check the missing values in the original data, and according to the guidance in the dataset configuration information, choose to delete the missing values, fill in estimated values (such as mean, median, etc.), or construct new variables to fill in the missing values. Identify and process outliers in the original data, such as deleting outliers and replacing them with reasonable values. Ensure that the data type in the original data is consistent with the requirements in the dataset configuration information, and perform necessary type conversions, such as converting text types to numeric types, converting date and time types to timestamps, etc. Scale the features in the original data, such as normalization or standardization. According to the guidance in the dataset configuration information, select the most useful features for target object analysis from the original data, and delete irrelevant or redundant features. According to the guidance in the dataset configuration information, determine the data partitioning strategy, such as the ratio of data partitioning into training sets, validation sets, and test sets. Divide the preprocessed target data according to the determined strategy to generate the original dataset corresponding to the target object. Through the automated and standardized data partitioning process, the error rate of manual operation and data processing can be reduced, and the efficiency and accuracy of data processing can be improved.
[0039] For example, data cleaning of raw data can be to remove invisible characters, normalize spaces, and remove possible garbled characters. In addition, it also has the function of converting traditional Chinese to simplified Chinese, removing identifiers in web pages, and cleaning up various emoticons. At the same time, it supports the removal of sensitive information such as email addresses, Internet Protocol (IP) addresses, and mobile phone numbers to ensure data security and compliance, and provide a purer and more reliable data foundation for subsequent data processing and analysis.
[0040] S130: Based on the annotation model library, annotate the unannotated items in the original data set to obtain a target data set corresponding to the target object.
[0041] The labeled model library may refer to a database for storing multiple labeled models. Unlabeled items may refer to data items that have not yet been assigned labels or annotations. The target dataset may refer to a data set for a specific data analysis task, which has been preprocessed and labeled and meets the requirements of the task.
[0042] Specifically, the annotation model library contains a variety of annotation models for data annotation processing. According to the needs of the target object and the data characteristics, a suitable annotation model is selected to annotate the unlabeled items in the original data set to obtain the target data set corresponding to the target object. The data set contains the annotated data, which improves the accuracy and efficiency of data annotation.
[0043] Exemplarily, S130 may include: screening the annotation model library based on the original data set corresponding to the target object to determine the annotation model corresponding to the original data set; detecting and annotating unlabeled items in the original data set based on the annotation model corresponding to the original data set to obtain the target data set corresponding to the target object.
[0044] The labeling model may refer to a neural network model used for data labeling.
[0045] Specifically, according to the target object and requirements, a suitable annotation model is selected from the existing annotation model library. The annotation model library may contain various types of models, such as image recognition models, text classification models, etc. The selected annotation model is evaluated using the original data set corresponding to the target object, including indicators such as the model's accuracy and recall rate. Through the evaluation, it can be determined which models are more suitable for the annotation task. According to the evaluation results, the annotation model with the best performance is selected as the annotation model corresponding to the original data set. According to the annotation model corresponding to the original data set, unlabeled items are identified from the original data set, and these items are the targets to be labeled. The unlabeled items are labeled using the loaded annotation model, and the labeled data are integrated to form the target data set corresponding to the target object. By utilizing the existing annotation model library, the annotation time can be greatly shortened, the annotation efficiency can be improved, and a large amount of data can be processed faster.
[0046] Exemplarily, the annotation model library is screened for annotation models based on the original data set corresponding to the target object to determine the annotation model corresponding to the original data set, including: determining the data format corresponding to the original data set based on the original data set corresponding to the target object; determining the annotation model corresponding to the original data set based on the mapping relationship between the annotation model and the data format in the annotation model library and the data format corresponding to the original data set.
[0047] Specifically, according to the characteristics and structure of the original data set, its corresponding data format is identified. The data format may include text format, image format, audio format, etc. The purpose of identifying the data format is to ensure that the data can be correctly processed and parsed in subsequent steps. In the annotation model library, each annotation model is usually associated with the data format it supports. Therefore, a mapping relationship between the annotation model and the data format can be established in advance. According to the data format corresponding to the original data set, a matching annotation model is found in the mapping relationship. If there are multiple annotation models that support the same data format, it may be necessary to further consider factors such as the model's performance, accuracy, and stability to select the most appropriate annotation model. By identifying the data format, you can quickly find a annotation model that matches the original data set from the annotation model library, thereby improving annotation efficiency.
[0048] S140: Perform data statistics based on the data statistical conditions corresponding to the target task and the target data set corresponding to the target object to generate a data statistical result corresponding to the target task.
[0049] Among them, the target task can refer to the specific, measurable, achievable, and time-limited work or tasks set by individuals or organizations, which are usually used to measure the actions required to achieve specific goals. Data statistical conditions can refer to the conditions or constraints set when conducting data statistical analysis. These conditions may involve multiple aspects such as the source, scope, time, and quality of the data. Data statistical results can refer to the results obtained after statistical analysis of the original data based on the data statistical conditions.
[0050] Specifically, according to the data statistical conditions corresponding to the target task, the target data set is statistically processed. During the statistical process, a variety of statistical methods and tools can be used to conduct in-depth analysis and mining of the data. Finally, the data statistical results corresponding to the target task are generated, which can show the data performance of the target object under different dimensions and indicators.
[0051] Exemplarily, S140 may include: based on the data statistical conditions corresponding to the target task, performing data statistical processing on the target data set corresponding to the target object, generating data statistical results corresponding to the target task, and presenting the data statistical results in the form of a report; wherein the data statistical conditions include: at least one of: statistical dimension, statistical indicator and statistical method.
[0052] Among them, statistical dimensions can refer to different aspects or characteristics used to describe and analyze data. For example, statistical dimensions can be time dimensions, geographical latitudes, etc. Statistical indicators can refer to categories and statistical values that reflect the overall comprehensive quantitative characteristics of a phenomenon, and are used to measure and describe the performance of the target object in different dimensions. For example, in sales data analysis, statistical indicators can be sales, market share, user growth rate, etc. Statistical methods can refer to methods related to collecting, collating, analyzing and interpreting statistical data, and drawing certain conclusions about the issues they reflect. For example, statistical methods can be descriptive statistics, inferential statistics, regression analysis, variance analysis, etc.
[0053] Specifically, according to the target tasks and data statistical conditions, the target data sets corresponding to the target objects are collected. These data sets should contain all the data points that need to be counted and analyzed to ensure the comprehensiveness and accuracy of the data statistical results. Use appropriate statistical methods to perform data statistical processing on the target data sets. According to the data statistical conditions, the processed data are calculated and summarized to generate the data statistical results corresponding to the target tasks. These results may include values, charts, trend lines, etc. of various statistical indicators. The data statistical results are presented in the form of reports, including elements such as titles, tables, and charts, so that the meaning of the data statistical results and the information behind them can be accurately understood.
[0054] The technical solution of the embodiment of the present invention extracts data from multiple data sources corresponding to the target object based on the graphical data extraction template, determines the original data corresponding to the target object, and improves the accuracy and efficiency of data extraction. Based on the data set configuration information, the original data is divided to determine the original data set corresponding to the target object, so that the data is organized into a more orderly and manageable structure. Based on the annotation model library, the unannotated items in the original data set are annotated to obtain the target data set corresponding to the target object, thereby improving the accuracy and efficiency of data annotation. Data statistics are performed based on the data statistical conditions corresponding to the target task and the target data set corresponding to the target object, and the data statistical results corresponding to the target task are generated, showing the data performance of the target object under different dimensions and indicators. The graphical interface makes the data extraction process more intuitive and easy to understand, improves the accuracy and efficiency of data extraction, divides the data into original data sets to help organize the data into a more orderly and manageable structure, introduces the annotation model library to improve the accuracy and efficiency of data annotation, and the task data statistics provide a comprehensive and in-depth understanding of the data performance of the target object, improves the compatibility and flexibility of data processing, and thus meets the complex and large-scale data processing requirements.
[0055] Embodiment 2
[0056] Figure 2 This is a flow chart of a data processing method provided in Embodiment 2 of the present invention. Based on the above embodiments, this embodiment optimizes the step of "extracting data from multiple data sources corresponding to the target object based on the graphical data extraction template to determine the original data corresponding to the target object". The explanations of the terms that are the same or corresponding to the above embodiments are not repeated here.
[0057] See also Figure 2 Another data processing method provided in this embodiment specifically includes the following steps:
[0058] S210: Analyze the target object and multiple data sources corresponding to the target object, and determine a data processing method corresponding to each data source.
[0059] Specifically, identify all data sources related to the target object, which may include databases, files, API interfaces, third-party data services, etc. Conduct in-depth analysis of each data source to understand its data structure, data format, data quality, data update frequency and other characteristics. Determine the appropriate data processing method based on the characteristics of the data source, such as whether the data needs to be cleaned, whether the data format needs to be converted, whether the data needs to be merged or split, etc. Ensure data accuracy and consistency by flexibly configuring data processing methods.
[0060] S220: configuring information of the graphical data extraction template based on the data processing method, and determining a target data extraction template corresponding to each data source.
[0061] The target data extraction template may refer to a tool or framework configured to extract target data from a specific data source.
[0062] Specifically, according to the characteristics of the target object and data source, select an existing graphical data extraction template. Configure information related to the data source and data processing method in the template, such as data source connection information, data field mapping information, data processing rules, etc. After the configuration is completed, verify the template to ensure that the template can correctly connect to the data source and extract and process data in the expected way to obtain the target data extraction template, thereby meeting the data extraction needs in different scenarios.
[0063] S230: Extract data from the data source based on the target data extraction template to obtain original data corresponding to the target object.
[0064] Specifically, use the configured target data extraction template to run the data extraction task and extract data from the data source. After the task is completed, the original data corresponding to the target object is obtained. The graphical interface and configuration method enable non-technical personnel to understand and operate the data extraction process, reducing the difficulty of data extraction.
[0065] Exemplarily, S230 may include: determining a data acquisition strategy in the target data acquisition template that corresponds to a data management strategy of the data source; and screening data related to the target object in the data source based on the data acquisition strategy to obtain original data corresponding to the target object.
[0066] Specifically, analyze the management requirements of the data source, such as data access rights, data security, and data backup and recovery strategies. According to the data management strategy of the data source, formulate a data acquisition strategy that matches it. The data acquisition strategy can be to determine the time window for data extraction, the frequency of data extraction, the method of data extraction (such as batch extraction, real-time stream processing), etc. According to the data acquisition strategy, configure the parameters and rules of the target data extraction template. For example, set data filtering conditions, such as time range, data field, data format, etc. Use the configured target data extraction template to filter the data in the data source. According to the filtering conditions, extract the original data related to the target object, so that the data related to the target object can be accurately extracted to avoid data redundancy or omission.
[0067] S240: Based on the data set configuration information, divide the original data into data sets to determine the original data set corresponding to the target object.
[0068] S250: Based on the annotation model library, annotate the unannotated items in the original data set to obtain a target data set corresponding to the target object.
[0069] S260: Perform data statistics based on the data statistical conditions corresponding to the target task and the target data set corresponding to the target object to generate data statistical results corresponding to the target task.
[0070] The technical solution of the embodiment of the present invention determines the data processing method corresponding to each data source by analyzing the target object and the multiple data sources corresponding to the target object. The data processing method can be flexibly configured according to the characteristics of different data sources and the needs of the target object to ensure the accuracy and consistency of the data. The graphical data extraction template is configured with information based on the data processing method to determine the target data extraction template corresponding to each data source, so that the data extraction task can be intuitively configured and managed. Data is extracted from the data source based on the target data extraction template to obtain the original data corresponding to the target object, greatly improving the efficiency of data extraction. Through the graphical data extraction template, data extraction tasks can be intuitively configured and managed, greatly improving the efficiency of data extraction, and the data processing method can be flexibly configured according to the characteristics of different data sources and the needs of the target object to ensure the accuracy and consistency of the data.
[0071] Embodiment 3
[0072] Figure 3 This is a schematic diagram of the structure of a data processing device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: an original data determination module 310, a data division module 320, a data labeling module 330 and a data statistics module 340.
[0073] The original data determination module 310 is used to extract data from multiple data sources corresponding to the target object based on the graphical data extraction template to determine the original data corresponding to the target object;
[0074] A data partitioning module 320 is used to partition the original data based on the data set configuration information to determine the original data set corresponding to the target object;
[0075] A data annotation module 330 is used to annotate the unannotated items in the original data set based on the annotation model library to obtain a target data set corresponding to the target object;
[0076] The data statistics module 340 is used to perform data statistics based on the data statistics conditions corresponding to the target task and the target data set corresponding to the target object, and generate the data statistics results corresponding to the target task.
[0077] The technical solution of this embodiment extracts data from multiple data sources corresponding to the target object based on the graphical data extraction template, determines the original data corresponding to the target object, and improves the accuracy and efficiency of data extraction. Based on the data set configuration information, the original data is divided to determine the original data set corresponding to the target object, thereby organizing the data into a more orderly and manageable structure. Based on the annotation model library, the unannotated items in the original data set are annotated to obtain the target data set corresponding to the target object, thereby improving the accuracy and efficiency of data annotation. Based on the data statistical conditions corresponding to the target task and the target data set corresponding to the target object, data statistics are performed to generate the data statistical results corresponding to the target task, showing the data performance of the target object under different dimensions and indicators. The graphical interface makes the data extraction process more intuitive and easy to understand, improves the accuracy and efficiency of data extraction, divides the data into original data sets to help organize the data into a more orderly and manageable structure, introduces the annotation model library to improve the accuracy and efficiency of data annotation, and the task data statistics provide a comprehensive and in-depth understanding of the data performance of the target object, improves the compatibility and flexibility of data processing, and thus meets the complex and large-scale data processing requirements.
[0078] Optionally, the original data determination module 310 includes:
[0079] A processing method determining unit, used to analyze the target object and multiple data sources corresponding to the target object, and determine the data processing method corresponding to each data source;
[0080] A target template determination unit, configured to configure information on the graphical data extraction template based on the data processing method, and determine a target data extraction template corresponding to each data source;
[0081] The original data acquisition unit is used to extract data from the data source based on the target data extraction template to obtain the original data corresponding to the target object.
[0082] Optionally, the original data acquisition unit is specifically used to: determine a data acquisition strategy in the target data acquisition template that corresponds to a data management strategy of a data source; and filter data related to the target object in the data source based on the data acquisition strategy to obtain original data corresponding to the target object.
[0083] Optionally, the data partitioning module 320 is specifically used to: perform data cleaning and normalization preprocessing on the original data to obtain preprocessed target data; perform data partitioning on the target data based on data set configuration information to determine the original data set corresponding to the target object, wherein the data set configuration information includes: at least one of: data type, data format, data version and storage location.
[0084] Optionally, the data annotation module 330 includes:
[0085] A labeling model determining unit, configured to screen the labeling model library based on the original data set corresponding to the target object, and determine the labeling model corresponding to the original data set;
[0086] The target data set determining unit is used to detect and label the unlabeled items in the original data set based on the labeling model corresponding to the original data set, so as to obtain the target data set corresponding to the target object.
[0087] Optionally, the annotation model determination unit is specifically used to: determine the data format corresponding to the original data set based on the original data set corresponding to the target object; determine the annotation model corresponding to the original data set based on the mapping relationship between the annotation model and the data format in the annotation model library and the data format corresponding to the original data set.
[0088] Optionally, the data statistics module 340 is specifically used to: perform data statistical processing on the target data set corresponding to the target object based on the data statistical conditions corresponding to the target task, generate the data statistical results corresponding to the target task, and present the data statistical results in the form of a report; wherein the data statistical conditions include: at least one of: statistical dimension, statistical indicator and statistical method.
[0089] The data processing device provided in the embodiment of the present invention can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0090] Figure 4 A schematic diagram of the structure of an electronic device 12 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as desktop computers, workstations, servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0091] like Figure 4 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).
[0092] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.
[0093] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.
[0094] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, usually called a "hard drive"). Although Figure 4 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.
[0095] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.
[0096] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 12, and / or may communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0097] The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implementing a data processing method provided by the embodiment of the present invention, the method comprising:
[0098] Based on the graphical data extraction template, data is extracted from multiple data sources corresponding to the target object to determine the original data corresponding to the target object;
[0099] Based on the data set configuration information, the original data is divided into data to determine the original data set corresponding to the target object;
[0100] Based on the annotation model library, annotate the unannotated items in the original data set to obtain a target data set corresponding to the target object;
[0101] Data statistics are performed based on the data statistical conditions corresponding to the target task and the target data set corresponding to the target object to generate the data statistical results corresponding to the target task.
[0102] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the data processing method provided by any embodiment of the present invention.
[0103] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the data processing method provided in any embodiment of the present invention are implemented. The method includes:
[0104] Based on the graphical data extraction template, data is extracted from multiple data sources corresponding to the target object to determine the original data corresponding to the target object;
[0105] Based on the data set configuration information, the original data is divided into data to determine the original data set corresponding to the target object;
[0106] Based on the annotation model library, annotate the unannotated items in the original data set to obtain a target data set corresponding to the target object;
[0107] Data statistics are performed based on the data statistical conditions corresponding to the target task and the target data set corresponding to the target object to generate the data statistical results corresponding to the target task.
[0108] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.
[0109] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0110] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0111] Computer program code for performing the operations of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0112] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0113] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A data processing method, characterized in that: include: Based on the graphical data extraction template, data is extracted from multiple data sources corresponding to the target object to determine the original data corresponding to the target object; Based on the data set configuration information, the original data is divided into data to determine the original data set corresponding to the target object; Based on the annotation model library, annotate the unannotated items in the original data set to obtain a target data set corresponding to the target object; Data statistics are performed based on the data statistical conditions corresponding to the target task and the target data set corresponding to the target object to generate the data statistical results corresponding to the target task.
2. The method according to claim 1, characterized in that The step of extracting data from multiple data sources corresponding to the target object based on the graphical data extraction template to determine the original data corresponding to the target object includes: Analyze the target object and multiple data sources corresponding to the target object, and determine the data processing method corresponding to each data source; Based on the data processing method, information is configured on the graphical data extraction template to determine the target data extraction template corresponding to each data source; Data is extracted from the data source based on the target data extraction template to obtain original data corresponding to the target object.
3. The method according to claim 2, characterized in that The step of extracting data from a data source based on the target data extraction template to obtain original data corresponding to the target object includes: Determine a data acquisition strategy in the target data acquisition template that corresponds to the data management strategy of the data source; The data related to the target object in the data source is screened based on the data acquisition strategy to obtain the original data corresponding to the target object.
4. The method according to claim 1, characterized in that: The dividing the original data based on the data set configuration information to determine the original data set corresponding to the target object includes: Performing data cleaning and normalization preprocessing on the raw data to obtain preprocessed target data; The target data is divided based on the data set configuration information to determine the original data set corresponding to the target object, wherein the data set configuration information includes at least one of: data type, data format, data version and storage location.
5. The method according to claim 1, characterized in that The step of labeling the unlabeled items in the original data set based on the labeling model library to obtain the target data set corresponding to the target object includes: Screening the annotation model library based on the original data set corresponding to the target object to determine the annotation model corresponding to the original data set; Unlabeled items in the original data set are detected and labeled based on the labeling model corresponding to the original data set to obtain a target data set corresponding to the target object.
6. The method according to claim 5, characterized in that The step of screening the annotation model library based on the original data set corresponding to the target object to determine the annotation model corresponding to the original data set includes: Based on the original data set corresponding to the target object, determining a data format corresponding to the original data set; Based on the mapping relationship between the annotation models and the data formats in the annotation model library and the data format corresponding to the original data set, the annotation model corresponding to the original data set is determined.
7. The method according to claim 1, characterized in that The performing data statistics based on the data statistical conditions corresponding to the target task and the target data set corresponding to the target object to generate the data statistical results corresponding to the target task includes: Based on the data statistical conditions corresponding to the target task, perform data statistical processing on the target data set corresponding to the target object, generate the data statistical results corresponding to the target task, and display the data statistical results in the form of a report; The data statistical conditions include: at least one of a statistical dimension, a statistical indicator and a statistical method.
8. A data processing device, characterized in that: include: The original data determination module is used to extract data from multiple data sources corresponding to the target object based on the graphical data extraction template to determine the original data corresponding to the target object; A data partitioning module, used to partition the original data based on the data set configuration information, and determine the original data set corresponding to the target object; A data annotation module is used to annotate the unannotated items in the original data set based on the annotation model library to obtain a target data set corresponding to the target object; The data statistics module is used to perform data statistics based on the data statistics conditions corresponding to the target task and the target data set corresponding to the target object, and generate the data statistics results corresponding to the target task.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the data processing method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing method according to any one of claims 1 to 7 when executed.