Rapid Data Exploration

The method of uploading and constructing a smaller data set in parallel with integrity checks and metadata derivation enables concurrent data exploration and manipulation, addressing the inefficiencies of large data set uploads, enhancing productivity and resource utilization for data scientists.

JP7582756B2Active Publication Date: 2024-11-13INTERNATIONAL BUSINESS MACHINE CORPORATION
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2022575919
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-07-01
Filing Date
2021-06-14
Publication Date
2024-11-13
Estimated Expiration
2041-06-14

AI Technical Summary

Technical Problem

The transmission or uploading of large data sets from local systems to servers for machine learning model development is time-consuming, reducing the productivity of data scientists and limiting their ability to experiment and adapt with the data before the training process.

Method used

A method and system for rapid data exploration that involves uploading a first data set from a local system, determining its integrity, constructing a smaller representative second data set in parallel, and deriving statistical data and metadata, allowing visualization and tool configuration on the local system during upload, thereby enabling concurrent data exploration and manipulation.

Benefits of technology

This approach allows data scientists to continue working on training data while it is being uploaded, saving time, increasing productivity, and optimizing the use of computing resources by using a smaller, representative data subset for experimentation and model training.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007582756000001
    Figure 0007582756000001
  • Figure 0007582756000002
    Figure 0007582756000002
  • Figure 0007582756000003
    Figure 0007582756000003
Patent Text Reader

Abstract

A computer-implemented method for rapid data exploration of data to be uploaded may be provided. The method includes uploading a first dataset of a first size from a local system and determining that the first dataset is not corrupted. The method also includes, concurrently with the upload, selecting a predetermined number of records from the first dataset to construct a second dataset, determining statistics and metadata about the first dataset, and visualizing the second dataset, the statistics, and the metadata.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates generally to uploading large amounts of data from a client system to a server, and more particularly to a computer-implemented method for rapid data exploration of data to be uploaded. The present invention further relates to a data exploration system, and a computer program product for rapid data exploration of data to be uploaded. [Background technology]

[0002] Nowadays, AI (Artificial Intelligence) is becoming more and more prevalent in businesses of all sizes and in various types of business and consumer applications. Thus, a large community of developers is involved in developing or integrating AI technologies. As a result, training machine learning models typically requires a large amount of computing resources, and such services are often implemented as centralized or cloud computing-based services.

[0003] Traditional machine model learning methods are based on machine learning algorithms and use a set of known data samples along with associated annotations to train a general model capable of predicting annotations for unknown data samples. For example, a deep learning model for image classification is trained on a set of input images and associated image class labels. This machine learning (ML) model is then used to predict image class labels for new, unknown images during a testing phase. The input data samples may include a mixture of annotated and unannotated data samples that are processed together during the training phase.

[0004] From a general perspective, input data samples may be available as captured input data or may be generated by a data generator. A data generator may be represented as a real-world sensor (e.g., a still or video camera, an audio system), a human (e.g., a text author), or a simulator (an artificial system aimed at generating a certain type of data sample). Traditional data recognition methods may operate on a fixed number of annotated training samples, which may form a training set of data samples. These training data sets are usually relatively large.

[0005] A developer of a machine learning model usually needs to develop an understanding of the structure of the available training data in order to select an appropriate machine learning algorithm. This phase usually requires many cycles of testing and error in a trial-and-error format to select the appropriate AI algorithm with the associated hyper-parameters for the given problem and the available training data.

[0006] Because a large amount of computing power is required during a training session, the training, data inspection, and testing phases are performed on a central computing resource while developers continue to work on their local workstations. However, typically large training data sets must be uploaded to the central resource, and this uploading can take a significant amount of time, making developers' time unproductive.

[0007] There are several disclosures relating to computer-implemented methods for rapid data exploration of data to be uploaded.

[0008] Patent Document 1 discloses systems, methods, and techniques for managing data sets, specifically for representing, retrieving, and distributing large data sets and data files using metadata. Data representation, retrieval, and distribution of large data sets and data files using metadata are described.

[0009] US Patent No. 5,393,636 discloses a data representation module that represents a panel driven by metadata to allow a user to configure the presentation of information from a computer database system. The data representation module is customized to a user's environment by configuring metadata tables upon initial installation of the data representation module software. Changes to the panel presented by the data representation module can be implemented by changing the metadata tables without modifying the data representation module software.

[0010] A drawback of known solutions can be that the transmission or uploading of data from local systems to a server continues to take a significant amount of time, reducing the productivity of data scientists when developing ML models.

[0011] Thus, particularly in the context of developing machine learning algorithms, it may be necessary to overcome the drawback of not being able to continue working with the data during upload. [Prior art documents] [Patent documents]

[0012] [Patent Document 1] U.S. Patent No. 8,805,835(B2) [Patent Document 2] U.S. Patent No. 6,308,168(B1) Summary of the Invention

[0013] According to one aspect of the present invention, a computer-implemented method for rapid data exploration of data to be uploaded may be provided. The method may include uploading a first data set of a first size from a local system and determining that the first data set is not corrupted. The method may also include, in parallel with the upload, selecting a predetermined number of records from the first data set and constructing a second data set, determining statistics and metadata about the first data set, and visualizing the second data set, the statistics, and the metadata.

[0014] According to another aspect of the present invention, a data exploration system for rapid data exploration of data to be uploaded may be provided. The system may include means for uploading a first data set of a first size from a local system and means for determining that the first data set is not corrupted. Additionally, the system may include means for selecting a predetermined number of records from the first data set and constructing a second data set, means for determining statistics and metadata regarding the first data set, and means for visualizing the second data set, the statistics, and the metadata. Thereby, the means for selecting, the means for determining statistics and metadata, and the means for visualizing may operate in parallel in time with the means for uploading.

[0015] The proposed computer-implemented method for rapid data exploration of data to be uploaded may provide multiple advantages, technical effects, contributions, or improvements, or a combination thereof.

[0016] A data scientist working in an environment that includes a local system, i.e., a local workstation using a thin client, e.g., a web browser, as the user interface, and a server system with much more computing resources that are used to perform the training of ML models, may be able to save a lot of time, increase productivity, and make better use of the available computing resources. Since the development of an ML model requires a lot of forward and backward steps, a lot of trial and error, and a significant amount of experimentation with data, the transmission or upload time from the local system to the server may significantly reduce the productivity of the data scientist. This is also a result of experimentation with different sets of training data that need to be uploaded from the local system to the server.

[0017] The proposed concept may enable data scientists to continue their experimentation and data exploration while training data is being uploaded from the local system to the server, based on a representative subset of the first data set that is in the process of being uploaded from the local system to the server.

[0018] The statistical data and metadata derived from the first data set may help a user or data scientist to get an impression about the selected training data and may build a basis for constructing additional tools for data cleansing, data normalization, data correction, or data adoption, or a combination thereof. The tools so constructed may then be applied to the first data set. This may allow the user or data scientist to not be limited to running the tools on a local system or on a server where the first data set was uploaded.

[0019] The proposed concept does not restrict the freedom of data scientists to experiment and adapt with data before starting the ML training process, but rather gives them more time and flexibility in working with only a subset, specifically the second data set, in order to increase their productivity and to use the available resources, specifically computing resources; data scientists can be much more productive since they can use a second data set that can be representative of the first one and requires fewer resources for rapid data exploration.

[0020] Below, additional embodiments of the inventive concepts applicable to methods and systems are described.

[0021] According to an advantageous embodiment of the method, the statistical data comprises at least one from the group consisting of: minimum, in particular minimum, maximum, median, mean, quartiles of data values ​​of a column, frequency of data, amount of missing data values ​​(e.g. in a particular record or in a particular column). It is also possible to determine additional statistical data beyond the traditional meaning of metadata of a data set.

[0022] According to a further advantageous embodiment of the method, the metadata comprises at least one from the group consisting of number of rows, number of columns and type of column fields. Furthermore, the metadata may also comprise additional data regarding the data in the first data set, such as size and type of records, total amount of data, as well as explicit metadata, for example added at the beginning or end of the first data set. Such additional metadata may be extracted only from the first data set.

[0023] According to one allowable embodiment, the method may also include transmitting the statistical data and metadata in parallel with the upload of the first data set. The destination of the transmission may be the same as the destination used for the first data set. The statistical data may or may not be stored locally, i.e., the statistical data may be transmitted from the local client system to the server via a representational state transfer (REST) ​​request.

[0024] According to an optional embodiment of the method, the determination that the first data set is not corrupted may be performed before the upload begins. This has the advantage that data corruption, and therefore the impossibility of correct transmission, can be detected with almost certainty early in the process. If corrupted data is detected, the upload may be inhibited and a warning may be generated to the user. Thus, failures may be detected early in the transmission rather than at the end ("fast failure"), if, for example, a checksum can be constructed to ensure that the transmission is unlikely to complete correctly.

[0025] According to one useful embodiment of the method, the destination of the upload may be a cloud computing based service, hi one embodiment, the cloud computing based service may be a machine learning algorithm that is trained using the uploaded first data set.

[0026] According to an advantageous embodiment, the method may also include configuring, possibly running, a tool (e.g., a data modeling tool or another data manipulation tool) in parallel with the uploading, by using the statistical data, the metadata, or the second data set, or a combination thereof. The configuration of the tool may thus be performed locally on the client system or remotely on the server system. However, in the second case, it may also be useful to upload the second data set to the server, specifically in parallel with the uploading of the first data set. On the other hand, it may be advantageous to configure the tool locally on the client during the uploading and use the uploaded configuration on the server side. Thus, a data scientist using the proposed method is not limited by the system on which the tool is configured and executed.

[0027] Thus, according to a further advantageous embodiment, the method may also include executing the configured tool against the first data set, in particular on the server. This feature may be used advantageously, especially when configuring the tool on the client side. Furthermore, it may be necessary to transmit the configuration data from the client to the server-side tool.

[0028] According to a preferred embodiment of the method, the size of the second data set may be a small fraction of the size of the first data set. Typically, the second data set may contain around 1000 records (or less). This may represent a typical number for initial training of a machine learning tool or to get an idea of ​​the necessary data modeling to train a machine learning system on the server side with much more training data. Furthermore, the relationship between the second data set and the first status set in terms of the size of the data sets may range, for example, 1 / 100, 1 / 1000, 1 / 10,000, etc. One thing that the data scientist needs to be aware of is that the data may be asymmetric, i.e., a prediction is to be made about the behavior of a group of people, and the training data is to represent that group of people equally in all its aspects, e.g., gender, ethnicity, age, health status, education, etc. Thus, the data scientist needs to ensure that the training data has approximately the same amount of records for each relevant aspect to avoid the risk of one-sided bias in the ML model.

[0029] According to an acceptable embodiment of the method, selecting from the first data set may include selecting data from the first data set from the beginning, middle, end, randomly or according to a predefined algorithm. The selection process may depend, in particular, on the type of data available, the machine learning system being trained, the source of the data, and expected biases in the data, as well as many more parameters.

[0030] Furthermore, embodiments may take the form of an associated computer program product accessible from a computer usable or computer readable medium that provides program code for use by or in connection with a computer or any instruction execution system. For purposes of this description, a computer usable or computer readable medium may be any apparatus that may encompass means for storing, communicating, transmitting, or transferring a program for use by or in connection with an instruction execution system, apparatus, or device.

[0031] It should be noted that embodiments of the present invention are described with reference to different subject matters. In particular, some embodiments are described with reference to method type claims, and other embodiments are described with reference to apparatus type claims. However, a person skilled in the art will infer from the above and following description that, unless otherwise indicated, any combination of features belonging to one type of subject matter, as well as any combination between features relating to different subject matters, in particular between features of a method type claim and a feature of an apparatus type claim, is also considered to be disclosed within this document.

[0032] The above-defined and further aspects of the invention are apparent from and will be explained with reference to the example embodiments described hereinafter, to which the invention is not limited.

[0033] Preferred embodiments of the present invention will now be described, by way of example only, with reference to the following drawings: [Brief description of the drawings]

[0034] [Figure 1] FIG. 2 is a block diagram of an embodiment of a computer-implemented method of the present invention for rapid data discovery of data to be uploaded. [Diagram 2] FIG. 1 is a block diagram of a subset of some technology components for performing a method, according to one embodiment. [Diagram 3] 1 is a flow chart of one embodiment of the concepts proposed herein. [Figure 4] FIG. 1 is a block diagram of one embodiment of a data probing system for rapid data probing of data to be uploaded. [Diagram 5] FIG. 5 illustrates an embodiment of a computing system comprising the data exploration system according to FIG. 4. [Figure 6] FIG. 1 illustrates a cloud computing environment in which at least some of the concepts of the present invention may be deployed, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] In the context of this description, the following conventions, terms or expressions, or combinations thereof, may be used.

[0036] The term "rapid data exploration" may refer to the ability of a data scientist (or any other user) to perform an evaluation on a large (first) data set, typically used to train an ML-based system based on the analysis of a much smaller, representative subset of the large data set. The term may particularly relate to the option of evaluating a representative subset of the large first data set using statistical data and metadata about the first data set. Rapid data exploration may be performed on the data scientist's local system, as opposed to having to access data on a server-side system. However, it may also be possible to transmit the subset, statistical data, and metadata from the local system to the server system.

[0037] The term "uploading" as used herein may refer to the process of sending or transmitting data, and in various embodiments may refer to the process of transmitting large amounts of data, typically stored in a local data storage system, to a remote computer system, e.g., a server, particularly a cloud-based computing system that has large amounts of computing resources compared to the local system.

[0038] The term "local system" may refer to a data scientist's workstation, a personal computer, a thin client such as a web browser, or any other user interface system.

[0039] The term "first data set" may refer to a data set typically used for machine learning purposes, e.g., training data. Generally, the volume of the first data set may be so large that it may take a significant amount of time to transmit from the local system to the server-side system, which may slow down the data scientist's activities regarding scarce resources. In conventional systems, the data scientist may not be able to work with the first data set while the first data set is being transmitted, i.e., uploaded, from the local system to the server.

[0040] The term "corrupted" may mean that the data is not useful, e.g., has an incorrect checksum at the end of the data transmission, or may otherwise not be transmittable from one computing system to another. It is understood that a data file may be in a corrupted state before it is used or transmitted. Data files may be in a variety of formats, e.g., CSV (comma separated values). As an example, a CSV file may be in a corrupted state for several reasons, such as invalid formatting, use of invalid characters, line breaks within a field, among others.

[0041] The term "second data set" may refer to a much smaller data set when compared to the first data set. A typical size of the second data set may be in the range of about 1000 records or even less. However, there may be a certain relationship between the size of the first data set and the size of the second data set, such as 1 / 100, 1 / 1,000, 1 / 10,000, etc. This certain relationship may be based on a specified sample size. The second data set may be stored on a local system or the second data set may be uploaded to a server. An advantageous feature of the second data set is that if it is stored on a local system, it may be processed with very few resources on the local system. On the other hand, if the second data set is stored on a server system, it may be easy to access and work with the second data set remotely via a web browser by the local system.

[0042] The term "statistical data" may refer to information that is determined and / or calculated about a first data set. This may include, but is not limited to, the minimum value in a column, the maximum value in a column, the median value in a column, the average value in a column, the quartile value in a column, the frequency of data in a column, the amount of missing data values, and a range of other statistical data that may go beyond the classical meaning of metadata. Statistical data may include some values ​​in a data set that do not conform to the specified format of the data set.

[0043] The term "metadata" may refer to information about data, specifically about a first data set. Typically, the metadata is descriptive and relates to structural or administrative components of the reference data of the first data set. Metadata may include, for example, the number of rows and columns in the data set, as well as the format of the data set. It is noted that a distinction should be made between this conventional form of metadata and statistical data about the first data set.

[0044] The term "cloud computing", and similarly the term "cloud computing-based services", in the context of this document, may be interpreted as a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. This cloud model promotes availability and consists of at least five fundamental characteristics, three service models, and four deployment models:

[0045] The basic characteristics of cloud computing include: (i) On-Demand Self-Service. Consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as needed, without the need for human interaction with the respective service provider. (ii) Pervasive Network Access. Functionality is available over a network and accessed through standard mechanisms that facilitate use by heterogeneous thin- or thick-client platforms (e.g., cell phones, laptops, and PDAs). (iii) Resource Pooling. A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated in response to consumer demand. There is location independence in that the customer generally has no control or knowledge of the exact location of the provided resources, although it may be possible to specify the location at a higher level of abstraction (e.g., country, state, or data center). Examples of resources include storage, processing, memory, network bandwidth, and virtual machines. (iv) Rapid Elasticity. Capabilities can be rapidly and elastically provisioned, sometimes automatically, to rapidly scale out, and rapidly released to rapidly scale in. To the consumer, the capabilities available for provisioning often appear unlimited and can be purchased at any time and in any quantity. (v) Metering of Services. Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services utilized.

[0046] Cloud computing service models used include: (i) Software as a Service (SaaS). The functionality provided to the consumer is the use of the provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). With the possible exception of limited user-specific application configuration settings, the consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or individual application functionality. (ii) Platform as a Service (PaaS): The functionality offered to the Consumer is the deployment of Consumer-created or acquired applications, written using programming languages ​​and tools supported by the Provider, onto a cloud infrastructure. The Consumer does not manage or control the underlying cloud infrastructure, such as networks, servers, operating systems, or storage, but does control the deployed applications and, in some cases, the application hosting environment configuration. (iii) Cloud Infrastructure as a Service (IaaS): The functionality provided to the consumer is the provisioning of processing, storage, network, and other basic computing resources on which the consumer may deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does control the operating systems, storage, deployed applications, and in some cases has limited control over selected network components (e.g., host firewalls).

[0047] Deployment models for cloud computing include the following: (i) Private Cloud. The cloud infrastructure is operated solely by an organization. This cloud infrastructure may be managed by the organization or a third party and may reside on-premises or off-premises. (ii) Community Cloud. Cloud infrastructure is shared by multiple organizations to support a particular community with shared concerns (e.g., mission, security requirements, policies, and compliance considerations). This cloud infrastructure may be managed by the organizations or a third party and may reside on or off premises. (iii) Public Cloud. Cloud infrastructure is available to the general public or large industry groups and is owned by organizations that sell cloud services. (iv) Hybrid Cloud: A cloud infrastructure is a composition of two or more clouds (private, community, or public) that remain unique entities but are tied together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0048] Note that cloud software takes full advantage of the cloud paradigm by being service-oriented with an emphasis on statelessness (with exceptions), low coupling, modularity, and semantic interoperability.

[0049] In the following, a detailed description of the drawings and preferred embodiments are given. All indications in the figures are schematic. First, a block diagram of an embodiment of a computer-implemented method of the present invention for rapid data exploration of data to be uploaded is given. Then, further embodiments and an embodiment of a data exploration system for rapid data exploration of data to be uploaded are described.

[0050] FIG. 1 shows a flow diagram of a preferred embodiment of a computer-implemented method 100 for rapid data exploration of data to be uploaded. The method 100 includes uploading 102 a first data set of a first size, specifically large, from a local system. The uploading can be performed from a local system, such as a PC (personal computer), a local workstation, or a local server, to a receiving server, which can be a more powerful server with much more computing resources. The uploading can be performed using a thin client, for example, a web browser running on a PC with relatively small processing and memory resources. Typically, machine learning algorithms, such as training of an ML model or ML system, can be executed on the receiving server, for example server 214. Also, the receiving server can be integrated into a larger cloud computing center. The local system can be a client system or can be only a web browser.

[0051] The method further includes determining (104) that the first data set is not corrupted. If so, the upload is not initiated (or completed if the corruption is detected after the upload has begun) to take advantage of the early failure effects and save time and resources for other activities. Although FIG. 1 illustrates uploading (102) the first data set followed by determining (104) that the first data set is not corrupted, it should be understood that this sequence is not required. In various embodiments, operation 104 may be performed prior to or concurrently with operation 102.

[0052] Also part of the method 100 is to perform, in parallel with the uploading (102), from the first data set, a predetermined number of records for constructing a second data set, thus a much smaller subset in size (106), determining statistical data and metadata about the first data set (108), and visualizing the second data set, the statistical data, and the metadata (110). The last step may be performed based on server activity, i.e., a request, or local system activity, i.e., a send command. Although the operations 106, 108, and 110 are preferably performed in parallel with the uploading operation 102, it is not necessary that the operations 106, 108, and 110 are performed completely in parallel with the operation 102. For example, the operations 106, 108, and 110 may start before the uploading operation 102 or end after the uploading operation 102.

[0053] FIG. 2 illustrates a block diagram 200 of a subset of related art components for performing the method 100, according to one embodiment. In the context of ML, a local system 202 may manage a large data set 208 on a storage system 206, which may be one or more storage devices, such as a hard drive or flash drive, or may include traditional computer components, such as a processor, a bus, and peripheral devices, along with one or more storage devices. The ML algorithm to be trained may reside on a server 214. Remote access to the storage system 206 from the server 214 may be constrained by low performance characteristics, e.g., a significant amount of time to complete ML training, and may require a relatively large amount of network bandwidth, thereby making it impractical or impossible. Thus, it may be necessary to transfer, upload, or transmit the first data set 208 from the local system 202 to the server 214, as indicated by the dashed line for the large data set 220 in the server-mounted storage system 218. The large data set 220 is a copy of the first data set 208 on the storage system 206. That is, a typical size of such a first data set 208 may be in the range of 50 GB, 100 GB, or more (e.g., image data). It will be understood that image data may often be used as a basis for training ML algorithms, and thus the amount of data mentioned is not unusual. However, there is a drawback in that uploading from the local system 202 to the server system 214 may take a lot of time, so the data scientist has to wait until all the data is transferred to the server 214 side, which may take a considerable amount of time.

[0054] As mentioned, the server 214 may be a more powerful computing system having more computing resources than the local system 202. The power of the server 214 may be the result of the server having more processing cores, more memory, faster clock speeds, etc., as compared to the local system 202, but no particular hardware configuration is required. In various embodiments, characteristics that distinguish the server 214 from the local system 202 are throughput, response time, processing speed, etc. In particular, the server 214 may be distinguished from the local system 202 based on its ability to train ML models using the first data set 208 / 220 in a small fraction of the time that would be required for the local system 202. For example, the server 214 may be capable of training ML models using the first data set 208 / 220 in five percent or ten percent of the time that would be required for the local system 202. As a specific example, the server 214 may be able to train an ML model using the first data set 208 / 220 in 50 minutes, while the same task takes 500 minutes (8.3 hours) on the local system 202. In other cases, the time savings may not be as great, but even if the server 214 requires 50 percent or 75 percent of the time that would be required for the local system 202, the time savings may still be significant and advantageous.

[0055] In various embodiments, the local system 202 has a first configuration of processing cores and memory capable of training a machine learning model using a first data set for a first time period, and the server 214 has a second configuration of processing cores and memory capable of training the same machine learning model using the same first data set for a second time period, where the second time period is shorter than the first time period because the server 214 is a more powerful system than the local system 202.

[0056] According to various embodiments, a data scientist may use the time required for uploading to explore the data locally in parallel with the upload, and optionally configure data modeling or data manipulation tools.

[0057] The local system 202 is triggered by the sender / uploader 204 to send a first data set 208 to the server 214, where the first data set is received by the receiver 216. The sender / uploader 204 may include data transmission hardware, such as a modem, and data transmission software implementing a data communication protocol, such as TCP, and an error detection protocol. Prior to (or concurrent with) the transmission, a check may be performed at the local system 202 to determine that the first data set is not corrupted. For example, a tool such as CSVLint may be used to check whether the first data set contains the columns and value types that it should contain, i.e., an automated tool may be used to check whether the file is parsable and readable. In the event of corrupted data, the data is not sent by the sender / uploader 204. Additionally, a check may be performed at the receiver 216 to determine that the first data set is still not corrupted after transmission.

[0058] In order to use the time of transmission from the local system 202 to the server 214, in parallel with the upload, i.e., data transmission, a subset, i.e., second data set 210, may be generated from the first data set 208 (e.g., the subset may be a small portion of the first data set), and from the first data set, statistical data and metadata 212 about the first data set may also be derived or determined, or both. To give the data scientist an opportunity to get a first impression of the data in the first data set, the second data set 210, the statistical data, and the metadata 212 may be visualized, e.g., displayed on a display device of the local system 202, to the data scientist. Based on the knowledge about the second data set 210 obtained from the visualization, the data scientist may decide to configure a data modeling tool to be applied to the first data set 208. The configuration of the data modeling tool and / or the selection of the ML algorithm may be performed on the local system 202 or on the server 214. Typically, during this phase, preparatory tasks for data cleansing activities, data normalization activities are tested on a second data set that may be representative of the first data set. In various embodiments, one or more data cleansing or data normalization software programs may be applied to the second data set on the local system 204 to determine which programs may be most effective and how to most effectively configure any parameters of the data cleansing or data normalization software.

[0059] Since ML model development is relatively trial-and-error approach, the proposed method may save data scientists a lot of time. Access to the first data set 220 on the server side may be by ML algorithms running on the server 214 or on another independent ML system 222, which like the server 214 may be a very powerful server with much more computing resources than the local system 202.

[0060] It should also be noted that the ML algorithms may be incorporated into a cloud computing environment, such as the one shown in FIG. 6, to which the server 214 and the ML system 222 may belong.

[0061] FIG. 3 shows a flow chart 300 of one embodiment of the concept proposed herein. The flow chart 300 starts with making available (302) a first data set on a local system, for example the local system 202. The process flow then diverges into branches. First, an upload of the first data set is initiated (304). As a first step in this activity, it is determined (306) whether the first data set is corrupted. Operation 306 may be completed prior to the upload of the first data set or may be performed simultaneously with the upload. A software tool such as CVSLint may be used to determine whether the first data set is parsable and readable. If there is a risk that the first data set is corrupted and, as a result, the transmission from the local system to the server may not be successful, the upload is stopped (310). In the second case, where the first data set is not corrupted, the upload continues (308) until completion, which may take a relatively long time due to the large amount of data to be uploaded.

[0062] Second, a second data set is extracted (312) from the first data set. A selection process may be performed such that the second data set (denoted as DS in the figure) is an adequate representation of the first data set. For example, to obtain an adequate representation, the selection process may include random sampling, systematic sampling, stratified sampling, clustered sampling, or a combination of two or more of these techniques. In the case of asymmetric data, the sampling technique may sample the first data set in such a way that each group of data having a particular aspect, e.g., gender, is equally or proportionately represented in the second data set. In various embodiments, the extraction of the second data set using the selection process (312) may be performed by the local system 202. Next, metadata (denoted as MD in the figure) and statistical data (denoted as SD in the figure) of the first data set are determined (314). In various embodiments, the determination of the MD and SD (314) may be performed by the local system 202. The statistical data, metadata, and second data set are then visualized (316), for example, to a data scientist, who evaluates the visualized data on a display device. Optionally, the statistical data and metadata of the second data set and the first data set may be uploaded (318) to a server (as shown in the dashed box) and visualized (316) in a web browser of the local system 202. The data scientist may then configure a tool (320) based on the evaluation of the visualized data. The tool may normalize, cleanse, correct, etc., the data of the second data set and the first data set; in other words, the tool may be a data modeling tool. In some embodiments, the data modeling tool may run on the local system 202, but typically the data modeling tool runs only on the server 214 and is accessed by the data scientist via a web browser of the local system 202.If a data modeling tool running on a server 214 is used, the second data set 210, MD, and SD, must be uploaded to the server 214.

[0063] Optionally, the configuration of the data modeling tool may also be uploaded to the server (322). The data modeling tool may then be applied to the first data set (324). Thus, after the upload of the first data set is complete, the data scientist may quickly perform the steps used to prepare the second, smaller data set on the original, larger, first data set. The data modeling tool may be configured to perform these steps on the first data set in response to a single command such as "Replace data set and rerun notebook?". After the transmission to the server is successfully completed, training of the ML model of the ML system may begin using the first data set (326). In various embodiments, training of the ML model may begin after the first data set is transmitted and after data cleansing or data normalization of the first data is performed using the configuration parameters determined at the local system 202 using the second data set. After the training of the ML model is completed, the process ends (332).

[0064] 4 shows a block diagram of one embodiment of a data probing system 400 for rapid data probing of data to be uploaded. The system includes at least a means for uploading a first data set of a first size from a local system, specifically a sender 402 (e.g., sender / uploader 204), and a means for determining that the first data set is not corrupted, specifically a first determining unit 404, e.g., a software tool that performs a test to determine whether a data file or other data structure is corrupted. Examples include CSVLint, and a tool that verifies checksums. If corrupted, the sender 402 is not activated.

[0065] Further, the data exploration system 400 includes means, specifically a selection unit 406, for selecting a predetermined number of records from the first data set and constructing a second data set. For example, the selection unit 406 may include software for random sampling of data, systematic sampling, stratified sampling, clustered sampling, or a combination of two or more of these techniques. The data exploration system 400 also includes means, specifically a second determination unit 408, for determining statistical data and metadata regarding the first data set. For example, the second determination unit 408 may include software necessary to calculate at least one statistical data from the group including minimum, maximum, median, mean, quartiles, frequency of data, amount of missing data values, and software necessary to extract metadata, the metadata including at least one from the group including number of rows, number of columns, type of column fields. As an example of software for determining statistical data or metadata regarding the first data, the custom or proprietary determination unit 408 may be written in JavaScript for a particular first data set. Additionally, custom or proprietary code may be used to determine a file's size, name, last modification time, and type (metadata) using a web browser.

[0066] The data exploration system 400 also includes means for visualizing the second data set, the statistical data, and the metadata, specifically a visualization unit 410. The visualization unit 410 can be any software for visualizing data, for example as a table, a chart, plain text. Examples of software for visualizing data can be proprietary tools, open source tools, or commercially available tools. The software for visualizing data can be written in JavaScript and realize the visualization of data within a web browser. In one embodiment, the visualization unit 410 can be a web-based application accessible via a thin client such as a web browser that presents the data to be visualized in a tabular format (rows, columns, column names) and a panel format (data summary, variable / column summary). The panel format can include information about the individual variables, such as histograms of variables, mean, median, mode, skewness, kurtosis, minimum, maximum, standard deviation of variables, whether the variables have missing values ​​(if so, how many), and box plots. For example, the second data set can be visualized in a tabular format. Further, the first data set may be visualized in a panel format, whereby the means for selecting, the means for determining statistics and metadata, and the means for visualizing operate in parallel in time with the means for uploading.

[0067] It should be noted that the hardware implementable units or modules of the data exploration system 400, namely the transmitter 402, the first determination unit 404, the specific selection unit, the second determination unit 408 and the visualization unit 410, may be electrically connected for data exchange, or alternatively, they may also be connected via an internal bus system of the data exploration system for data and signal exchange.

[0068] Before moving to FIG. 5, FIG. 6 illustrates a cloud computing environment in which at least a portion of the inventive concepts, specifically the server-side components, may be deployed.

[0069] FIG. 6 illustrates a cloud computing environment 600 in which at least some of the concepts of the present invention may be deployed. A series of functional abstraction layers provided by the cloud computing environment are illustrated. It should be understood in advance that the components, layers, and functions illustrated in FIG. 6 are for illustrative purposes only, and the present invention is not limited thereto. As illustrated, the following layers and corresponding functions are provided: Hardware and software layer 602 includes hardware components and software components. Examples of hardware components include mainframe 604, server 606, RISC (reduced instruction set computer) architecture based server 608, blade server 610, storage device 612, network 614, and networking components 614. In some embodiments, the software components include network application server software 616 and / or database software 618.

[0070] The virtualization layer 620 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 622, virtual storage 624, virtual networks including virtual private networks 626, virtual applications and operating systems 628, and virtual clients 630. In one example, the management layer 632 may provide the functionality described below. Resource provisioning 634 provides dynamic procurement of computing and other resources utilized to perform tasks within the cloud computing environment. Metering and pricing 636 provides cost tracking as resources are utilized within the cloud computing environment, and billing or charging for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. A user portal 638 provides consumers and system administrators with access to the cloud computing environment. Service level management 640 provides allocation and management of cloud computing resources such that requested service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 642 provides for the pre-arrangement and procurement of cloud computing resources anticipated to be required in the future in accordance with the SLA.

[0071] The workload tier 644 provides examples of functions for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this tier include mapping and navigation 646, software development and lifecycle management 648, virtual classroom instructional delivery 650, data analytics processing 652, transaction processing 654, as well as server-side components of the solutions proposed herein 656.

[0072] Embodiments of the present invention may be implemented with virtually any type of computer, regardless of the platform suitable for storing and / or executing program code. Figure 5 illustrates, by way of example, a computing system 400 suitable for executing program code associated with the proposed method.

[0073] Computing system 500 is only one example of a suitable computer system, and whether or not computer system 500 is capable of implementing and / or performing any of the functionality described herein is not intended to suggest any limitation as to the scope of use or functionality of the embodiments of the invention described herein. There are components of computer system 500 that are operable in numerous other general purpose or special purpose computing system environments or computing system configurations. Examples of well-known computing systems, computing environments, or computing configurations, or combinations thereof, that may be suitable for use with computer system / server 500 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices. The computer system / server 500 may be described in the general context of computer system executable instructions, such as program modules, executed by the computer system 500. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system / server 500 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0074] As shown in the figure, computer system / server 500 is shown in the form of a general-purpose computing device. Components of computer system / server 500 may include, but are not limited to, one or more processors or processing units 502, a system memory 504, and a bus 506 coupling various system components including the system memory 504 to the processor 502. Bus 406 represents any one or more of several types of bus structures including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standard Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus. Computer system / server 500 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer system / server 500 and includes both volatile and non-volatile media, removable and non-removable media.

[0075] The system memory 504 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 508 or cache memory 510, or both. The computer system / server 500 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 512 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from and writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical medium. In such a case, each may be connected to the bus 506 by one or more data media interfaces. As further illustrated and described below, the memory 504 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of an embodiment of the present invention.

[0076] A program / utility having a set (at least one) of program modules 516 may be stored in memory 504, as well as, by way of example and not limitation, an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may include an implementation of a networking environment. The program modules 516 generally perform the functions and / or methodologies of embodiments of the present invention as described herein.

[0077] The computer system / server 500 may also communicate with one or more external devices 518, such as a keyboard, a pointing device, a display 520, one or more devices that allow a user to interact with the computer system / server 500, or any device that allows the computer system / server 500 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.), or a combination thereof. Such communication may occur through an input / output (I / O) interface 514. Additionally, the computer system / server 500 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, through a network adapter 522. As shown, the network adapter 522 may communicate with other components of the computer system / server 500 through a bus 506. Although not shown, it is understood that other hardware and / or software components may be used with the computer system / server 500. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0078] Additionally, a data probing system 400 for rapid data probing of data to be uploaded may be attached to the bus system 506 .

[0079] The description of various embodiments of the present invention is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0080] The present invention may be embodied as a system, method, or computer program product, or a combination thereof. The computer program product may include a computer readable storage medium (or multiple computer readable storage media) having computer readable program instructions for causing a processor to perform aspects of the present invention.

[0081] The medium may be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system for a propagation medium. Examples of a computer-readable medium may include a semiconductor or solid-state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk, and an optical disk. Current examples of optical disks include read-only memory compact disks (CD-ROMs), read / write compact disks (CD-R / Ws), DVDs, and Blu-ray disks.

[0082] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable read-only memory compact disks (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, punch cards or mechanically encoded devices such as raised structures in grooves with instructions recorded thereon, and any suitable combination of the above. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0083] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to the respective computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0084] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk, C++, and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer as a stand-alone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions to perform aspects of the invention by utilizing state information of the computer readable program instructions to personalize the electronic circuitry.

[0085] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0086] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, produce means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable medium, such that the computer readable storage medium on which the instructions are stored includes an article of manufacture including instructions implementing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams, and can direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner.

[0087] The computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or another device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to create a computer-implemented process such that the instructions, which execute on the computer, other programmable apparatus, or another device, perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0088] The flowcharts and / or block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously or the blocks may be executed in reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or actions or executes a combination of dedicated hardware and computer instructions.

[0089] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly dictates otherwise. It will be further understood that the terms "comprises" and / or "comprising" as used herein indicate the presence of stated features, integers, steps, operations, elements, or components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof, or combinations thereof.

[0090] The corresponding structures, materials, acts, and equivalents of all means or steps and functional elements within the scope of the appended claims are intended to include any structure, material, or act for performing a function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the invention to the disclosed form. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The embodiments have been selected and described in order to best explain the principles and practical applications of the invention, and to enable those skilled in the art to understand the invention in various embodiments with various modifications suitable for the particular use contemplated.

[0091] Briefly, the concepts of the present invention can be summarized in the following items. 1. 2. A computer-implemented method for rapid data discovery of data to be uploaded, comprising: uploading a first data set of a first size from a local system; determining that the first data set is not corrupted; and - In parallel with the upload, - selecting a predetermined number of records from the first data set to construct a second data set; determining statistical data and metadata about the first data set; and - Visualize second data sets, statistics, and metadata Also, 4. A computer-implemented method comprising: 3. The method according to item 1, wherein the statistical data comprises at least one from the group consisting of minimum, maximum, median, mean, quartile, frequency of data, and amount of missing data values. 4. The method of claim 1 or 2, wherein the metadata includes at least one from the group including number of rows, number of columns, and type of column fields. 5. - Send statistical data and metadata in parallel with the upload of the first data set The method according to any of the preceding items. 6. A method according to any of the preceding items, wherein a determination that a data set is not corrupted is first made before uploading begins. 7. The method according to any of the preceding items, wherein the upload destination is a cloud computing based service. 8. - Configure the tool with statistical data, metadata and / or a second data set in parallel with the upload The method according to any of the preceding items. 9. -running a configured tool on a first data set The method according to item 7, further comprising: 10. The method of any of the preceding items, wherein the size of the second data set is a small fraction of the size of the first data set. 11. Selection from the first data set - selecting from the first data set comprises selecting data from the beginning, middle or end of the first data set randomly or according to a predetermined algorithm; The method according to any of the preceding items, comprising: 12. A data mining system for rapid data mining of data to be uploaded, comprising: - means for uploading a first data set of a first size from a local system; - a means for determining that the first data set is not corrupted; and - a means for selecting a predetermined number of records from the first data set to construct a second data set; - a means for determining statistical data and metadata about the first data set; -Second, a means to visualize data sets, statistics, and metadata wherein the means for selecting, the means for determining statistics and metadata, and the means for visualizing operate in parallel in time with the means for uploading. 13. The data exploration system of item 11, wherein the statistical data includes at least one from the group including minimum, maximum, median, mean, quartile, frequency of data, and amount of missing data values. 14. The data exploration system of item 11 or 12, wherein the metadata includes at least one from the group including number of rows, number of columns, and types of column fields. 15. - Means for uploading statistical data and metadata in parallel with the upload of the first data set 14. The data exploration system according to any one of items 11 to 13, further comprising: 16. The data exploration system of any of items 11 to 14, wherein the means for determining that the first data set is not corrupted is executed before the means for uploading the first data set is initiated. 17. A data exploration system according to any one of items 11 to 15, wherein the upload destination is a cloud computing-based service. 18. - Means for configuring the tool with statistical data, metadata, or a second data set, or a combination thereof, in parallel with the upload 17. The data exploration system according to any one of items 11 to 16, further comprising: 19. The data exploration system of item 17, wherein the configured tool is executed on the first data set. 20. The data exploration system of any of items 11 to 18, wherein the size of the second data set is a small fraction of the size of the first data set. 21. A computer program product for rapid data exploration of data to be uploaded, comprising a computer-readable storage medium having program instructions embodied thereon, the program instructions being executable by one or more computing systems or controllers, the one or more computing systems being configured to: uploading a first data set of a first size from a local system; - determining that the first data set is not corrupted; - In parallel with the upload, - selecting a predetermined number of records from the first data set to construct a second data set; determining metadata about the first data set; and - Visualize second data sets, statistics, and metadata visualizations A computer program product that causes the

Claims

1. 1. A method for performing rapid data discovery of data to be uploaded by computer information processing, comprising: uploading a first data set of a first size from a local system to a server; determining, by the local system, that the first data set is not corrupted; In parallel with the above upload, selecting a number of records from the first data set and constructing a second data set using the selected records; determining statistics and metadata about the first data set; and Visualizing the second data set, the statistical data, and the metadata. To carry out 4. A computer-implemented method comprising:

2. The method of claim 1 , wherein the statistical data comprises at least one from the group comprising: minimum, maximum, median, mean, quartile, frequency of data values, and amount of missing data values.

3. The method of claim 1 or 2, wherein the metadata includes at least one from the group including: number of rows, number of columns, and types of column fields.

4. 4. The method according to claim 1, further comprising transmitting the statistical data and metadata to the server, wherein the transmitting of the statistical data and metadata to the server is performed in parallel with the uploading of the first data set to the server.

5. The method of any one of claims 1 to 4, wherein the determination that the first data set is not corrupted is performed before the upload begins.

6. 6. The method of claim 1, wherein the local system has a first configuration of processing cores and memory capable of training a machine learning model using the first data set for a first time period, and the server has a second configuration of processing cores and memory capable of training the machine learning model using the first data set for a second time period, the second time period being shorter than the first time period.

7. 7. The method of claim 1, further comprising configuring a data modeling tool on the local system or on the server using the statistical data, the metadata, or the second data set, wherein a data modeling tool is configured in parallel with the uploading of the first data set.

8. The method of claim 7, further comprising running the configured data modeling tool on the first data set.

9. The method of any one of claims 1 to 8, wherein the second data set is of a second size, and the first size of the first data set is greater than the second size.

10. 10. The method of any one of claims 1 to 9, wherein the selecting from the first data set comprises selecting data from the first data set from the beginning, middle, end, randomly or according to a predetermined algorithm.

11. A computer program for causing a computer to execute the method according to any one of claims 1 to 10.

12. A computer-readable storage medium having the computer program according to claim 11 recorded thereon.

Citation Information

Patent Citations

  • Interactive system and method for analyzing and displaying mass data set

    CN110413765A

  • Systems and methods for rapidly building, managing, and sharing machine learning models

    US20200202171A1

  • Metadata-driven data presentation module for database system

    US6308168B1

  • Systems and methods for integrated management of large data sets

    US8805835B2

  • Application development platform and software development kits that provide comprehensive machine learning services

    WO2018217635A1