User interface for machine language model creation

Abstract

The present disclosure presents devices, methods, and computer-readable media for creating a user interface for a machine learning model. Application developers can select a machine learning template (304) from a plurality of templates suitable for the data type used in their application. The template may include a plurality of templates for classification of image, text, sound, motion, and tabular data. The graphical user interface (300) allows intuitive selection of training data (316), validation data (318), and integration of the trained model (314) into the application. The user interface also displays numerical scores for both training accuracy (508) and validation accuracy (510) using test data. The application provides a live mode that allows execution of the machine learning model on a mobile device to allow the model to be tested based on data from one or more of the sensors (i.e., camera or microphone) on the mobile device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to U.S. Provisional Application No. 62 / 855,958, filed on June 1, 2019, and U.S. Patent Application No. 16 / 670,914, filed on October 31, 2019; the entire text of each of the above patent applications is incorporated herein by reference and for all purposes. Technical Field

[0003] The present disclosure generally relates to techniques and user interfaces for generating and training machine learning models for application development. Background Art

[0004] Machine learning techniques can be implemented by application developers to make predictions and decisions without explicit instructions, relying instead on patterns and influences. Machine learning techniques create mathematical models based on sample data (called "training data"). Machine learning techniques can be used in a variety of applications, such as email filtering, computer vision, and predictive analysis. Models developed for machine learning are often developed using separate applications from those used to create the application. These separate machine learning tools may also involve programming languages ​​different from the application. Providing technology that implements machine learning tools into the same integrated development environment in which applications are developed can allow models to be more easily trained, tested, and seamlessly integrated into applications that employ these machine learning techniques. Summary of the invention

[0005] Certain embodiments relate to techniques (e.g., methods, memory or non-transitory computer-readable media storing code or instructions executable by one or more processors) for creating user interfaces for machine learning models in an integrated development environment. The disclosed machine learning tool may allow the entire process of training and testing models to be integrated into an application using an interactive graphical user interface. Machine learning (ML) templates may allow developers to easily create custom models without having to program separate machine learning models. For example, training data may be obtained from a storage device or may be integrated from live capture / recording of images or sounds. Both stored data and live data may be used to train and test models, all within the application.

[0006] In various embodiments, the technology can be implemented in an application development program. The technology can allow developers to select from a number of available templates. For example, templates can be categorized as images, text, sound, motion, and tabular data. The machine learning model can allow for the classification of data and / or the regression of a real-valued target within a set of data.

[0007] These and other embodiments of the present invention are described in detail below.For example, other embodiments relate to systems, devices, and computer-readable media associated with the methods described herein.

[0008] Further areas of applicability of the present disclosure will become apparent from the detailed description provided hereinafter.It should be understood that the detailed description and specific examples, while indicating various embodiments, are intended for purposes of illustration only and are not intended to necessarily limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 An exemplary simplified diagram of a system for generating trained machine learning models for application development is shown.

[0010] Figure 2 An exemplary graphical user interface for selecting a template for generating a trained machine learning model is shown.

[0011] Figure 3 An exemplary graphical user interface for selecting training data for a sound classifier is shown.

[0012] Figure 4 An exemplary graphical user interface for pulling in training data for a sound classifier is shown.

[0013] Figure 5 An exemplary graphical user interface for training a machine learning model is shown.

[0014] Figure 6 An exemplary graphical user interface for the output of a machine learning model, specifically, a sound classifier, is shown.

[0015] Figure 7 An exemplary graphical user interface showing a progress bar for a technique for generating a trained machine learning model.

[0016] Figure 8 An exemplary graphical user interface for the output of a machine learning model, specifically, an image classifier, is shown.

[0017] Fig. 9 is an exemplary flow chart of a technique for generating a trained machine learning model for application development.

[0018] Fig.10 A block diagram is shown for an exemplary device to perform one or more techniques described herein.

[0019] In the drawings, similar components and / or features may have the same reference label. In addition, various components of the same type may be distinguished by a reference label followed by a hyphen and a second label that distinguishes the similar components. If only the first reference label is used in the specification, the description applies to any one of the similar components having the same first reference label regardless of the second reference label. DETAILED DESCRIPTION

[0020] Certain embodiments of the present disclosure relate to devices, computer-readable media, and methods for creating machine learning models in an integrated development environment. The machine learning tool provides developers with options about data types to choose from. Developers can select templates that are suitable for the data types used in their applications.

[0021] The machine learning tool can allow developers to easily create trained machine learning models for incorporation into applications. The machine learning tool can be application-based and can run as a standalone application or in an integrated development environment (IDE) of an operating system. In some embodiments, the trained machine learning model created by the tool (e.g., provided as a standalone application) can then be incorporated into a program being developed in the IDE. This enables developers to take advantage of the inherent development tools provided by the operating system rather than trying to incorporate models developed by third-party programs. This also allows for easier testing of applications in an integrated development environment.

[0022] Applications in conjunction with machine learning models can be executed on portable computing devices such as laptops or tablets. Machine learning tools can be user-friendly by allowing the selection of model types and / or model sources from a graphical user interface. For example, various model types may include image analysis models, sound analysis models, text analysis models, tabular data analysis models, etc. The model source may include a computer or network location of training data. The graphical user interface may allow developers to drag and drop training data into a specified area. Similarly, developers can select evaluation or test data through an intuitive user interface. Machine learning tools can provide graphical workflows to assist developers in the various stages of developing and training machine learning models. In addition, the graphical user interface may provide developers with graphical depictions of the various stages and sub-stages for generating trained models. In some examples, the graphical user interface may display numerical scores for both training accuracy and validation accuracy using test data.

[0023] I. Generate a trained machine learning model

[0024] The disclosure herein describes various user-friendly techniques for generating trained machine learning models. In supervised learning, the technique builds a mathematical model from a set of data containing both input and expected output. For example, if the task is to determine whether an image contains a certain object, the training data for the supervised learning technique may include images with and without the object (input), and each image may have a label (output) specifying whether it contains an object. The technique can be used for various different types of data. The technique may include selecting a template for a model type (e.g., image analysis, text analysis, sound analysis, motion analysis, and table data analysis). Next, the technique may include inputting training data for training the model. A part of the training process may include identifying and selecting test data to evaluate the model. After the model has been trained, the test data may be used to evaluate the model. The technique may generate one or more scores as a result of the evaluation. The trained model may be stored and exported for use in other applications.

[0025] Figure 1 A simplified diagram of a technique 100 for machine learning model generation is shown. The machine learning tool 102 may be a software application executed on a computing device. The machine learning tool 102 may include a plurality of graphical user interfaces to allow a developer to interact with the routine.

[0026] A. Select a template

[0027] Developers can first consider whether their applications should incorporate machine learning. Machine learning is suited to performing specific tasks without explicit instructions. Machine learning techniques accomplish these tasks by learning and using patterns found in data sets. Some typical uses of machine learning techniques include assisting with manual data entry, detecting unwanted email messages (otherwise known as spam), recommending products, diagnostic techniques, customer segmentation and lifetime value prediction, predictive maintenance, financial analysis, sound analysis, image recognition, text recognition, and tabular data analysis.

[0028] If machine learning techniques are appropriate for the application, the developer can use the machine learning tool 102 to generate a trained machine learning model 104. The machine learning tool 102 can present a template selector that presents options for different data categories from which the developer can choose. Categories can include general categories of image, text, sound, motion, and tabular data. Each category can also have subcategories. For example, image analysis can have subcategories of image classification and object detection.

[0029] The machine learning tool 102 may provide a template module 110. The template module 110 may provide a graphical user interface to allow developers to select a template from a plurality of templates. The templates may be organized by categories and subcategories, corresponding to different model types. Such a graphical user interface allows developers to select from different graphical representations of the aforementioned categories. The graphical user interface also provides a description of a particular template. For example, a developer may select a template by clicking on the description with a pointing device or selecting it on a touch screen. After selecting a template, the machine learning tool 102 may open a project view that may guide the developer through the steps of building a trained model. The stages for generating a trained machine learning model may include problem development, data collection, training the model, and deploying the trained model.

[0030] Figure 2 An exemplary graphical user interface 200 for selecting a template for generating a trained machine learning model is shown. The template GUI 200 depicts a template list 202 that may include image, text, sound, motion, and table templates, but in other embodiments, other templates may be provided. The developer may also display "all", such as Figure 2 , which displays all templates in a scrollable display. Categories 204 may be displayed above icons 206 for the templates. Figure 2 As shown, GUI 200 shows two templates for image category 204 . Figure 2 An icon 206 and a label 208 are depicted for each template in the category 204. For example, Figure 2 Icons 206 and labels 208 for an "Image Classifier" template and an "Object Detector" template are depicted. Figure 2 An image classifier template is shown selected and a text description box 210 provides additional information for the template.

[0031] Below is a brief description of the different classification types of models.

[0032] 1. Image Analysis Template

[0033] In some embodiments, the image analysis classification may have two different types of models. One model type is an image classification model that can be trained to classify objects depicted in a visual image. The second type of model is an object detection model. The object detection model allows for the detection of specific objects, animals, or people in an image. In one non-limiting example, a developer may want to create a flower image classification model. The flower classification model may analyze images of flowers to improve the core image classification model to detect specific flowers. For example, the flower classification model may correctly classify images of roses, daffodils, sunflowers, etc. from images captured by a user's smartphone.

[0034] Image classification is the problem of identifying what label to apply to an image from a set of categories. The first step is to collect training data 106. The training data 106 may include multiple image files. The image files may be in a variety of image file formats. Image file formats may include Joint Photographic Experts Group (JPEG), Exchangeable Image File Format (Exif), Tagged Image File Format (TIFF), Graphics Interchange Format (GIF), Windows Bitmap (BMP), Portable Network Graphics (PNG), and Better Portable Graphics (BPG). Other formats exist and can be used. Developers can collect many different types of images that reflect real data that the classifier can eventually see, and then label them. Ideally, the data should reflect the actual use of the application. Therefore, for example, if a developer is creating a custom image classifier that will be used by users on their smartphones, the developer should collect pictures from smartphones and possibly fewer screenshots.

[0035] Image classification models can provide initial class labels that provide insights about the image (e.g., whether the image was taken indoors or outdoors). Image classification models can also detect the room in which the image was captured (e.g., the kitchen). Object detection can add bounding boxes around objects in an image and suggest labels for the objects in the bounding boxes.

[0036] The machine learning tool 102 also incorporates a live mode. Using the live mode, developers can take images from a smartphone or other electronic device and test the accuracy of the image classifier during development.

[0037] 2. Text Analysis Template

[0038] The machine learning tool 102 can use at least two types of models for text analysis. One type of model is natural language processing, which helps process and understand the meaning of human language text. All machine learning in natural language processing driven by natural language is performed on the device to protect the privacy of the user. Text analysis models can include text classification and word tagging. A text classifier takes a block of text, such as a sentence or paragraph or an entire document, and applies a label to it. Text classification can be used to determine sentiment analysis, spam analysis, and topic analysis.

[0039] In the test classification workflow, the training data 106 may include raw text files organized in folders whose true labels are the names of the folders. Alternatively, the training data may use comma-separated values, where the text labels and true labels are separated by commas. JavaScript format is also supported.

[0040] In a text classifier, a language identifier can identify the native language of a text. A language identifier can identify the dominant language of a text file. A language hypothesis API can specify the maximum number of languages ​​contained in a text file and calculate the associated probability that the text is one of these languages. The text can then be converted into tokens, which can be further changed into feature values. The training data can then be used to train the core model of the text classifier by comparing the output with real data.

[0041] For example, a text classifier can be used to automatically evaluate multiple reviews. Thus, for text classification, developers can create applications to understand user reviews and expectations to automatically know whether a given review is a positive review or a negative review, or something in between. Such a process may be referred to as sentiment classification. Another exemplary use of a text classifier is to categorize articles by using article summaries or perhaps even just article titles. Another possible implementation is an automated travel agent that can determine the category of topics a customer is asking about. A text classifier can determine whether a customer is interested in a hotel or a restaurant or a flight.

[0042] A word tagger considers a sentence as a sequence of words and then applies a label to each word in the sentence in context. A word tagger can generate custom name taggers for automatic name recognition of custom names such as product names. A word tagger divides the sentence parts of the text into either sequential words or chunks of text in a text classifier. For example, a word tagger can tag words as people, organizational entities, slang words, etc. A word tagger parses unformatted text and interprets the structure of the words.

[0043] All machine learning for natural language processing can be performed on-device to protect the user's privacy.

[0044] Other features of text analysis include automatic speech tagging and named entity recognition. The named entity recognition API returns search results related to named entities in the text. Based on what the user has browsed in the web search tool, named entity recognition can suggest names and locations related to the browsing history, thereby enhancing the search experience in the application.

[0045] 3. Sound Analysis Template

[0046] The machine learning tool 102 may employ one or more sound analysis templates. A sound classifier may be used to identify specific instruments, such as cowbells, cymbals, drums, guitars, etc. Laughter and silence may also be classified. A sound classifier may be trained using multiple sound clips. Once trained, the sound classifier may analyze sample sound clips and determine the classification of the sound at different time windows.

[0047] The sound classifier can be combined with other features (e.g., live recording features), which can enable live listening and near real-time analysis of sound. In this live analysis mode, the machine learning tool 102 can listen to the microphone on the computing device locally or remotely by using the sound input from the microphone of a companion device such as a smart phone.

[0048] 4. Motion Analysis Template

[0049] The machine learning tool 102 may employ one or more motion analysis templates. A motion classifier may identify a specific motion captured from one or more sensors (e.g., accelerometers) on, for example, a smartphone or wearable device. For example, a motion analysis template may be trained in the appropriate form for forehand and backhand swings in tennis. The machine learning tool 102 enables developers to integrate the motion analysis template into an application for improving a tennis swing. Similar routines may be written to identify specific exercises. The motion analysis model may be trained to identify specific exercises and capture the number of repetitions for each exercise.

[0050] 5. Tabular data analysis template

[0051] The machine learning tool 102 may use one or more templates for analyzing tabular data. Tabular data may be structured data organized in a tabular format. An example may be house sales information for a given area. House sales information may include features of a house, such as the number of bedrooms, the number of bathrooms, the total square footage of the house, the proximity to schools, and the sale price of the house. Developers may want to use this information to predict the sale price based on one or more of these features. The machine learning tool 102 may import data as a comma separated value (CSV) file or in JavaScript format. Templates may allow easy calculations, such as price per square foot. Templates may also allow easy filtering of data, such as filtering out houses with less than a certain number of bedrooms.

[0052] B. Input for generating training data for machine learning models

[0053] return Figure 1, the machine learning tool 102 may allow the developer to input training data 106. The training data 106 may include multiple files stored in the memory of the computing device or on a network. The training module 116 may use the training data 106 to modify the selected template parameters to cause the trained model to be able to identify or classify image, sound, motion, or tabular data based on the selected template and training data. The training data 106 may include metadata that identifies one or more characteristics of the file (e.g., image, sound, or sensor data features). The machine learning tool 102 supports a large number of algorithms, such as boosted tree regression, random forests, etc. All of these algorithms are represented by their classification. In order to train the selected model, the developer will only have to write a line of code that identifies the target and data source. Therefore, in this case, the developer can choose linear regression or regularized linear regression, and the developer will identify the house data and the column is price.

[0054] The training data 106 may be structured data files in different formats. The data files may be associated with metadata files. In the case of structured data files, folders may provide information about the files contained within the files. For example, for a flower classifier, a file containing an image of a rose may be labeled "rose". The machine learning tool 102 may identify the name of the folder and associate features of the images contained in the folder with roses. In the case of sound analysis, a folder may contain sound clips of a guitar. The folder may be labeled "guitar" and the machine learning tool 102 may associate features of the sound clips contained within the folder with a guitar.

[0055] The training data 106 may be contained in the memory of the computing device on which the machine learning tool 102 is executing. The training data 106 may also be located in a storage device connected to the computing device. Alternatively, the training data 106 may be stored in a database provided from a cloud-based service.

[0056] In various embodiments, the data may be collected and labeled as a dictionary where a string label corresponds to an array of images. The data set may also be organized into a hierarchical directory structure so that the label is the name of the folder containing all the images. Other data sources may also exist, such as a single folder containing labeled file names.

[0057] The machine learning tool 102 may include a training data input module 112. The training data input module may include an application programming interface (API) for extracting training data 106 and associated data structures from a memory of a computing device. The training data 106 and associated data structures may be imported from a database server. The database server may be a cloud-based server accessible via a network (e.g., the Internet).

[0058] The machine learning tool 102 can provide errors and warnings to developers at various trigger points in the process. For example, when the developer drags and drops the training data 106, the technology can perform data analysis on the training data 106, and errors and warnings can be displayed on the user interface. At each stage of the process, the machine learning tool 102 can identify and display errors and warnings to the developer. Some errors and warnings can send the developer back to the previous stage for correction. In addition, the machine learning tool 102 also has a completely separate view to provide insights into the data independently of a specific task. For example, this separate view can inform the developer by drawing the distribution of labels that may be classified. In a separate view, a histogram can depict multiple annotations for each classification in different classifications of the classifier for the identified data source. The detailed view can inform the developer whether several images are duplicated or very similar. This allows the developer to understand the richness of the data.

[0059] Go to Figure 3 An input graphical user interface 300 for selecting training data for a sound classifier is shown. The input graphical user interface 300 lists the name 302 of a project file and a model template 304. Figure 1 The machine learning tool 102 shown may allow developers to select model templates from various model sources that can be further trained using training data. Figure 3 As shown, the name 302 of the project file is "MySoundClassifier2" and the model template 304 is "First Test". In input box 306, the number of detected classifications for the "First Test" model template may be displayed. Training box 308 may indicate the accuracy of the trained model on the selected training data. Validation box 310 may show the accuracy of the trained model on the validation data. Test box 312 may show the accuracy of the trained model on the test data. Output box 314 may display the name of the output file of the trained model. Figure 3 , output box 314 indicates that the model has not been trained.

[0060] The input graphical user interface 300 allows the developer to select a model template 304, training data 316, validation data 318, and test data 320. The graphical user interface 300 provides a first drop-down menu 322 to allow the developer to select the location of the training data 316. The training data 316 can be uploaded from a file in the memory of the computing device. Alternatively, the training data 316 can also be dragged and dropped from another file on the computing device onto the training data box 317. The developer can select one or more files to be used as validation data 318 and test data 320. In some embodiments, the developer can select a specific classification of data to be used as training data 316.

[0061] The input graphical user interface 300 provides a second drop-down menu 324 for selecting verification data 318. The verification data 318 can be uploaded from a file in the memory of the computing device. Alternatively, the verification data 318 can be dragged and dropped onto the verification data box 319. The developer can select one or more files to use as verification data 318. The developer can select "Automatic", in which the machine learning tool 102 suppresses a preselected amount of training data 316 (e.g., 10% of the training data 316) as verification data 318.

[0062] The input graphical user interface 300 provides a third drop-down menu 326 for test data 320. After the model has been trained and validated, the developer can upload additional test data 320 to test the classification of the trained model. The test data 320 can be uploaded from a file in the memory of the computing device. Alternatively, the test data 320 can be dragged and dropped onto the test data box 321.

[0063] The input graphical user interface 300 provides a status box 328 indicating "Training Data Required." This prompts the developer to identify the training data 316 using the interface.

[0064] Figure 4 An exemplary graphical user interface 400 for selecting training data for a sound classifier is shown. Figure 4 for Figure 3 Continuation of graphical user interface 300. Graphical user interface 400 lists the name of the project file 402 and the model template 404. Figure 4 As shown, the name 402 of the project file is "MySoundClassifier2" and the model template 404 is "First Test". In the input box 406, the number of detected classes in the training data 416 is displayed. The training box 408 shows the accuracy of the model on the selected training data. The validation box 410 shows the accuracy of the model on the validation data. The test box 412 shows the accuracy of the trained model on the test data. The output box 414 shows the name of the output file. Figure 4 , output box 414 indicates that the model has not been trained.

[0065] exist Figure 4 , graphical user interface 400 shows that training data 416 has been added. Machine language tool 102 has identified 1,608 items in training data 416. Input box 404 indicates that machine language tool 102 has identified 50 data classifications in training data 416.

[0066] The graphical user interface 400 allows the developer to select training data 416, validation data 418, and test data 420. The graphical user interface 400 provides a first drop-down menu 422 for selecting the location of the training data 416. The training data 416 can be uploaded from a file in the memory of the computing device. Alternatively, the training data 416 can be dragged and dropped onto the training data box 417. The developer can select one or more files to use as the training data 416. In some embodiments, the developer can select a specific classification of data to use as the training data 416. When the training data 416 is selected, the graphical user interface can display the number of items 421 contained in the training data 416.

[0067] The graphical user interface 400 provides a second drop-down menu 424 for selecting verification data 418. The verification data 418 may be uploaded from a file in the memory of the computing device. Alternatively, the verification data 418 may be dragged and dropped onto the verification data box 419. The developer may select one or more files to be used as the verification data 418. The developer may also select "Automatic" from the second drop-down menu 424, in which the machine learning tool 102 suppresses a preselected amount of the training data 416 as the verification data 418.

[0068] The graphical user interface 400 provides a third drop-down menu 420 for test data 420. After the model has been trained and validated, the developer can upload additional test data 420 to test the classification of the trained model. The test data 420 can be uploaded from a file in the memory of the computing device. Alternatively, the test data 420 can be dragged and dropped onto the test data box 423.

[0069] The graphical user interface 400 provides a status box 428 indicating that the model is “ready to train.” The “ready to train” notification prompts the developer to execute the training process by selecting a play button 430 .

[0070] return Figure 1 , the training data 106 can be stored in a file on the desktop of the integrated development environment. The developer can simply drag the training data folder to the designated area. The machine learning tool 102 can determine how many categories there are by the number of folders of training data and provide some high-level summary. For example, in the flower classifier example, the machine learning tool 102 can understand that the developer is generating a 17-category classifier with 2000 training images.

[0071] C. Training machine learning models

[0072] For example, the following machine learning models can be used: linear regression, logistic regression, decision tree, support vector machine (SVM), naive Bayes, K nearest neighbor (kNN), K means clustering, random forest, dimensionality reduction algorithm, gradient boosting algorithm (GBM) (e.g., XGBoost, LightGBM and CatBoost). Various techniques can use one or more algorithms to train the model to achieve the desired purpose of target accuracy, such as using gradient methods with back propagation, ensemble learning, variational Bayesian methods and adaboost.

[0073] After importing the training data 106, the model can be trained. The developer can start training by selecting the play button 430, such as Figure 4 As shown. The training module 116 uses the training data 106 to modify the model to predict the desired outcome. The training data 106 may extract one or more features from each training data asset. The training module 116 may associate one or more features of the training data asset with a label provided by an associated data structure. The label may also take the form of metadata associated with the training data 106. The training module 116 analyzes hundreds, thousands, or even millions of data elements to improve the machine learning (ML) model into a trained model. Ideally, after training, when presented with new, unseen data (i.e., test data 108), the model can predict the desired output.

[0074] The training module 116 can be used in conjunction with an auto-complete feature to incorporate the training data 106. The training module 116 allows for identification of training data in the form of a tag dictionary and data array, data source, or to specify model training parameters. The graphical user interface of the training module can include a console that informs the developer that the training module 116 is extracting features and the number of iterations the training module is running. The graphical user interface can provide a quick overview to inform the developer how many examples the training module 116 has evaluated and the number of categories that have been evaluated.

[0075] Figure 5 A training graphical user interface 500 for generating a machine learning model is shown. Figure 5 yes Figure 4 Continuation of graphical user interface 400 depicted in FIG. Graphical user interface 500 lists the name of the project file 502 and the model template 504. Figure 5, the name 502 of the project file is "MySoundClassifier" and the model template 504 is identified as "First Test". The graphical user interface 500 depicts a progress indicator 516 that shows the progress of the machine learning technique. The input box 506 shows that 50 classifications were detected in the training data. The training box 508 shows that the current training accuracy of the model is 93%. The validation box 510 shows that the accuracy of the trained model on the validation data is 82%. The test box 512 shows that the accuracy of the trained model on the test data is 73%. The output box 514 shows an icon 516 and the size of the output file (in bytes). Figure 5 The output file shown is 9.3 megabytes (MB).

[0076] The training graphical user interface 500 depicts a line graph of the training accuracy line 518 and the validation accuracy line 520 over the number of iterations of the model. The accuracy box 522 can provide further insights into the accuracy of the trained model for different classifications to the developer. For example, the training graphical user interface 500 can depict the precision 526 and recall 528 for each classification 524 of the data. Precision refers to the percentage of relevant results. Precision can be calculated as true positive results divided by actual results (true positive results plus false positive results). Recall refers to the percentage of total relevant results correctly calculated by the trained model. Recall can be calculated as true positive divided by predicted results or (true positive plus false negative). The accuracy of the training model can be calculated as true positive plus true negative divided by total results. Figure 5 The trained model shown is a sound classifier, so the model indicates various classifications of sounds (i.e., glass breaking, sneezing, chicken crowing, etc.).

[0077] Graphical user interface 500 provides a status box 528 indicating "Training completed after 1 hour and 30 minutes at 11:31 PM on May 26, 2019." The developer can make a copy of the results by selecting a "Make a Copy" button 534. The developer can also drag and drop the executable output file icon 515 into a file directory on the computing device.

[0078] Figure 6 An output graphical user interface 600 for the output of a machine learning model, specifically, a sound classifier, is shown. Figure 6 yes Figure 5 The output graphical user interface 600 may be selected by selecting the output file box 614. The output graphical user interface 600 lists the name of the project file 602 and the model template 604. Figure 66 shows that the name of the project file 602 is "MySoundClassifier" and the model template 604 is "First Test". The input box 606 shows that 50 classifications were detected in the training data. The training box 608 shows that the current training accuracy of the trained model is 93%. The validation box 610 shows that the accuracy of the trained model on the validation data is 82%. The test box 612 shows that the accuracy of the model on the test data is 73%. The output box 614 shows an icon 615 and the size of the output file (in bytes).

[0079] The output graphical user interface 600 allows selection of one of several test files 616. Figure 6 6-98866-A-47.wav" has been selected. The audio file 618 can be played by selecting a play button 620. The output graphical user interface 600 provides a graphical depiction 622 of a sound wave. The graphical depiction 622 displays a bar 624 showing the current position of the execution of the audio file 618. The graphical depiction 622 displays a current time 626 and a category label 628 above and below the graphical depiction 622. Figure 6 The portion of the audio file 618 in has received the classification label 628 of "aircraft". The output graphical user interface 600 provides a details section 630. For example, the details section 630 may include: the full file name of the output file, the size of the file (in megabytes), the model name, the author of the file, a license identifier, and a file description.

[0080] The output graphical user interface 600 provides a status box 632 indicating that "Training completed after 1 hour and 30 minutes at 11:31 PM on May 26, 2019." The developer can make a copy of the results by selecting the "Make a Copy" button 634. The developer can also drag and drop the executable output file 614 into a file directory on the computing device.

[0081] Figure 7 A progress bar 700 of a technique for generating a trained machine learning model is shown. The progress bar 700 is divided into three stages: setup 702, training 704, and output 706. The progress bar 700 shows that the input box 708 shows that 50 classifications were detected in the training data. The training box 710 shows that the current training accuracy of the trained model is 93%. The validation box 712 shows that the accuracy of the trained model on the validation data is 82%. The test box 714 shows that the accuracy of the trained model on the test data is 73%. The output box 716 shows the name and size (in bytes) of the output file.

[0082] Figure 8An output graphical user interface 800 for the output of a machine learning model, specifically, an image classifier, is shown. The output graphical user interface 800 may be selected by selecting an output file 818. The output graphical user interface 800 lists the name of the project file 802 and the model template 804. Figure 8 804. As shown, the name of the project file 802 is "FlowerClassifier" and the model template 804 is "My Model". The output graphical user interface 800 shows two data sources 806, specifically, flower training data 807 and flower test data 809. The setup box 810 shows that 17 classifications were detected in the training data. The training box 812 shows that the current training accuracy of the trained model is 100%. The validation box 814 shows that the accuracy of the trained model on the validation data is 73%. The test box 816 shows that the accuracy of the trained model on the test data is 82%. The output box 818 shows an icon 819 and the size of the output file (in bytes).

[0083] Figure 8 820 is shown displayed. An information box 822 for the test image 820 may also be displayed. The information box 822 may present the file name of the test image and the percentage at which the trained model depicts the test image 820 as a particular classification. For example, for Figure 8 , information box 822 indicates that the trained model classified the image 100% as Campanula rotundifolia and 0% as Snowdrop, Fritillary, Crocus, and Tulip. The output graphical user interface 800 provides a drag-and-drop interface 824 for evaluating other test images. The output graphical user interface 800 also provides summary details 826 for the image classifier being executed.

[0084] In some embodiments, transfer learning techniques may be employed. Transfer learning is a machine learning method in which a trained model developed for a task is reused as the starting point for a model on a second task. Transfer learning differs from traditional machine learning in that a pre-trained model that has been used for another task is used to jump-start the development process on a new task or problem. The benefit of transfer learning is that it can speed up the time it takes to develop and train a model by reusing these pieces or modules of the developed model. This helps speed up the model training process and accelerates results.

[0085] Models that have been trained on large datasets can often contribute parts of their code to other models. Modules can be used to build new models on new datasets. This is another way to speed up the model building process and gain efficiencies in your machine learning workflow.

[0086] Transfer learning techniques can be applied on top of this model that already exists in the machine learning tool. Using these techniques, the last few layers of the model can be retrained to the specific data, so the system no longer needs millions of data sets to train. In addition to shortening the time required for training, this can also result in much smaller models, from hundreds of megabytes down to just a few megabytes or even kilobytes for thousands of images.

[0087] return Figure 1 , the user interface of the machine learning tool 102 can allow developers to drag and drop add training data to the model and observe the builder in a live view. The training module 116 can use the graphics processing unit of the computing system to perform the analysis. If there are multiple sub-stages, the training module 116 can provide progress indications during training or even within the training stage. The progress indication can be a graphical representation or text information to inform the developer of the progress of the training. As the training progresses, the machine learning tool 102 can calculate and display the accuracy of the training model. The machine learning tool can dynamically update the accuracy as the training progresses. The machine learning tool can even distinguish the accuracy of different training classifications. For example, in a flower classifier, the machine learning tool 102 can distinguish the accuracy of roses from the accuracy of carnations. This can provide developers with insights into the model.

[0088] Previous machine learning tools do not save information at the various steps in generating a trained model. Therefore, developers cannot go back to see what was done to create a machine learning model. The disclosed technology allows developers to see how the trained model was developed to make modifications easier. By saving information to include accuracy and validation settings, developers can make changes to the training data to test the impact on model generation. In addition, previous models do not provide live interaction with mobile devices. By allowing developers to use data from mobile devices (e.g., a user's smartphone), this allows testing of machine learning tools by providing end-user interaction with the model before the application into which the machine language model is incorporated is even completed.

[0089] The training module 116 may detect errors during data training. In some cases, the data may be in an incorrect format. In some cases, there may be insufficient data for one or more classifications of the data. The training module 116 may display warnings and errors. An exemplary warning may be due to data imbalance. For example, there may be 100 more images of roses than chrysanthemums. Errors and warnings may even be represented graphically. The training module 116 may provide developers with suggestions for fixing errors and warnings. In some embodiments, the training module 116 may automatically correct errors (e.g., rebalance the classifications).

[0090] One way a developer can influence model development is by balancing the data. One way to balance the data can be by removing duplicates. Another way a developer can modify a model is by adding noise to the image if the image is very clear. The developer can also modify the orientation of the image.

[0091] The training module 116 inserts checkpoints at various points during the training of the model. The use of checkpoints allows the developer to return to a previous version of the model. Therefore, if the developer finds that including a new batch of training data adversely affects the trained model, the developer can easily return to the version of the trained model at a previous checkpoint that occurred before the inclusion of the new batch of training data. This saves the developer time during the training process because the developer can not only easily see the impact of the new data, but can also return to an earlier version of the model to modify the training data to improve the trained model.

[0092] Training module 116 can also be scheduled when the resources of the computing device are available (e.g., after hours or overnight). This allows developers to easily use the resources of the computing device to perform computing tasks and run intensive processing when demand for other tasks is low.

[0093] D. Select test data to evaluate the model

[0094] The machine learning tool 102 may allow a developer to input test data 108. Test data 108 is data that the machine learning tool 102 has not used for training and has not been seen by the machine learning tool 102 before.

[0095] The test data 108 may include one or more metadata files associated with the test data 108. The metadata files may identify the true identity of the files. For a classification model, the test data 108 should include files for several different classifications. For example, for a flower image classifier, the test data 108 should include multiple images of different flowers (e.g., roses, carnations, daffodils, etc.). Each image will have metadata that provides the true identity of each image of the test data 108.

[0096] The machine learning tool 102 may include a test data input module 114. The test data input module 114 may include an application programming interface (API) for extracting test data 108 and associated data structures from a memory of a computing device. The test data 108 and associated data structures may be imported from a database server. The database server may be a cloud-based server accessible via a network (e.g., the Internet). In some embodiments, the test data input module 114 may be the same module as the training data input module 112.

[0097] E. Evaluate the trained model using test data

[0098] The machine learning tool 102 may include an evaluation module 118 for evaluating the model using the test data 108. The evaluation module 118 may evaluate the trained model by analyzing features of the test data 108 that have not been seen before. The test data 108 may include metadata that may provide the true identity of the test data. In this way, the machine learning tool 102 may compare the output of the trained model with the real data. In this way, the machine learning tool 102 may calculate the accuracy of the trained model. The graphical user interface may display the percentage accuracy of the model. For example, for a flower classifier, the test data 108 may include multiple images of different types of flowers. The evaluation module 118 may test the accuracy of the flower classifier model by comparing the output of the flower classifier with the ground truth. The machine learning tool 102 allows the categories to be sorted by accuracy.

[0099] Another feature of the machine learning tool 102 is that it allows developers to drag images into the trained model and analyze the output. Figure 8 A drag and drop interface 824 is shown that allows a user to drag an image to test the trained model. Other models may provide similar interfaces to test other types of files (e.g., a sound classifier may allow a sound clip to be dragged into a drag and drop interface to classify a sound file). This provides the developer with details about a single prediction of the model. The developer may want to use an image from a camera on an electronic device (e.g., a smartphone). The machine learning tool 102 provides a live mode where the developer can switch to a mode that allows images to be captured from a smartphone and tested. Previously, the developer would have to write an application to play on a smartphone with the model, then use the application on the smartphone and write code to display some graphical details there. The machine learning model 102 is application-based and allows the model to be easily tested on a smartphone during development without writing a single line of code for the application.

[0100] F. Storing and exporting trained models

[0101] The machine learning tool 102 may include a storage module 120 for storing the trained and evaluated models in a memory of a computing system. The models may also be stored in a cloud storage accessible via a network (ie, the Internet).

[0102] The machine learning tool 102 may also include an export / import module 122. The export / import module 122 may convert the trained model into lines of code for incorporation into an application. Trained model export may be done in an integrated development environment. Code export may be as simple as dragging the trained model's icon into an application's folder.

[0103] The export / import module 122 may also allow developers to import models stored in the memory of a computing device. Once imported, the model may be retrained using additional training data and re-evaluated using additional test data.

[0104] II. Techniques for Generating Machine Learning Models Using Graphical User Interfaces

[0105] The technology discloses a graphical user interface for easily building machine learning models. These techniques can be used by novice developers who may have little experience programming in machine learning languages. The user interface guides the developer to select a template based on the type of model, select training data, identify test data, and export the trained model. The technology can also utilize real-time sensor information from one or more devices to evaluate the trained model. Fig. 9 A flow chart of a process 900 of techniques for generating a machine learning model using a graphical user interface is shown. The method can be performed on an electronic device (e.g., a laptop, desktop, or tablet).

[0106] At 902, the technology may include receiving a selection of a template from a plurality of templates as a machine learning model, each template corresponding to a classification of a data type. The selection may be made via a user interface. The template may be presented on a graphical user interface. The template may be presented by a classification of a plurality of classifications. The classification may include images, text, sound, motion, and tables. The template may be selected by interacting with a pointing device (e.g., a mouse, trackball, touch screen display). The selection of the template may be indicated on the graphical user interface by highlighting or changing the color and / or shading of icons and labels on the display.

[0107] At 904, the technique may include identifying a location of a plurality of training data, the plurality of training data including a first plurality of structured data records and a first associated metadata record, each metadata record in the first associated metadata record identifying at least a classification label of the first plurality of structured data records. The training data may be located on the graphical user interface using a drop-down menu. The training data may be located in a folder stored in a memory of the computing device. The training data may be located in a folder in a database in a cloud-based network. The developer identifies the location of the training data by dragging and dropping the selected training data into a designed training data box on the graphical user interface.

[0108] In some embodiments, the technology may include analyzing the plurality of training data to determine the number of classifications. In some embodiments, the technology may determine whether each of the plurality of classifications conforms to the plurality of training data. Thus, if, for example, audio training data is identified for an image classifier, the technology may generate an error or an alarm, and vice versa. The technology may also include selecting one or more of the plurality of determined classifications for training a machine learning (ML) model. A developer may want to exclude one or more determined classifications when training a model. For example, a developer may exclude "rose" from the classification of a flower classifier. The number of determined classifications may be displayed on a graphical user interface.

[0109] At 906, the technique may include training a machine learning (ML) model by analyzing each of the first plurality of structured data records and the first associated metadata records. The training may analyze each of the plurality of structured data records and the first associated metadata records for one or more elements to train the model. The elements may depend on the type of classifier. For example, image elements may be identified for an image classifier, text elements may be used for a text analyzer, and the like.

[0110] At 908, the technique may include identifying a location of a plurality of validation data, the plurality of validation data including a second plurality of structured data records and a second associated metadata record, each metadata record in the second associated metadata record identifying at least a classification label of the second plurality of structured data records. The validation data may be located on the graphical user interface using a drop-down menu. The validation data may be located in a folder stored in a memory of the computing device. The validation data may be located in a folder in a database in a cloud-based network. The developer identifies the location of the validation data by dragging and dropping the selected training data into a designed training data box on the graphical user interface. The technique may also receive a selection that the plurality of validation data may be automatically selected from a portion of the training data (i.e., 10% of the training data may be suppressed and not used for training but for validation).

[0111] At 910, the technique may include validating the trained model by analyzing each of the second plurality of structured data records to generate an identification of each of the second plurality of structured data records. The accuracy of the trained data may be determined by comparing the identified classifications with classification labels in metadata of the second plurality of structured data records. Precision and recall may be calculated for each data classification.

[0112] At 912, the technique may include displaying an accuracy score by comparing an identification of each of the second plurality of structured data records to a second associated metadata record to generate an accuracy score. The validation accuracy may be calculated during training and displayed on a graphical user interface as a numerical percentage or plotted in a line graph relative to the number of iterations. Precision and recall may be displayed for each data classification.

[0113] At 914, the technology may include generating code for a trained model, the code including the trained model and being executable on a mobile device to classify data obtained from a sensor of the mobile device. In various embodiments, the code may be executable code. The code may be saved as a file. Since the file may include executable code, the model may run on other mobile devices (e.g., smart phones, wearable devices, or tablet computing devices). In this way, developers can test machine learning models on devices for one or more beta testers without having to complete the coding of the application. Testing with images, sounds, or motions received from other mobile devices provides developers with insights into how the model works from actual sensors on the mobile device. The sensor may include a camera, a microphone, a motion sensor (e.g., an accelerometer), a keyboard, a keypad, or a touch screen display. In some embodiments, the technology may receive feedback from the one or more beta users on the accuracy of the classification provided by the model.

[0114] In some embodiments, the technique may include analyzing the plurality of training data to determine a plurality of classifications, each classification corresponding to a different classification label. The technique may include comparing the classification labels of the plurality of training data to determine consistency with the category of the data type.

[0115] In some embodiments, the technique may include analyzing the plurality of training data to determine a plurality of classifications, each classification corresponding to a different classification label. The technique may include displaying a list of the classifications of the plurality of training data on a graphical user interface. The technique may include receiving a selection of one or more of the plurality of classifications for use in training the machine learning model.

[0116] In some embodiments, the technique may include displaying a list of the classifications of the plurality of training data on a graphical user interface. The technique may include receiving a selection of one or more of the plurality of classifications for use in training the machine learning model.

[0117] In some embodiments, the technique may include receiving identification of locations of the plurality of training data by selecting an icon associated with the training data and dragging the icon to a designated area on a graphical user interface.

[0118] In some embodiments, identifying the location of the plurality of validation data includes automatically selecting a random portion of the training data. The random portion of the training data may be prevented from training the machine learning model.

[0119] In some embodiments, the technique may include receiving a selection to automatically select the plurality of validation data from the training data. The technique may include blocking a preselected percentage of the training data from training. The blocked training data may include a second plurality of structured data records. The technique may include validating the trained model by analyzing each of the second plurality of structured data records to generate an identification of each of the second plurality of structured data records and second associated metadata records. The technique may include generating an accuracy score by comparing the identification of each of the second plurality of structured data records to the second associated metadata record.

[0120] The technique may include detecting reaching a threshold of the plurality of training data. The threshold may be the number of training data records required to generate the machine language model. The technique may include automatically training the machine learning model upon identifying the threshold amount of training data. The technique may include generating code for the trained model.

[0121] In various aspects, a computing device includes one or more memories and one or more processors in communication with the one or more memories and configured to execute instructions stored in the one or more memories to perform any of the operations disclosed herein.

[0122] In various aspects, a non-transitory computer-readable medium stores a plurality of instructions that, when executed by one or more processors of a computing device, cause the one or more processors of the computing device to perform operations including any of the operations disclosed herein.

[0123] It should be understood that Fig. 9 The specific steps shown provide specific techniques for capturing and aggregating operational metrics for third-party applications according to various embodiments of the present disclosure. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps briefly described above in a different order. In addition, Fig. 9 The various steps shown may include multiple sub-steps, which may be performed in various sequences suitable for the various steps. Also, additional steps may be added or removed depending on the specific application. Those of ordinary skill in the art will recognize that there are many variations, modifications, and alternatives.

[0124] III. Exemplary Devices

[0125] Each of the methods described herein can be implemented by a computer system. Each step of these methods can be automatically performed by a computer system, and / or can have input / output involving a user. For example, a user can provide input for each step in the method, and each of these inputs can be in response to a specific output requesting such input, wherein the output is generated by a computer system. Each input can be received in response to a corresponding request output. In addition, the input can be received as a data stream from a user, from another computer system, retrieved from a memory location, retrieved through a network, requested from a web service, etc. Similarly, the output can be provided to a user, provided to another computer system, stored in a memory location, sent through a network, provided to a web service, etc. as a data stream. In short, each step of the method described herein can be performed by a computer system, and can involve any number of inputs, outputs, and / or requests to and from a computer system, which may or may not involve a user. Those steps that do not involve a user can be said to be automatically performed by a computer system without human intervention. Therefore, it can be understood in accordance with the present disclosure that each step of each method described herein can be changed to include inputs and outputs to and from a user, or can be automatically completed by a computer system without human intervention, wherein any determination is made by a processor. Furthermore, some embodiments of each of the methods described herein may be implemented as a set of instructions stored on a tangible, non-transitory storage medium to form a tangible software product.

[0126] The computing device can be a laptop computer with data stored locally in memory. Alternatively, multiple computing devices can be used for distributed processing using data located in a cloud service. The machine learning tool can also be executed on a tablet computer.

[0127] Fig.10 1 is a block diagram of an exemplary device 1000, which may be a mobile device on which the described techniques may be performed. The device 1000 generally includes a computer-readable medium (memory) 1002, a processing system 1004, an input / output (I / O) subsystem 1006, a wireless circuit 1008, and an audio circuit 1010 including a speaker 1050 and a microphone 1052. These components may be coupled via one or more communication buses or signal lines 1003. The device 1000 may be any portable electronic device, including a handheld computer, a tablet computer, a mobile phone, a laptop computer, a tablet device, a media player, a wearable device, a personal digital assistant (PDA), a key fob, a car key, an access card, a multifunction device, a mobile phone, a portable gaming device, an automotive display unit, etc., including a combination of two or more of these items.

[0128] Device 1000 can be a multifunctional device with display 1054. According to some embodiments, display 1054 can be a touch screen. The touch screen optionally displays one or more graphics in a user interface (UI). In some embodiments, the user can select one or more of these graphics by, for example, making gestures on the graphics with one or more fingers or with one or more styluses. In some embodiments, when the user interrupts contact with one or more graphics, selection of one or more graphics will occur. In some embodiments, gestures optionally include one or more taps, one or more swipes (from left to right, from right to left, up and / or down) and / or rolling of fingers that have been in contact with device 1000 (from right to left, from left to right, up and / or down). In some specific implementations or in some cases, inadvertent contact with a graphic will not select a graphic. For example, when the gesture corresponding to the selection is a tap, a swipe gesture that sweeps over an application icon optionally does not select the corresponding application. Device 1000 may optionally also include one or more physical buttons, such as a "home" button or a menu button. Optionally, the menu button is used to navigate to any application in a set of applications, optionally executed on computing device 1000. Alternatively, in some embodiments, the menu button is implemented as a soft key in a graphical user interface displayed on the touch screen.

[0129] Device 1000 may incorporate a display 1054. The display 1054 may be a liquid crystal display (LCD), an organic light emitting diode (OLED), an active matrix organic light emitting diode (AMOLED), an ultra-active matrix light emitting diode (AMOLED), a thin film transistor (TFT), an in-plane switch (IPS), or a thin film transistor liquid crystal display (TFT-LCD), which are commonly found in computing devices. Display 1054 may be a touch screen display of a computing device.

[0130] In one embodiment, the device 1000 includes a touch screen, a menu button, a push button for turning the device on and off and locking the device, one or more volume adjustment buttons, a user identity module (SIM) card slot, a headset jack, and a docking / charging external port. The push button is optionally used to turn the device on / off by pressing the button and keeping the button in the pressed state for a predefined time interval; lock the device by pressing the button and releasing the button before the predefined time interval passes; and / or unlock the device or initiate an unlocking process. In an alternative embodiment, the device 1000 also accepts voice input for activating or deactivating certain functions through a microphone. The device 1000 also optionally includes one or more contact strength sensors for detecting the intensity of contact on the touch screen, and / or one or more tactile output generators for generating tactile output for a user of the device 1000.

[0131] In an exemplary configuration, the device 1000 may include at least one computer-readable medium (memory) 1002 and one or more processing units (or processors) 1018. The processor 1018 may be suitably implemented in hardware, software, or a combination thereof. The computer-executable instructions or firmware implementation of the processor 1018 may include computer-executable instructions written in any suitable programming language to perform the various functions described.

[0132] Computer readable medium (memory) 1002 can store program instructions that can be loaded and executed on processor 1018 and data generated during the execution of these programs. Depending on the configuration and type of device 1000, memory 1002 may be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Device 1000 may have one or more memories. Device 1000 may also include additional removable storage devices and / or non-removable storage devices, including but not limited to magnetic storage devices, optical disks and / or tape storage devices. Disk drives and their associated non-transitory computer readable media can provide non-volatile storage of computer readable instructions, data structures, program modules and other data for the device. In some specific implementations, memory 1002 may include multiple different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM) or ROM. Although the volatile memory described herein may be referred to as RAM, any volatile memory that will not retain the data stored therein after being unplugged from the host and / or power supply is appropriate.

[0133] The memory 1002 and the additional storage device, whether removable or non-removable, are examples of non-transient computer-readable storage media. For example, a non-transient computer-readable storage medium may include volatile or non-volatile, removable or non-removable media, which is implemented by any method or technology to store information such as computer-readable instructions, data structures, program modules or other data. The memory 1002 and the additional storage device are examples of non-transient computer storage media. Additional types of computer storage media that may be present in the device 1000 may include, but are not limited to: phase change RAM (PRAM), SRAM, DRAM, RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital video disk (DVD) or other optical storage device, cassette, tape, disk storage device or other magnetic storage device or any other medium that can be used to store the desired information and can be accessed by the device 1000. Any combination of the above should also be included in the scope of non-transient computer-readable storage media. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art can think of other ways and / or methods of implementing various embodiments. However, as described above, computer-readable storage media do not include transient media such as carrier waves or the like.

[0134] Alternatively, the computer-readable communication medium may include computer-readable instructions, program modules, or other data transmitted within a data signal such as a carrier wave or other transmission means. However, as used herein, computer-readable storage media does not include computer-readable communication media.

[0135] The device 1000 may also include a communication connection 1008 that allows the device 1000 to communicate with a data repository, another device or server, a user terminal, and / or other devices via one or more networks. Such networks may include any one or combination of many different types of networks, such as a wired network, the Internet, a wireless network, a cellular network, a satellite network, other private networks, and / or public networks, or any combination thereof. The device 1000 may also include an I / O device 1006, such as a touch input device, a keyboard, a mouse, a pen, a voice input device, a display, a speaker, a printer, and the like.

[0136] Obviously, Fig.10 The architecture shown is merely one example of an architecture for device 1000 , and device 1000 may have more or fewer components or a different configuration of components than shown. Fig.10 The various components shown in the drawings may be implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing circuits and / or application specific integrated circuits.

[0137] Wireless circuitry 1008 is used to send and receive information to conventional circuitry (such as an antenna system, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, memory, etc.) of one or more other devices over a wireless link or network. Wireless circuitry 1008 may use various protocols, such as those described herein. For example, wireless circuitry 1008 may have a wireless protocol (e.g., ) and a separate component for another wireless protocol (e.g., ultra-wideband (UWB)). Different antennas can be used for different protocols.

[0138] The wireless circuit 1008 is coupled to the processing system 1004 via the peripheral device interface 1016. The interface 1016 may include conventional components for establishing and maintaining communications between peripheral devices and the processing system 1004. Voice information and data information (e.g., in voice recognition or voice command applications) received by the wireless circuit 1008 is sent to the one or more processors 1018 via the peripheral device interface 1016. The one or more processors 1018 can be configured to process various data formats of the one or more application programs 1034 stored on the computer-readable medium (memory) 1002.

[0139] Peripheral interface 1016 couples the device's input peripherals and output peripherals to processor 1018 and computer-readable medium 1002. Processor(s) 1018 communicate with computer-readable medium 1002 via controller 1020. Computer-readable medium 1002 may be any device or medium capable of storing code and / or data for use by processor(s) 1018. Medium 1002 may include a memory hierarchy including cache, main memory, and secondary memory.

[0140] Device 1000 may also include a power system 1042 for powering various hardware components. Power system 1042 may include a power management system, one or more power sources (e.g., batteries, alternating current (AC)), a recharging system, power fault detection circuits, power converters or inverters, power status indicators (e.g., light emitting diodes (LEDs)), and any other components typically associated with the generation, management, and distribution of power in a mobile device.

[0141] In some embodiments, the device 1000 includes a camera 1044. In some embodiments, the device 1000 includes a sensor 1046. The sensor 1046 may include an accelerometer, a compass, a gyroscope, a pressure sensor, an audio sensor, a light sensor, a barometer, etc. The sensor 1046 may be used to sense aspects of a location, such as an auditory marker or a light marker of a location.

[0142] In some embodiments, the device 1000 may include a GPS receiver, sometimes referred to as a GPS unit 1048. The mobile device may use a satellite navigation system such as a global positioning system (GPS) to obtain positioning information, timing information, altitude, or other navigation information. During operation, the GPS unit may receive signals from GPS satellites flying around the earth. The GPS unit analyzes the signals to estimate the transmission time and distance. The GPS unit may determine the current location (current position) of the mobile device. Based on these estimates, the mobile device may determine the position, altitude, and / or current speed. The position may be a geographic coordinate, such as latitude information and longitude information.

[0143] The one or more processors 1018 run various software components stored in the medium 1002 to perform various functions of the device 1000. In some embodiments, the software components include an operating system 1022, a communication module (or instruction set) 1024, a location module (or instruction set) 1026, a machine learning module 1028, an interactive development environment 1030, and other applications (or instruction sets) 1034.

[0144] Operating system 1022 can be any suitable operating system, including iOS, Macintosh operating system (MacOS), Darwin, Quatros real-time operating system (RTXC), LINUX, UNIX, OS X, Microsoft Windows or embedded operating system such as VxWorks. The operating system may include various programs, instruction sets, software components, and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.), and facilitate communication between various hardware and software components. Operating system 1022 is a system software that manages computer hardware and software resources and provides general services for computer programs. For example, operating system 1022 can manage the interaction between user interface module and one or more user applications. Various embodiments can also be implemented in a variety of operating environments, and in some cases, operating environments can include one or more user computers, devices or processing devices that can be used to operate any of multiple applications. User equipment or client equipment can include any of many general personal computers, such as desktop computers or laptop computers running standard operating systems, and cellular devices, wireless devices and handheld devices that run mobile software and can support multiple networking protocols and instant messaging protocols. This system may also include multiple workstations running any of a variety of commercially available operating systems and other known application programs for purposes such as development and database management, etc. These devices may also include other electronic devices such as virtual terminals, thin clients, gaming systems, and other devices capable of communicating via a network.

[0145] The communication module 1024 facilitates communication with other devices through one or more external ports 1036 or via the wireless circuit 1008, and includes various software components for processing data received from the wireless circuit 1008 and / or the external port 1036. The external port 1036 (e.g., a universal serial bus (USB), FireWire, a lightning connector, a 60-pin connector, etc.) is suitable for coupling to other devices directly or indirectly through a network (e.g., the Internet, a wireless local area network (LAN), etc.).

[0146] The location / motion module 1026 can help determine the current location (e.g., coordinates or other geographic location identifiers) and movement of the device 1000. Modern positioning systems include satellite-based positioning systems such as the Global Positioning System (GPS), cellular network positioning based on "cell IDs," and Wi-Fi positioning technology based on Wi-Fi networks. GPS also relies on the visibility of multiple satellites to determine location estimates, which may not be visible (or have weak signals) indoors or in "urban canyons." In some embodiments, the location / motion module 1026 receives data from the GPS unit 1048 and analyzes the signals to determine the current location of the mobile device. In some embodiments, the location / motion module 1026 can use Wi-Fi or cellular location technology to determine the current location. For example, the location of the mobile device can be estimated using knowledge of nearby cell locations and / or Wi-Fi access points and knowledge of their locations. Information identifying the Wi-Fi or cellular transmitter is received at the wireless circuit 1008 and transmitted to the location / motion module 1026. In some embodiments, the location module receives one or more transmitter IDs. In some embodiments, the sequence of transmitter IDs may be compared to a reference database (e.g., a cell ID database, a Wi-Fi reference database) that maps or associates transmitter IDs to location coordinates of corresponding transmitters, and based on the location coordinates of the corresponding transmitters, an estimated location coordinate of the device 1000 is calculated. Regardless of the particular positioning technology used, the position / motion module 1026 receives information from which a position fix can be derived, interprets the information, and returns location information, such as geographic coordinates, latitude / longitude, or other location fix data.

[0147] The electronic device may include a machine learning module 1028. The machine learning module 1028 receives a selection of one or more model templates, inputs a plurality of training data, trains a model on the training data, validates the model, and generates an executable file for the trained model.

[0148] The electronic device may also include an interactive development environment 1030. The interactive development environment in the operating system 1022 may include a software development tool kit for developers to generate software applications. The integrated development environment 1030 may support source code for the following programming languages: C, C++, Objective-C, Objective-C++, Java, AppleScript, Python, Ruby, ResEdit (Rez), and Swift, with various programming models, including but not limited to Cocoa, Carbon, and Java. Some integrated development environments also support GNU Pascal, Free Pascal, Ada, C#, Perl, and D.

[0149] The integrated development environment 1030 may include developer documentation, built-in interface builders, applications for constructing graphical user interfaces. Versions of the integrated development environment may include a modified version of the GNU compiler collection. The integrated development environment 1030 may support the development of microcomputers, portable computing system operating systems, smart phones, wearable devices, media access devices, and software development kits. The integrated development environment 1030 may also include a downloadable simulator.

[0150] The one or more applications 1034 on the mobile device may include any application installed on the device 1000, including but not limited to a browser, an address book, a contact list, email, instant messaging, word processing, keyboard emulation, desktop applets, JAVA-enabled applications, encryption, digital rights management, voice recognition, voice replication, a music player (playing back recorded music stored in one or more files such as MP3 files or Advanced Audio Coding (AAC) files), and the like.

[0151] There may be other modules or instruction sets (not shown), such as a graphics module, a time module, and the like. For example, the graphics module may include various conventional software components for rendering, animating, and displaying graphical objects (including but not limited to text, web pages, icons, digital images, animations, and the like) on a display surface. In another example, the timer module may be a software timer. The timer module may also be implemented in hardware. The timer module may maintain various timers for any number of events.

[0152] The I / O subsystem 1006 may be coupled to a display system (not shown) which may be a touch-sensitive display. The display system displays visual output to the user in the GUI. The visual output may include text, graphics, video, and any combination thereof. Some or all of the visual outputs may correspond to user interface objects. Although the display may use LED (light emitting diode) technology, LCD (liquid crystal display) technology, or LPD (light emitting polymer display) technology, other display technologies may be used in other embodiments.

[0153] In some embodiments, the I / O subsystem 1006 may include a display and a user input device such as a keyboard, a mouse and / or a touchpad. In some embodiments, the I / O subsystem 1006 may include a touch-sensitive display. The touch-sensitive display may also accept input from the user based on tactile and / or tactile contact. In some embodiments, the touch-sensitive display forms a touch-sensitive surface for accepting user input. The touch-sensitive display / surface (together with any associated modules and / or instruction sets in the medium 1002) detects contact (and any movement or release of contact) on the touch-sensitive display, and converts the detected contact into interaction with a user interface object, such as one or more soft keys displayed on the touch screen when contact occurs. In some embodiments, the contact point between the touch-sensitive display and the user corresponds to one or more fingers of the user. The user may contact the touch-sensitive display with any suitable object or accessory such as a stylus, a pen, a finger, etc. The touch-sensitive display surface may use any suitable touch-sensitive technology to detect contact and any movement or release thereof, including capacitive technology, resistive technology, infrared technology, and surface acoustic wave technology, as well as other proximity sensor arrays or other elements, for determining one or more contact points with the touch-sensitive display.

[0154] In addition, the I / O subsystem may also be coupled to one or more other physical control devices (not shown), such as buttons, keys, switches, rocker buttons, dials, slide switches, joysticks, LEDs, etc., for controlling or performing various functions, such as power control, speaker volume control, phone ring volume, keyboard input, scrolling, holding, menus, lock screens, clearing and ending communications, etc. In some embodiments, in addition to the touch screen, the device 1000 may include a touch pad (not shown) for activating or deactivating specific functions. In some embodiments, the touch pad is a touch-sensitive area of ​​the device that, unlike a touch screen, does not display visual output. The touch pad may be a touch-sensitive surface separate from a touch-sensitive display or an extension of a touch-sensitive surface formed by the touch-sensitive display.

[0155] In some embodiments, an application executed on a user's device may be used to perform some or all of the operations described herein. Circuits, logic modules, processors, and / or other components may be configured to perform the various operations described herein. Those skilled in the art will appreciate that, depending on the specific implementation, such configuration may be accomplished by design, setup, interconnection, and / or programming of specific components, and again depending on the specific implementation, the configured components may be reconfigurable or not reconfigurable for different operations. For example, a programmable processor may be configured by providing appropriate executable code; a dedicated logic circuit may be configured by appropriately connecting logic gates and other circuit elements; and so on.

[0156] Most embodiments utilize at least one network familiar to those skilled in the art to support communications using any of a variety of commercially available protocols, such as Transmission Control Protocol / Internet Protocol (TCP / IP), Open Systems Interconnection Model (OSI), File Transfer Protocol (FTP), Universal Plug and Play (UPnP), Network File System (NFS), Common Internet File System (CIFS), and AppleTalk. The network can be, for example, a local area network, a wide area network, a virtual private network, the Internet, an intranet, an extranet, a public switched telephone network, an infrared network, a wireless network, and any combination thereof.

[0157] In embodiments utilizing a web server, the web server may run any of a variety of server or middle-tier applications, including a Hypertext Transfer Protocol (HTTP) server, a File Transfer Protocol (FTP) server, a Common Gateway Interface (CGI) server, a data server, a Java server, and a business application server. The server or servers may also be capable of executing programs or scripts in response to requests from user devices, such as by executing one or more application programs, which may be implemented in any programming language such as One or more scripts or programs written in C, C# or C++ or any scripting language such as Perl, Python or TCL and their combinations. The one or more servers may also include a database server, including but not limited to and Those commercially available.

[0158] Such programs can also be encoded and transmitted using carrier signals, which are suitable for transmitting via wired networks, optical networks and / or wireless networks that meet various protocols, including the Internet. Similarly, the computer-readable medium according to an embodiment of the present invention can be created using data signals encoded by such programs. The computer-readable medium encoded with program code can be encapsulated with compatible devices or provide (for example, downloading via the Internet) independently of other devices. Any such computer-readable medium can exist in or be located in a single computer product (for example, hard disk drive, CD or entire computer system), and can exist in or be located in different computer products in a system or network. Computer systems can include monitors, printers or other suitable displays for providing any results mentioned herein to users.

[0159] Environment may include various data repositories and other memory and storage media, as described above. These may reside in various locations, such as on a storage medium local to one or more computers or away from the storage medium of any or all computers on the network (and / or reside in one or more computers). In a specific set of embodiments, information may reside in a storage area network (SAN) familiar to those skilled in the art. Similarly, any necessary files for performing functions belonging to a computer, server or other network device may be stored locally and / or remotely as required. When the system includes a computerized device, each such device may include a hardware element electrically coupled via a bus, the element including, for example, at least one central processing unit (CPU), at least one input device (e.g., mouse, keyboard, controller, touch screen or keypad), and at least one output device (e.g., display device, printer or speaker). Such systems may also include one or more storage devices, such as disk drives, optical storage devices and solid-state storage devices such as RAM or ROM, and removable media devices, memory cards, flash memory cards, etc.

[0160] Such devices may also include a computer-readable storage medium reader, a communication device (e.g., a modem, a network card (wireless or wired), an infrared communication device, etc.) and a working memory as described above. A computer-readable storage medium reader may be connected to or configured to receive a non-transient computer-readable storage medium representing a remote, local, fixed and / or removable storage device, and a storage medium for temporarily and / or more permanently containing, storing, transmitting and retrieving computer-readable information. Systems and various devices may also typically include a plurality of software applications, modules, services or other elements located in at least one working memory device, including an operating system and an application, such as a client application or a browser. It should be understood that alternative embodiments may have many variations according to those described above. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software (including portable software, such as applets), or both. In addition, connections with other devices such as network input / output devices may be used.

[0161] Any software component or function described in this patent application can be implemented as a software code executed by a processor, which uses any suitable computer language, such as, for example, Java, C, C++, C#, Objective-C, Swift, or a scripting language such as Perl or Python using, for example, conventional or object-oriented techniques. The software code can be stored on a computer-readable medium as a series of instructions or commands to achieve storage and / or transmission. Suitable non-transient computer-readable media may include random access memory (RAM), read-only memory (ROM), magnetic media (such as a hard drive or floppy disk), or optical media (such as a compact disk (CD) or DVD (digital versatile disk)), flash memory, etc. The computer-readable medium may be any combination of such storage devices or transmission devices.

[0162] Computer programs incorporating various features of the present disclosure may be encoded on various computer-readable storage media; suitable media include disks or tapes, optical storage media such as compact disks (CDs) or DVDs (digital versatile disks), flash memory, etc. Computer-readable storage media encoded with program code may be packaged with compatible devices or provided separately from other devices. In addition, program code may be encoded and transmitted via wired optical and / or wireless networks (including the Internet) that conform to a variety of protocols, thereby allowing distribution, for example, via Internet downloads. Any such computer-readable medium may reside in or be located in a single computer product (e.g., a solid-state drive, a hard drive, a CD, or an entire computer system), and may be present in or located in different computer products within a system or network. The computer system may include a monitor, a printer, or other suitable display for providing any results mentioned herein to a user.

[0163] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present disclosure as set forth in the claims.

[0164] As described above, one aspect of the present technology is to collect and use data from various sources to display rectangular content in non-rectangular areas. The present disclosure contemplates that, in some instances, these collected data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, twitter IDs, home addresses, data or records related to the user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying information or personal information.

[0165] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit the user. For example, the personal information data can be used to display information about the operational metrics of a third-party application. Therefore, the use of such personal information data can be presented to the user on a display. In addition, the present disclosure also anticipates other uses of personal information data that benefit the user. For example, health and fitness data can be used to provide insights into the user's overall health status, or can be used as positive feedback to individuals who use technology to pursue health goals.

[0166] The present disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data can comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate uses. In addition, such collection / sharing should be performed after receiving informed consent from the user. In addition, such entities should consider taking any necessary steps to defend and secure access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including specific considerations of jurisdiction. For example, in the United States, the collection or access of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained in each country for different types of personal data.

[0167] Regardless of the foregoing, the present disclosure also contemplates implementation schemes in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, with respect to third-party application evaluation technology, the technology of the present invention may be configured to allow users to choose to "opt in" or "opt out" to participate in the collection of personal information data at any time during or after registration for the service. As another example, a user may choose not to provide personal information that will be displayed. As another example, a user may choose to limit the amount of personal data to maintain or completely prohibit the display of personal data. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data can be accessed when downloading an application, and then the user may be reminded again just before the personal information data is accessed by the application.

[0168] In addition, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than at the address level), controlling how data is stored (e.g., aggregating data between users), and / or other methods when appropriate.

[0169] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, the various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, content may be selected and delivered to a user by inferring preferences based on non-personal information data or an absolute minimum amount of personal information, such as content requested by a device associated with the user, other non-personal information available to third-party application evaluation technology, or publicly available information.

[0170] Other variations are within the spirit of the present disclosure. Thus, while the disclosed technology is susceptible to various modifications and alternative constructions, certain illustrative embodiments thereof are shown in the drawings and have been described in detail above. However, it should be understood that it is not intended to limit the present disclosure to the particular forms disclosed, but on the contrary, it is intended to cover all modifications, alternative constructions, and equivalent forms that fall within the spirit and scope of the present disclosure as defined by the appended claims.

[0171] The use of the terms "one" and "an" and "the" and similar indicators in the context of describing the disclosed embodiments (particularly in the context of the claims below) will be interpreted as covering the singular and plural, unless otherwise stated or clearly contradicted by the context. Unless otherwise stated, the terms "comprise", "have", "include" and "contain" should be interpreted as open terms (i.e., meaning "including but not limited to"). The term "connection" is interpreted as being partially or completely included, attached or joined together even if there are intervening things. The phrase "based on" should be understood as open, and is not limited in any way, and is intended to be interpreted or otherwise understood as "at least partially based on" where appropriate. Unless otherwise stated herein, the description of the numerical range herein is intended only to be used as a simple method of referring to each individual value falling within the range individually, and each individual value is incorporated into the specification, as if quoted separately in this article. All methods described herein can be performed in any suitable order, unless otherwise stated herein or otherwise clearly contradicted by the context. Unless otherwise stated, the use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate the embodiments of the present disclosure and will not limit the scope of the present disclosure. No language in the specification should be construed as indicating that any unclaimed element is essential to the practice of the present disclosure.

[0172] Unless otherwise specifically stated, disjunctive language such as the phrase "at least one of X, Y, or Z" is understood in context as generally used to present an item, term, etc., which can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended to, and should not, imply that certain embodiments require that at least one of X, at least one of Y, or at least one of Z each be present. Additionally, conjunctive language such as the phrase "at least one of X, Y, and Z" is also understood to mean X, Y, Z, or any combination thereof, including "X, Y, and / or Z," unless otherwise specifically stated.

[0173] Preferred embodiments of the present disclosure are described herein, including the best mode known to the inventor for carrying out the present disclosure. After reading the foregoing description, variations of those preferred embodiments may become apparent to those of ordinary skill in the art. The inventor expects that the technician will appropriately adopt such variations, and the inventor intends to practice the present disclosure in a manner different from that specifically described herein. Therefore, as permitted by applicable law, the present disclosure includes all modifications and equivalents of the subject matter recorded in the appended claims. In addition, unless otherwise noted herein or clearly contradictory to the context, the present disclosure includes any combination of all possible variations of the above-mentioned elements.

[0174] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

Claims

1. A method for generating a machine learning model using a graphical user interface, the method being performed by an electronic device, the method comprising: receiving, via a user interface, a selection of a template from among a plurality of templates defining a machine learning model, each template corresponding to a category of data type, wherein the category of the data type comprises one of image, text, sound, or tabular data; identifying locations of a plurality of training data, the plurality of training data comprising a first plurality of structured data records and a first associated metadata record, each metadata record in the first associated metadata record identifying at least a classification label of the first plurality of structured data records; training the machine learning model by analyzing each of the first plurality of structured data records and the first associated metadata record to generate a trained model; identifying a location of a plurality of verification data, the plurality of verification data comprising a second plurality of structured data records and a second associated metadata record, each metadata record in the second associated metadata record identifying at least the classification label of the second plurality of structured data records; validating the trained model by analyzing each of the second plurality of structured data records to generate an identification of each of the second plurality of structured data records; displaying an accuracy score for the identification provided by the machine learning model, the accuracy score generated by comparing the identification for each of the second plurality of structured data records to the second associated metadata record; as well as An executable code for the trained model is generated, the executable code including the trained model and executable on a mobile device to classify data obtained from a sensor of the mobile device, wherein the sensor corresponds to the data type of the selected template.

2. The method according to claim 1, further comprising: Analyzing the plurality of training data to determine a plurality of classifications, each classification corresponding to a different classification label; as well as The classification labels of the plurality of training data are compared to determine consistency with the category of the data type.

3. The method according to claim 1, further comprising: Analyzing the plurality of training data to determine a plurality of classifications, each classification corresponding to a different classification label; displaying the classified list of the plurality of training data on the graphical user interface; as well as Receive a selection of one or more of the plurality of classifications for use in training the machine learning model.

4. The method according to claim 2, further comprising: displaying the classified list of the plurality of training data on the graphical user interface; as well as Receive a selection of one or more of the plurality of classifications for use in training the machine learning model.

5. The method according to any one of claims 1 to 3, further comprising receiving the identification of the locations of the plurality of training data by selecting an icon associated with the training data and dragging the icon to a designated area on the graphical user interface.

6. A method according to any one of claims 1 to 3, wherein identifying the location of the multiple verification data includes automatically selecting a random portion of the training data, wherein the random portion of the training data is blocked from training the machine learning model.

7. The method according to any one of claims 1 to 3, further comprising: receiving a selection to automatically select the plurality of validation data from the training data; blocking a preselected percentage of the training data from the training, the blocked training data comprising the second plurality of structured data records; validating the trained model by analyzing each of the second plurality of structured data records to generate the identification of each of the second plurality of structured data records and the second associated metadata records; as well as The accuracy score is generated by comparing the identification of each of the second plurality of structured data records to the second associated metadata record.

8. The method according to any one of claims 1 to 3, further comprising: Detecting a threshold value reaching the plurality of training data; as well as Automatically training the machine learning model; as well as The executable code for the trained model is generated.

9. A computing device comprising: one or more memories; and One or more processors in communication with the one or more memories and configured to execute instructions stored in the one or more memories to perform operations including: receiving, via a user interface, a selection of a template from among a plurality of templates defining a machine learning model, each template corresponding to a category of data type, wherein the category of the data type comprises one of image, text, sound, or tabular data; identifying locations of a plurality of training data, the plurality of training data comprising a first plurality of structured data records and a first associated metadata record, each metadata record in the first associated metadata record identifying at least a classification label of the first plurality of structured data records; training the machine learning model by analyzing each of the first plurality of structured data records and the first associated metadata record to generate a trained model; identifying a location of a plurality of verification data, the plurality of verification data comprising a second plurality of structured data records and a second associated metadata record, each metadata record in the second associated metadata record identifying at least the classification label of the second plurality of structured data records; validating the trained model by analyzing each of the second plurality of structured data records to generate an identification of each of the second plurality of structured data records; displaying an accuracy score for the identification provided by the machine learning model, the accuracy score generated by comparing the identification for each of the second plurality of structured data records to the second associated metadata record; as well as An executable code for the trained model is generated, the executable code including the trained model and executable on a mobile device to classify data obtained from a sensor of the mobile device, wherein the sensor corresponds to the data type of the selected template.

10. The computing device of claim 9, further comprising instructions stored in the one or more memories for performing operations comprising: Analyzing the plurality of training data to determine a plurality of classifications, each classification corresponding to a different classification label; and The classification labels of the plurality of training data are compared to determine consistency with the category of the data type.

11. The computing device of claim 9, further comprising instructions stored in the one or more memories for performing operations comprising: Analyzing the plurality of training data to determine a plurality of classifications, each classification corresponding to a different classification label; displaying the classified list of the plurality of training data on a graphical user interface; and Receive a selection of one or more of the plurality of classifications for use in training the machine learning model.

12. The computing device of any one of claims 10 to 11, further comprising instructions stored in the one or more memories for performing operations comprising: displaying the classified list of the plurality of training data on the user interface; and Receive a selection of one or more of the plurality of classifications for use in training the machine learning model.

13. The computing device of claim 9 further comprises instructions stored in the one or more memories for performing operations comprising: receiving the identification of the locations of the plurality of training data by selecting an icon associated with the training data and dragging the icon to a designated area on a graphical user interface.

14. The computing device of claim 9, wherein identifying the locations of the plurality of verification data comprises automatically selecting a random portion of the training data, wherein the random portion of the training data is blocked from training the machine learning model.

15. The computing device of claim 9, further comprising instructions stored in the one or more memories for performing operations comprising: receiving a selection to automatically select the plurality of validation data from the training data; blocking a preselected percentage of the training data from the training, the blocked training data comprising the second plurality of structured data records; validating the trained model by analyzing each of the second plurality of structured data records to generate the identification of each of the second plurality of structured data records and the second associated metadata records; as well as The accuracy score is generated by comparing the identification of each of the second plurality of structured data records to the second associated metadata record.

16. The computing device of claim 9, further comprising instructions stored in the one or more memories for performing operations comprising: Detecting a threshold value reaching the plurality of training data; automatically training the machine learning model; and The executable code for the trained model is generated.

17. A non-transitory computer readable medium storing a plurality of instructions that, when executed by one or more processors of a computing device, cause the one or more processors of the computing device to perform operations comprising: receiving, via a user interface, a selection of a template from among a plurality of templates defining a machine learning model, each template corresponding to a category of data type, wherein the category of the data type comprises one of image, text, sound, or tabular data; identifying locations of a plurality of training data, the plurality of training data comprising a first plurality of structured data records and a first associated metadata record, each metadata record in the first associated metadata record identifying at least a classification label of the first plurality of structured data records; training the machine learning model by analyzing each of the first plurality of structured data records and the first associated metadata record to generate a trained model; identifying a location of a plurality of verification data, the plurality of verification data comprising a second plurality of structured data records and a second associated metadata record, each metadata record in the second associated metadata record identifying at least the classification label of the second plurality of structured data records; validating the trained model by analyzing each of the second plurality of structured data records to generate an identification of each of the second plurality of structured data records; displaying an accuracy score for the identification provided by the machine learning model, the accuracy score generated by comparing the identification for each of the second plurality of structured data records to the second associated metadata record; as well as An executable code for the trained model is generated, the executable code including the trained model and executable on a mobile device to classify data obtained from a sensor of the mobile device, wherein the sensor corresponds to the data type of the selected template.

18. The non-transitory computer readable medium of claim 17, further instructions which, when executed by one or more processors of the computing device, cause the one or more processors of the computing device to perform operations comprising: Analyzing the plurality of training data to determine a plurality of classifications, each classification corresponding to a different classification label; and The classification labels of the plurality of training data are compared to determine consistency with the category of the data type.

19. The non-transitory computer readable medium of claim 17, further instructions which, when executed by one or more processors of the computing device, cause the one or more processors of the computing device to perform operations comprising: Analyzing the plurality of training data to determine a plurality of classifications, each classification corresponding to a different classification label; displaying the classified list of the plurality of training data on a graphical user interface; and Receive a selection of one or more of the plurality of classifications for use in training the machine learning model.

20. The non-transitory computer readable medium of any one of claims 17 to 19, further instructions which, when executed by one or more processors of the computing device, cause the one or more processors of the computing device to perform operations comprising: receiving a selection to automatically select the plurality of validation data from the training data; blocking a preselected percentage of the training data from the training, the blocked training data comprising the second plurality of structured data records; validating the trained model by analyzing each of the second plurality of structured data records to generate the identification of each of the second plurality of structured data records and the second associated metadata records; as well as The accuracy score is generated by comparing the identification of each of the second plurality of structured data records to the second associated metadata record.

Images