Building object detection and image processing applications

Abstract

Systems / methods for developing and deploying ML-based object detection applications provide an object detection application builder that streamlines the building of object detection applications. The application builder is composed of a series of user interface screens and underlying processes that allow a user to label training data, train an object detection model, evaluate model performance, and deploy the model to end users, all without requiring coding or requiring only a minimal amount of coding. Such an application builder can then be hosted on a third-party network and provided as a service to customers, or it can reside on a private enterprise network for access and use by enterprise employees and personnel. In either case, users who are non-data scientists and non-programmers can then use the application builder to quickly and easily develop and deploy object detection and image processing applications for their specific use cases and requirements.

Description

BUILDING OBJECT DETECTION AND IMAGE PROCESSING APPLICATIONSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This international application for patent claims the benefit of priority to and incorporates herein by reference Indian Provisional Application No. 202311057121, entitled "Building Object Detection and Image Processing Applications," filed August 25, 2023.TECHNICAL FIELD

[0002] Embodiments of the present disclosure relate to the use of artificial intelligence (Al), or machine learning (ML), to analyze and detect objects within images and videos and, more particularly, to systems and methods that allow users to easily and quickly build and deploy applications that use ML-based object detection models to analyze and detect objects within images and videos.BACKGROUND

[0003] Object detection generally refers to the use of machine learning to detect or recognize objects present in images and videos. The process typically involves training a machine learning model to recognize specific objects by exposing the model to images, videos, and other training data that contain the target objects. From the training data, the model learns to recognize certain physical features and characteristics that distinguish the target objects. The model then looks for statistically similar patterns in subsequent images and videos to detect and recognize the target objects in the images and videos.

[0004] Developing and deploying an object detection application has heretofore been an enormously complex undertaking requiring data scientists, software programmers, network engineers, and other highly trained technical personnel, all working together in a coordinated effort. Consequently, only companies and organizations with significantresources were able to take advantage of the capabilities and benefits of object detection and image processing. Although recent offerings from cloud computing platforms like Amazon Web Services (AWS) and Microsoft Azure have helped make this technology more feasible, for the majority of companies, technical hurdles remain to developing and deploying their own object detection and image processing applications.SUMMARY

[0005] Embodiments of the present disclosure relate to systems and methods for developing and deploying applications that use artificial intelligence (Al) and machine learning (ML) to perform object detection and image processing. The systems and methods provide an object detection application builder that streamlines the building of object detection and image processing applications. The application builder is composed of a series of user interface screens and underlying processes that allow a user to label training data (i.e., images), train an object detection model, evaluate model performance, and deploy the model to end users, all without requiring coding or requiring only a minimal amount of coding. Such an application builder can then be hosted on a third-party network and provided as a service to customers, or it can reside on a private enterprise network for access and use by enterprise employees and personnel. In either case, users who are non-data scientists and non-programmers can then use the application builder to quickly and easily develop and deploy object detection and image processing applications for their specific use cases and requirements.

[0006] In general, in one aspect, embodiments of the present disclosure relate to a system for building and deploying an object detection application. The system comprises, among other things, a processor, and a storage unit communicatively coupled to the processor. The storage unit stores an object detection application builder thereon that, when executed bythe processor, causes the system to perform a process that allows a user to provide a plurality of digital images to a computing resource, the plurality of digital images specified by the user based on a particular use case. The object detection application builder, when executed by the processor, additionally causes the system to perform a process that allows the user to label one or more objects to be detected in the plurality of images using the computing resource. The object detection application builder, when executed by the processor, also causes the system to perform a process that uses the plurality of images to perform machine learning based training of an object detection model using the computing resource. The object detection application builder, when executed by the processor, further causes the system to perform a process that generates an endpoint for the object detection model using the computing resource, and perform a process that deploys the endpoint for the object detection model to a hosted environment using the computing resource, the hosted environment specified by the user.

[0007] In general, in another aspect, embodiments of the present disclosure relate to a method for building and deploying an object detection application. The method comprises, among other things, providing, at a computing resource, a plurality of digital images, the plurality of digital images provided to the computing resource by a user based on a particular use case. The method also comprises allowing, at the computing resource, the user to label one or more objects to be detected in the plurality of images, and training, at the computing resource, an object detection model using the plurality of images that have been labeled, wherein the training is machine learning based training. The method further comprises generating, at the computing resource, an endpoint for the object detection model, and deploying, at the computing resource, the endpoint for the object detection model to a hosted environment, the hosted environment specified by the user.

[0008] Tn general, in yet another aspect, embodiments of the present disclosure relate to a non-transitory computer-readable medium storing computer-readable instructions thereon. The computer-readable instructions comprise, among other things, instructions for causing a processor to perform a process that allows a user to provide a plurality of digital images to a computing resource, the plurality of digital images specified by the user based on a particular use case. The instructions additionally comprise instructions for causing a processor to perform a process that allows the user to label one or more objects to be detected in the plurality of images using the computing resource. The instructions also comprises instructions for causing a processor to perform a process that uses the plurality of images that have been labeled to perform machine learning based training of an object detection model using the computing resource. The instructions further comprise instructions for causing a processor to perform a process that generates an endpoint for the object detection model using the computing resource, and instructions for causing a processor to perform a process that deploys the endpoint for the object detection model to a hosted environment using the computing resource, the hosted environment specified by the user.

[0009] In accordance with any one or more of the foregoing embodiments, the object detection application builder further causes the system to perform a process that uses a set of evaluation images to perform testing of the object detection model using the computing resource, the set of evaluation images specified by the user based on the particular use case. In accordance with any one or more of the foregoing embodiments, the object detection application builder further causes the system to perform a process that displays one or more performance metrics for the object detection model based on the testing of the object detection model.

[0010] In accordance with any one or more of the foregoing embodiments, the object detection application builder causes the system to perform a process that allows the user tolabel one or more objects to be detected in the plurality of images by allowing the user to graphically draw a bounding box around each object to be detected in the plurality of images.

[0011] In accordance with any one or more of the foregoing embodiments, the object detection application builder causes the system to perform a process that allows the user to label one or more objects to be detected in the plurality of images by allowing the user to assign an object class to each object to be detected in the plurality of images.

[0012] In accordance with any one or more of the foregoing embodiments, the object detection application builder causes the system to perform a process that allows the user to provide the plurality of digital images to the computing resource by allowing the user to specify a location of the digital images, or allowing the user to drag and drop the digital images into a designated field.BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The foregoing and other advantages of the disclosed embodiments will become apparent upon reading the following detailed description and upon reference to the drawings, wherein:

[0014] FIG. 1 illustrates an exemplary process for ML-based object detection in accordance with embodiments of this disclosure;

[0015] FIG. 2 illustrates an exemplary process for an object detection application builder in accordance with embodiments of this disclosure;

[0016] FIG. 3 illustrates an exemplary application builder screen for allowing a user to upload images for training purpose in accordance with embodiments of this disclosure;

[0017] FIG. 4 illustrates an exemplary application builder screen for allowing a user to label images for training purpose in accordance with embodiments of this disclosure;

[0018] FIG. 5 illustrates an exemplary application builder screen for allowing a user to train models in accordance with embodiments of this disclosure;

[0019] FIG. 6 illustrates an exemplary application builder screen for allowing a user to evaluate trained models in accordance with embodiments of this disclosure;

[0020] FIG. 7 illustrates an exemplary application builder screen for allowing a user to deploy trained models in accordance with embodiments of this disclosure;

[0021] FIG. 8 illustrates an exemplary deployment of an object detection application in accordance with embodiments of this disclosure;

[0022] FIG. 9 illustrates a flowchart for an exemplary method of building an object detection application in accordance with embodiments of this disclosure;

[0023] FIG. 10 illustrates an exemplary computing device that may be used to implement various embodiments of this disclosure; and

[0024] FIG. 11 illustrates an exemplary storage system that may be used to implement various embodiments of this disclosure.DETAILED DESCRIPTION

[0025] The features and other details of the concepts, systems, and techniques sought to be protected herein will now be more particularly described. It will be understood that any specific embodiments described herein are shown by way of illustration and not as limitations of the disclosure and the concepts described herein. Features of the subject matter described herein can be employed in various embodiments without departing from the scope of the concepts sought to be protected.

[0026] As alluded to above, the systems and methods herein allow users to easily and quickly build and deploy applications that use ML-based object detection models to analyze and detect objects within digital images and videos of the type frequently captured usingpicture and video cameras. Tn some embodiments, the systems and methods provide an object detection application builder having a series of user interface screens or display tabs and underlying processes that simplify the building of object detection and image processing applications. The application builder lets users label training data (i.e., images), train an object detection model using the training data, evaluate model performance, and deploy the model to end users, without requiring the users to perform coding (i.e., write software code) or have coding experience.

[0027] Although the application builder of the present disclosure can use any ML-based object detection model, for illustrative purposes, the exemplary implementation discussed herein employs a YOLO ("You Only Look Once") object detection model. The YOLO algorithm, particularly version 7 ("YOLOv7"), is considered by many to be the fastest and most accurate object detection model available. Similarly, although the application builder of the present disclosure can use any available computing resource, whether a private enterprise computing resource or cloud-based computing resources, for illustrative purposes, the exemplary implementation discussed herein leverages Amazon Web Services (AWS). AWS is a general-purpose cloud computing platform that provides users with access to, among other things, data storage services, compute services, and other cloudbased computing services. Indeed,

[0028] Referring now to FTG. 1, a high-level flow diagram 100 is shown representing an exemplary ML-based object detection process on which an object detection application may be built in accordance with embodiments of this disclosure. In this example, a raw image file 102 containing a certain target object 104 is processed by an ML-based object detection model 106 that has been trained to detect the target object 104. The target object 104 in this example is a traffic light and the ML-based object detection model 106 is a YOLOv / model. It will be appreciated that other object detection models known to those skilled in the artmay also be used, such as R-CNN (Region-based Convolutional Neural Networks), Fast R-CNN, SSD (Single- Shot Detector), among others. Upon processing the image 102, the object detection model 106 recognizes the presence of the traffic light in the image 102 and places a box 108 around the portion of the image that contains the traffic light. Although an image is used in this example, it should be understood that videos may be processed in a similar manner using an appropriately trained object detection model.

[0029] FIG. 2 shows another high-level flow diagram, this one representing an exemplary process 200 that can be used by or with the object detection application builder herein to develop and deploy an object detection application. As can be seen, the application building process 200 represented by the flow diagram is composed of a series of graphical icons, each icon representing a certain function or a particular service. It should be understood that two or more of these icons may be combined into a single function or service, and any single icon may be divided into two or more constituent functions or services, as needed. The process 200 allows virtually any user to quickly and easily develop and deploy an object detection application for their specific use case or requirements. Once deployed, images and videos can be fed to the application to detect, for example, defective components on a manufacturing assembly line, suspicious activity in a parking lot, customer movement in a store, and numerous other use cases.

[0030] As FIG. 2 shows, the application building process 200 includes four main phases, an annotation phase 202 in which users label or annotate raw images and videos for model training, a training phase 204 in which the labeled or annotated images and videos are used to train a model, an evaluation phase 206 in which the performance of the trained model is presented to the user for evaluation, and a deployment phase 208 in which the trained model is packaged in a ready-to-deploy form that the user can easily deploy as an endpoint on a hosted environment. In some embodiments, the process 200 may use a cloud computingplatform like AWS, as shown here, to implement the various phases of the process 200. In some embodiments, a graphical user interface 210 is provided to allow the user to easily navigate the different phases of the process 200. The user interface 210 may be a Web interface created using Shiny, for example, or a similar web application development package.

[0031] In general, the application building process 200 begins with the annotation phase 202 where an AWS Lambda compute service or other serverless computing service is used at 212 to provision a pre-signed Amazon S3 (Simple Storage Service) URL that a user can access at 214 to upload raw images 216 for model training, for example, as a zipped or other compressed file format. Provisioning of the pre-signed S3 URL may involve action by an Amazon API Gateway via a POST or functionally similar request method. Note that while the embodiment here uses images for model training, in other embodiments, raw videos may be used for model training by following a similar process. At 218, a JSON (JavaScript Object Notation) file is generated containing information about the uploaded images, and the uploaded images and JSON file are then stored in an Amazon S3 folder at 220. The AWS Lambda compute service is used to extract the uploaded images at 222, decompress the images at 224, and open a manifest file and request a labeling job at 226. Labeling is performed at 228 and involves the user identifying target objects (e.g., traffic light, etc.) in an uploaded image by marking the area in the image that contains the objects. In some embodiments, the labeling may be implemented using Amazon SageMaker Ground Truth, although other data labeling services may also be used. The labeling that the user applies to each image is then stored in the manifest file, which may be a JSON file.

[0032] After the annotation phase 202 is complete, the training phase 204 of the process 200 generally involves sending the manifest file at 230 to the AWS Lambda compute service, which may involve action by an Amazon API Gateway at 232. The AWS Lambda computeservice applies the manifest file to an ML-based object detection model at 234 to train the model. The object detection model being trained in this example is a YOLOv7 model, although as mentioned above, other object detection models known to those having ordinary skill in the art may also be used. The training of this model can be accomplished at 236 via the previously labeled images (indicated by dotted line arrow) using, for example, AWS Batch, which is a service that can efficiently run large volumes of machine learning computing jobs. The trained model is then stored in an Amazon S3 folder at 238.

[0033] After completion of the training phase 204, the evaluation phase 206 of the process 200 can be performed. The evaluation phase 206 generally involves applying one or more hold-out images to the trained model at 240. The hold-out images are a subset of images that are chosen by the user from the originally uploaded images and are set aside for model evaluation purposes and not used as part of the model training. Alternatively, the user may upload one or more additional sets of images that are similar in nature to the originally uploaded images for use as the valuation images. In either case, the trained model is then used (indicated by dotted line arrow) to process these hold-out images via an AWS Lambda compute service at 242 (or other serverless computing service) as a test of the model's ability to detect target objects of the type identified in the labeled images. In some embodiments, the trained model is packaged in a Docker container or similar container that supports YOLO inference requests. The Docker container may be run from Amazon SageMaker or similar cloud-based machine learning platform to provide an instance of the model. Other suitable containerization options known to those skilled in the art besides Docker may also be used as appropriate for the purposes herein, including Kubernetes, Podman, Singularity, and the like.

[0034] In the example shown, there are Docker containers for three different YOLOvZ models, indicated at 244, 246, and 248, respectively, each having been trained using threedifferent sets of labeled images. The user may then access one of these trained models by operation of an Amazon API Gateway and process the hold-over images to evaluate that model. The results of the evaluation (i.e., how well the trained models were able to detect objects in the hold-over images) are then presented to the user at 250. The user has the option to repeat any of the previous phases using additional uploaded images, or proceed to the deployment phase 208 if he or she is satisfied with the performance trained models.

[0035] The deployment phase 208 of the process 200 generally involves submitting a request at 252, which may involve using a JSON file, to deploy one of the trained models indicated at 244, 246, and 248. Submission of the request may be accomplished by action of an Amazon API Gateway at 254. The SageMaker compute instance for the relevant model thereafter generates the YOLOv7 weights file or model endpoint and stores the file in an Amazon S3 folder. The user may then deploy the weights file or model endpoint to a hosted environment as needed.

[0036] FIG. 3 shows an exemplary object detection application builder 300 that implements the underlying application building process 200 from FIG. 2. As mentioned previously, the application builder 300 is composed of a series of user interface screens, depicted here as display tabs, that each implement a portion of the process 200 from FIG. 2. Thus, there is a label screen 302 that implements the annotation phase 202, a train screen 304 that implements the training phase 204, an evaluate screen 306 that implements the evaluation phase 206, and a deploy screen 308 that implements the deployment phase 208. As alluded to earlier, the user interface screens 302, 304, 306, 308 may be Web pages created using, for example, Shiny or a similar Web application development package.

[0037] In general, the label screen 302 allows a user to easily upload images for labeling and to easily label the images. Labeling begins with a bucket field 310 that allows the user to enter a bucket name or project name for a particular project, such as "nam-ai-YOLOv7test-zb2." There is also a browse button 314 for allowing the user to search for and upload a set of images to be labeled for the particular project, preferably as a zipped or other compressed file format. Clicking on this button 314 uploads the selected set of images to an Amazon S3 folder via a pre-signed Amazon S3 URL. Alternatively, instead of browsing for the images, the user can simply drag and drop the images, or a compressed file containing the images, into a field next to the browse button 314.

[0038] The label screen 302 also includes a class number field 316 for allowing the user to specify how many classes of objects to be detected are in this project. Here, the user has specified four object classes, which has spawned four class name fields 318a, 318b, 318c, 318d. These fields allow the user to specify the name of each class of objects to be detected in the project. For example, the user may specify a stop sign class, a crosswalk sign class, a traffic light class, and a speed limit sign class. It is thus up to the user to choose which images to train, and to label or annotate the images, classify the objects, and determine which object detection model is best suited for their particular purpose. Each project can use a different object recognition model or a version of the same model trained using a different set of labeled images.

[0039] Clicking on an execute ("Go!") button 320 saves the project name, number of classes, class names, and other information entered in this screen, and automatically takes the user to a screen for labeling the uploaded images, shown in FIG. 4.

[0040] FIG. 4 shows an exemplary screen 400 for allowing a user to easily label uploaded images in some embodiments for model training purposes. This screen may be provided as part of SageMaker Ground Truth in some embodiments, as mentioned earlier. From here, the user may view an image 402 from the set of uploaded images and identify any target objects found in the image. For example, there are two traffic light objects in the image 402, one indicated at 404 and another indicated at 406, as well as a stop sign object, indicated at408. The user may then choose a "traffic light" class from the previously defined classes 318a-d, and graphically draw a bounding box 410 and 412 (e.g., using a mouse, stylus, etc.) around each traffic light object 404 and 406, respectively. The user may next choose a "stop sign" class from the previously defined classes 318a-d, and graphically draw a bounding box 414 (e.g., using a mouse, stylus, etc.) around the stop sign object 408. If there are further target objects to be to be labeled in the image, the user may repeat the process for each remaining target object.

[0041] Clicking on a submit button 416 completes the labeling for the current image and automatically brings up the next image in the set of images.

[0042] FIG. 5 shows an example of the train screen 304. In general, the train screen 304 allows a user to easily train an object detection model using a set of labeled images. As discussed, the object detection model used herein is the YOLOv7 model. Training begins by selecting a particular project via a bucket name or project name field 320. The labeled images from this project will then be used to train the object detection model. An epochs field 322 allows the user to specify the number of epochs to be run in a given training run, or job, and a run name or job name field 324 allows the user to name each training run or job. Clicking on an execute ("Go!") button 326 saves the information entered in this screen and automatically initiates the training session.

[0043] The train screen 304 also includes a job history section 328 that allows the user to view the progress of the current training run and the statuses of previous training run, including date and time when they were created, date and time when they were started, and date and time when they were stopped. Clicking on a refresh button 330 causes the list of jobs to update. Similarly, a logs section 332 allows the user to view various entries in the log for the current training job, including a timestamp and message for each entry. Clicking on another refresh button 334 causes the log to update.

[0044] FIG. 6 shows an example of the evaluate screen 306. In general, the evaluate screen 306 allows a user to evaluate the performance of the trained YOLOv7 model using a holdover set of images and view the results of the evaluation. Evaluation begins by again selecting a particular project via a bucket name or project name field 336. A select run field 338 allows the user to specify the particular training run to be used. The YOLOv7 model that was trained during the selected training run is then used for the evaluation. Clicking on an execute ("Go!") button 340 causes the hold-over images to be processed through the YOLOv7 model.

[0045] A metrics section 342 is provided that displays various performance metrics that a user may use to evaluate the performance of the YOLOv7 model. Common performance metrics include Box, Objectiveness, Classification, Precision, Mean Average Precision (mAP), Recall, and the like. These metrics are well known to those having ordinary skill in the field of object detection models are only briefly mentioned here. For example, Classification is a well-known metric for assessing the accuracy of a model and typically refers to the ratio of correct detections to possible detections. Objectness provides a probability that an object exists in a particular region of interest, with a high Objectness indicating that the region likely contains an object. In some embodiments, the user may select one or more of these metrics for display in the metrics section 342 according to the particular needs of the application. Other performance metrics besides the ones listed here may also be used within the scope of the disclosed embodiments.

[0046] A confusion matrix 344 is also provided that visually renders the performance of the YOLOv7 model. The confusion matrix 344, as its name suggests, presents the performance of the model in the form of a matrix that can be used to derive various performance metrics. For example, the accuracy of the model can be calculated by taking the average of the values in the cells forming the main diagonal of the matrix, and so forth.Finally, the processed hold-over images are shown at 346 along the bottom of the evaluate screen 306. As can be seen from these images, this model was able to accurately recognize speed limit signs 348, traffic lights 350, stop signs 352, as well as crosswalks 354 in the images.

[0047] FIG. 7 shows an example of the deploy screen 308. In general, the deploy screen 308 allows a user to generate a model endpoint for the trained YOLOv / model and subsequently deploy the endpoint to an intended hosted environment, such as a cloud computing environment. Deployment begins by once again selecting a particular project via a bucket name or project name field 356. A select run field 358 allows the user to specify the particular training run to be used. The YOLOv7 model that was trained during the selected training run is then used for the deployment. Clicking on an execute ("Go!") button 360 causes the SageMaker compute instance to create the model endpoint, which includes the weights file for the YOLOv7 model.

[0048] As those having ordinary skill in the art understand, weights files, or model weights files, are specially formatted files containing all parameters, including trainable and non-trainable parameters, required to make a model work. These parameters, particularly the trainable ones, are the result of the training process discussed herein and provide the model with its predictive capabilities. Model weights work directly with the specific model's architecture, such as the YOLOV7 architecture. Deployment of a model generally involves loading the trained weights into the model architecture and hosting the combination such that it can receive user requests and then return predictions to the user. This means exposing the model to a group of users, primarily over the Internet using the HTTP protocol, either within a private network like a corporation for security, or to the Web for public access.

[0049] An endpoint refers to the location where a model (i.e., weights plus architecture) is deployed such that a user can access the model. The endpoint allows the model to be requested or invoked, typically via a URL, in order to perform real-time inferencing on data using the model. Such an endpoint encompasses how a user is expected to interact, including how to package requests, how the response will be returned, format of the response, and the technology infrastructure used to power the endpoint (i.e., hosting service). There are several ways to deploy models, including, in some embodiments, a one-click deployment option that can deploy the models to Amazon Web Services. This approach is particularly advantageous because the model weights are saved on AWS S3 after training. And because the model architecture is already available, a Docker container can be used to deploy the model as an AWS SageMaker Model Endpoint.

[0050] Accordingly, once the model endpoint, including the weights file, for the YOLOv7 model are created, the user can click on a deploy model button 362 to deploy the weights file to a user-selected hosted environment. The user could also deploy the weights file in a Docker container or similar container that supports YOLOv7 inference requests. In general, any suitable self-contained, portable, stand-alone, runnable software application or service, whether open-source or proprietary, that can containerize the necessary files to run an application, including all code and their dependencies, may be used by the application builder 300 herein to package and deploy the trained object detection model.

[0051] The user has now created or built an object detection application and has deployed the application to the intended hosted environment, all without requiring the user to perform any coding or only a minimal amount of coding. The user may also test the application by clicking on a load image button 364 to load a test image to the application. An example of the object detection application as deployed by the application builder 300 is shown in FIG. 8.

[0052] FIG. 8 depicts an exemplary object detection application 800 that was built and deployed in accordance with embodiments of the object detection application builder described herein. The example shown here is a simplistic example of the object detection application 800 deployed as a Web interface for illustrative purposes. Alternatives and variations to the object detection application 800 are available within the scope of the disclosed embodiments.

[0053] As can be seen, the exemplary object detection application 800 includes an upload image field 802 into which users may drag and drop one or more images to be processed. The object detection application 800 may also include a select image source field 804 that allows users to browse for and select a source of images to be processed. The source of images may be, for example, a repository or folder residing locally or on a network where a plurality of images have been collected and stored for subsequent processing. In some embodiments, the source of images may also be a video stream showing, for example, a manufacturing assembly line, and which may be a real-time stream or a prerecorded stream. In these latter embodiments, the images that are process may be selected frames of the video stream, for example, one frame each second, or one frame per hundred frames, and so forth, depending on the particular application. Clicking on an execute button 806 ("Go!") causes the selected images to be processed through the object detection model (e.g., a YOLOv7 model) associated with and trained for the object detection application 800.

[0054] In some embodiments, the object detection application 800 displays the results of the processing in an objects detected area 808, for example, along the bottom of the screen. In the example shown, the model was able to accurately recognize several objects in the images that were processed, including speed limit signs 810, traffic lights 812, stop signs 814, as well as crosswalks 816.

[0055] FIG. 9 is a flowchart 900 showing an exemplary method of building an object detection application that may be used by or with the object detection application builder 300 herein in some embodiments. The method generally begins at block 902 where a user uploads or otherwise provides a plurality of images to a computing resource for use in building an object detection application. In some embodiments, the images are related to a particular use case, such as roadway-related images, assembly-line related images, security- related images, and the like, as selected by the user. At block 904, the user is provided with a user interface containing tools and functionality that allows the user to easily label the objects to be detected in the plurality of images. For example, the user may be provided with the ability to graphically draw a bounding box around each object to be detected in the images, assign an object class to each object to be detected in the images, and the like. It is also possible in some embodiments to use pre-labeled images, for example, images in which the objects therein were pre-labeled by a third party and subsequently provided to the user as a service.

[0056] Once the objects to be detected have been labeled, whether by the user or otherwise provided, then at block 906, the user may begin training of one or more object detection models using the images. In some embodiments, the training of the object detection models may be machine learning based training. As mentioned above, the one or more object detection models be one or more YOLO models, such as YOLOv7. At block 908, testing may be performed on the object detection model to evaluate the efficacy of the training. The testing may be performed using evaluation images from a subset of the original images that were held back (i.e., hold-out images), or a different set of images may be used as the evaluation images. At block 910, the results of the testing of the one or more object detection models are evaluated and analyzed by the user. In some embodiments, thismay entail displaying to the user one or more performance metrics for the object detection models, such as Classification or Objectness, and may involve the use of a confusion matrix.

[0057] Once the user is satisfied with the performance of the one or more object detection models, then at block 912, the weights and the endpoint for the one or more object detection model is generated along with any files that may be needed to deploy the object detection models, depending on the particular application. For example, the object detection models may be deployed using a Docker container in some embodiments or any other suitable containerization options. Thereafter, at block 914, the one or more object detection models are deployed to a hosted environment or otherwise made available for use by others. Users may then use the deployed object detection models to detect objects within their images, as appropriate for the particular use case.

[0058] Turning now to FIG. 10, an exemplary computing device 1000 is shown that may be used to implement various embodiments of this disclosure. In general, any general- purpose computing device may be used to implement various aspects of this disclosure. For example, various embodiments of the disclosure may be implemented as specialized software, including cloud-based software, executing in a computing device 1000 such as that shown in FIG. 10.

[0059] The computing device 1000 may include a processor 1020 connected to one or more memory devices 1030. Memory 1030 is typically used for storing programs and data during operation of the computing device 1000. The computing device 1000 may also include a storage system 1050 that provides additional storage capacity. Components of the computing device 1000 may be coupled by an interconnection mechanism 1040, which may include one or more busses. The interconnection mechanism 1040 enables communications (e.g., data, instructions) to be exchanged between components of the computing device 1000.

[0060] The computing device 1000 also includes one or more inputs 1010 (e.g., for receiving data, instructions) and one or more outputs 1060 (e.g., for providing data, instructions). In addition, the computing device 1000 may contain one or more interfaces (not shown) that connect the computing device 1000 to a communication network (in addition or as an alternative to the interconnection mechanism 1040).

[0061] The storage system 1050, shown in greater detail in FIG. 11, typically includes a computer readable and writeable nonvolatile recording medium 1110 in which signals are stored that define a program to be executed by the processor 1020 or information stored on or in the medium 1110 to be processed by the program to perform one or more functions associated with embodiments described herein. To this end, the processor 1020 may be any suitable processing unit, such as a microprocessor, microcontroller, ASIC, and the like, and the medium any suitable recording medium, such as a magnetic or solid-state memory. Typically, in operation, the processor 1020 causes data to be read from the nonvolatile recording medium 1110 into storage system memory 1120 that allows for faster access to the information by the processor than does the medium 1110. This storage system memory 1120 is typically a volatile, random access memory such as a dynamic random-access memory (DRAM) or static memory (SRAM). This storage system memory 1120 may be located in storage system 1050, as shown, or in the system memory 1030. The processor 1020 generally manipulates the data within the memory system 1120 and then copies the data to the medium 1110 after processing is completed. A variety of mechanisms are known for managing data movement between the medium 1110 and the integrated circuit memory element 1120, and the disclosure is not limited thereto. The disclosure is not limited to a particular memory 1120, memory 1030 or storage system 1050.

[0062] The computing device 1000 may include specially programmed, special-purpose hardware, for example, an application-specific integrated circuit (ASIC). Aspects of thedisclosure may be implemented in software, hardware or firmware, or any combination thereof. Further, such methods, acts, systems, system elements and components thereof may be implemented as part of the computing device described above or as an independent component.

[0063] Although the computing device 1000 is shown by way of example as one type of computing device upon which various aspects of the disclosure may be practiced, it should be appreciated that aspects of the disclosure are not limited to being implemented on the computing device as shown in FIG. 10. The computing device 1000 may be a general- purpose computing device that is programmable using a high-level programming language. The computing device 1000 may be also implemented using specially programmed, special purpose hardware. In the computing device 1000, processor 1020 is typically a commercially available processor, such as one of several series of microprocessors available from Intel Corporation, and the like. Various aspects of the disclosure may be practiced on one or more devices having a different architecture or components from that shown in FIG. 10. Further, where functions or processes of embodiments of the disclosure are described herein (or in the claims) as being performed on a processor or controller, such description is intended to include systems that use more than one processor or controller to perform the functions.

[0064] In the preceding, reference is made to various embodiments. However, the scope of the present disclosure is not limited to the specific described embodiments. Instead, any combination of the described features and elements, whether related to different embodiments or not, is contemplated to implement and practice contemplated embodiments. Furthermore, although embodiments may achieve advantages over other possible solutions or over the prior art, whether or not a particular advantage is achieved by a given embodiment is not limiting of the scope of the present disclosure. Thus, thepreceding aspects, features, embodiments and advantages are merely illustrative and are not considered elements or limitations of the appended claims except where explicitly recited in a claim(s).

[0065] It will be appreciated that the development of an actual commercial application incorporating aspects of the disclosed embodiments will require many implementationspecific decisions to achieve a commercial embodiment. Such implementation specific decisions may include, and likely are not limited to, compliance with system related, business related, government related and other constraints, which may vary by specific implementation, location and from time to time. While a developer's efforts might be considered complex and time consuming, such efforts would nevertheless be a routine undertaking for those of skill in this art having the benefit of this disclosure.

[0066] It should also be understood that the embodiments disclosed and taught herein are susceptible to numerous and various modifications and alternative forms. Thus, the use of a singular term, such as, but not limited to, "a" and the like, is not intended as limiting of the number of items. Similarly, any relational terms, such as, but not limited to, "top," "bottom," "left," "right," "upper," "lower," "down," "up," "side," and the like, used in the written description are for clarity in specific reference to the drawings and are not intended to limit the scope of the invention.

[0067] This disclosure is not limited in its application to the details of construction and the arrangement of components set forth in the following descriptions or illustrated by the drawings. The disclosure is capable of other embodiments and of being practiced or of being carried out in various ways. Also, the phraseology and terminology used herein is for the purpose of descriptions and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "involving," and variations herein, are meant to be open-ended, i.e., "including but not limited to."

[0068] The various embodiments disclosed herein may be implemented as a system, method or computer program product. Accordingly, aspects may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system." Furthermore, aspects may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code embodied thereon.

[0069] Any combination of one or more computer-readable medium(s) may be utilized. The computer-readable medium may be a non-transitory computer-readable medium. A non-transitory computer-readable medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or system, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the non-transitory computer-readable medium can include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage system, a magnetic storage system, or any suitable combination of the foregoing. Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0070] Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages. Moreover, such computer program code can execute using a single computer system or bymultiple computer systems communicating with one another (e.g., using a local area network (LAN), wide area network (WAN), the Internet, etc.). While various features in the preceding are described with reference to flowchart illustrations and / or block diagrams, a person of ordinary skill in the art will understand that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer logic (e.g., computer program instructions, hardware logic, a combination of the two, etc.). Generally, computer program instructions may be provided to a processor(s) of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus. Moreover, the execution of such computer program instructions using the processor(s) produces a machine that can carry out a function(s) or act(s) specified in the flowchart and / or block diagram block or blocks.

[0071] One or more portions of the computer system may be distributed across one or more computer systems coupled to a communications network. For example, as discussed above, a computer system that performs annotation and other frontend processing of the images may be located remotely from a computer system that performs training of the object detection models and other backend processing of the images. These computer systems also may be general-purpose computer systems. For example, various aspects of the disclosure may be distributed among one or more computer systems configured to provide a service (e.g., servers) to one or more client computers, or to perform an overall task as part of a distributed system. For example, various aspects of the disclosure may be performed on a client-server or multi-tier system that includes components distributed among one or more server systems that perform various functions according to various embodiments of the disclosure. These components may be executable, intermediate (e.g., IL) or interpreted (e.g.,Java) code which communicate over a communication network (e.g., the Internet) using a communication protocol (e.g., TCP / IP).

[0072] Various embodiments of the present disclosure may be programmed using an object-oriented programming language, such as SmallTalk, Java, C++, Ada, or C# (C-Sharp). Other object-oriented programming languages may also be used. Alternatively, functional, scripting, and / or logical programming languages may be used, such as BASIC, Fortran, Cobol, TCL, Lua, Python, Rust or basic C. Various aspects of the disclosure may be implemented in a non-programmed environment (e.g., analytics platforms, or documents created in HTML, XML or other format that, when viewed in a window of a browser program render aspects of a graphical-user interface (GUI) or perform other functions). Various aspects of the disclosure may be implemented as programmed or non-programmed elements, or any combination thereof.

[0073] The flowchart and block diagrams in the Figures illustrate the architecture, functionality and / or operation of possible implementations of various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardwarebased systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0074] Tt should also be understood that the above description is intended to be illustrative, and not restrictive. Many other implementation examples are apparent upon reading and understanding the above description. Although the disclosure describes specific examples, it is recognized that the systems and methods of the disclosure are not limited to the examples described herein but may be practiced with modifications within the scope of the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative sense rather than a restrictive sense. The scope of the disclosure should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

CLAIMSI / We claim:

1. A system for building and deploying an object detection application, the system comprising: a processor; a storage unit communicatively coupled to the processor, the storage unit storing an object detection application builder thereon that, when executed by the processor, causes the system to: perform a process that allows a user to provide a plurality of digital images to a computing resource, the plurality of digital images specified by the user based on a particular use case; perform a process that allows the user to label one or more objects to be detected in the plurality of images using the computing resource; perform a process that uses the plurality of images that have been labeled to perform machine learning based training of an object detection model using the computing resource; perform a process that generates an endpoint for the object detection model using the computing resource; and perform a process that deploys the endpoint for the object detection model to a hosted environment using the computing resource, the hosted environment specified by the user.

2. The system of claim 1, wherein the object detection application builder further causes the system to perform a process that uses a set of evaluation images to perform testing of the object detection model using the computing resource, the set of evaluation images specified by the user based on the particular use case.

3. The system of claim 2, wherein the object detection application builder further causes the system to perform a process that displays one or more performance metrics for the object detection model based on the testing of the object detection model.

4. The system of claim 1, wherein the object detection application builder causes the system to perform a process that allows the user to label one or more objects to be detected in the plurality of images by allowing the user to graphically draw a bounding box around each object to be detected in the plurality of images.

5. The system of claim 1, wherein the object detection application builder causes the system to perform a process that allows the user to label one or more objects to be detected in the plurality of images by allowing the user to assign an object class to each object to be detected in the plurality of images.

6. The system of claim 1, wherein the object detection application builder causes the system to perform a process that allows the user to provide the plurality of digital images to the computing resource by allowing the user to specify a location of the digital images in, or allowing the user to drag and drop the digital images into a designated field.

7. The system of claim 1, wherein the object detection model is a YOLOv7 model.

8. A method for building and deploying an object detection application, the method comprising: providing, at a computing resource, a plurality of digital images, the plurality of digital images provided to the computing resource by a user based on a particular use case; allowing, at the computing resource, the user to label one or more objects to be detected in the plurality of images; training, at the computing resource, an object detection model using the plurality of images that have been labeled, wherein the training is machine learning based training; generating, at the computing resource, an endpoint for the object detection model; and deploying, at the computing resource, the endpoint for the object detection model to a hosted environment, the hosted environment specified by the user.

9. The method of claim 8, further comprising using a set of evaluation images to perform testing of the object detection model using the computing resource, the set of evaluation images specified by the user based on the particular use case.

10. The method of claim 9, further comprising displaying one or more performance metrics for the object detection model based on the testing of the object detection model.

11. The method of claim 8, wherein allowing the user to label one or more objects to be detected in the plurality of images is performed by allowing the user to graphically draw a bounding box around each object to be detected in the plurality of images.

12. The method of claim 8, wherein allowing the user to label one or more objects to be detected in the plurality of images is performed by allowing the user to assign an object class to each object to be detected in the plurality of images.

13. The method of claim 8, wherein providing the plurality of digital images to the computing resource is performed by allowing the user to specify a location of the digital images, or allowing the user drag and drop the digital images into a designated field.

14. The method of claim 8, wherein the object detection model is a YOLOv7 model.

15. A non-transitory computer-readable medium storing computer-readable instructions thereon, the computer-readable instructions comprising: instructions for causing a processor to perform a process that allows a user to provide a plurality of digital images to a computing resource, the plurality of digital images specified by the user based on a particular use case; instructions for causing a processor to perform a process that allows the user to label one or more objects to be detected in the plurality of images using the computing resource; instructions for causing a processor to perform a process that uses the plurality of images that have been labeled to perform machine learning based training of an object detection model using the computing resource; instructions for causing a processor to perform a process that generates an endpoint for the object detection model using the computing resource; and instructions for causing a processor to perform a process that deploys the endpoint for the object detection model to a hosted environment using the computing resource, the hosted environment specified by the user.

16. The computer-readable medium of claim 15, wherein the computer -readable instructions further cause a processor to perform a process that uses a set of evaluation images to perform testing of the object detection model using the computing resource, the set of evaluation images specified by the user based on the particular use case.

17. The computer-readable medium of claim 16, wherein the computer -readable instructions further cause the processor to perform a process that displays one or more performance metrics for the object detection model based on the testing of the object detection model.

18. The computer-readable medium of claim 15, wherein the computer -readable instructions cause the processor to perform a process that allows the user to label one or more objects to be detected in the plurality of images by allowing the user to graphically draw a bounding box around each object to be detected in the plurality of images.

19. The computer-readable medium of claim 15, wherein the computer-readable instructions cause the processor to perform a process that allows the user to label one or more objects to be detected in the plurality of images by allowing the user to assign an object class to each object to be detected in the plurality of images.

20. The computer-readable medium of claim 15, wherein the computer -readable instructions cause the processor to perform a process that allows the user to provide the plurality of digital images to the computing resource by allowing the user to specify a location of the digital images, or allowing the user to drag and drop the digital images into a designated field.

Images