Annotation system for surgical content

Abstract

An annotation system facilitates collection of tags for image, video, or other content items related to a trained machine learning model associated with a surgical or other medical application. The annotation system enables an administrator to configure annotation jobs associated with training of a machine learning model. The job configuration controls presentation of content items to respective participating annotators via an annotation application, and collection of the tags via a user interface of the annotation application. The annotation application enables the participating annotators to provide input in a simple and efficient manner, such as by providing gesture-based input or selecting graphical elements associated with different possible tags.

Description

Background Art Technical Field

[0001] The described embodiments relate to an annotation system for obtaining annotations of surgical content for training a machine learning model.

[0002] Description of Related Technology

[0003] Machine learning models allow predictions to be automatically generated from videos, images, animations (e.g., gifs), and other medical data. These predictions can be used to assist doctors in diagnosing patients, performing surgical procedures, and recommending treatments. In the supervised learning process, a large data set that has been annotated to describe its content or features is used to train a machine learning model. However, obtaining reliable annotations traditionally involves a large investment of time from physicians or researchers with deep expertise in a specific related field. Therefore, it can be challenging to obtain enough training data to meaningfully train or improve these types of machine learning models. Summary of the invention

[0004] In a first embodiment, a method facilitates training of a machine learning model for predicting features of a content item including an image or video associated with a surgical application. An annotation job associated with the machine learning model is configured to specify at least one annotator, a set of predefined selectable tags, and a target judgment quantity. A content item is obtained. The machine learning model is applied to generate a prediction associated with the content item and a confidence metric associated with the prediction. The confidence metric is evaluated to determine whether the confidence metric meets a predefined confidence threshold. In response to the confidence metric failing to meet the confidence threshold, the content item is added to an annotation set for receiving annotations via an annotation application. The annotation application facilitates presenting the content item to the at least one annotator via a user interface of the annotation application. At least one selected tag from a set of predefined selectable tags is obtained from the annotation application. Determine whether the content item meets the target judgment quantity. In response to meeting the target judgment quantity, update parameters of the machine learning model based on the at least one selected tag and the content item.

[0005] In one embodiment, obtaining the at least one selected label includes: identifying a slide gesture performed relative to the user interface; and selecting between a set of predefined labels based on a direction of the slide gesture.

[0006] In one embodiment, obtaining the at least one selected tag includes: facilitating presentation of user interface elements respectively associated with a set of predefined tags via a user interface, and selecting between the set of predefined tags based on selection of one of the user interface elements.

[0007] In one embodiment, configuring the annotation job further includes obtaining a labeling rule indicating a number of selectable tags that can be selected by a single annotator for the content item.

[0008] In one embodiment, obtaining the at least one selected tag includes performing a selection of one and only one tag from a set of predefined tags via a user interface.

[0009] In one embodiment, obtaining the at least one selected tag includes enabling, via a user interface, selection of any number of tags from a set of predefined tags.

[0010] In one embodiment, in response to obtaining the at least one selected tag, the annotation system automatically facilitates presentation of another content item via a user interface of the annotation application of the client device.

[0011] In one embodiment, the target judgment number includes the number of unique annotators that have provided at least one label for the content item.

[0012] In another embodiment, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform the following steps. Present content items associated with an annotation job to an annotator via a user interface of an annotation application. The content item includes a surgical image or video and a set of predefined selectable tags associated with the surgical image or video. Identify a sliding gesture performed by the annotator via the user interface. Determine at least one selected tag from a set of predefined selectable tags based on the sliding gesture. Store the selected tag in association with the content item. Present additional content items associated with the annotation job via the user interface.

[0013] In one embodiment, determining the at least one selected tag includes selecting only a single selected tag to be associated with the content item.

[0014] In one embodiment, determining the at least one selected tag includes selecting a plurality of selected tags to associate with the content item.

[0015] In one embodiment, determining the at least one selected label includes selecting between a first predefined label in response to performing a swipe gesture in a first direction and a second predefined label in response to performing a swipe gesture in a second direction opposite the first direction.

[0016] In one embodiment, determining the at least one selected label includes selecting between a first predefined label in response to a swipe gesture performed in a first direction, a second predefined label in response to a swipe gesture performed in a second direction, a third predefined label in response to a swipe gesture performed in a third direction, and a fourth predefined label in response to a swipe gesture performed in the third direction.

[0017] In one embodiment, the content item includes at least one of: an image, a video, and an animation.

[0018] In one embodiment, the instructions, when executed, further cause the one or more processors to perform steps comprising: presenting a control element to mark the content item for review by an administrator; and in response to selection of the control element, storing the mark in association with the content item.

[0019] In one embodiment, the instructions, when executed, also cause the one or more processors to perform steps including: tracking a status of an annotation job that records the progress of annotations received for a set of content items associated with the annotation job; and configuring the annotation job based on the tracked status in response to the annotation application being closed and reopened.

[0020] In another embodiment, a method facilitates training of a machine learning model for predicting features of a content item including an image or video associated with a surgical application. An annotation job associated with the machine learning model is configured to specify a set of predefined selectable tags for labeling the content item and a labeling rule indicating the number of selectable tags that can be selected for the content item by a single annotator. The content item is obtained. Presentation of the content item is facilitated via a user interface of an annotation application. At least one selected tag from a set of predefined selectable tags is obtained from the annotation application. Parameters of the machine learning model are updated based on the at least one selected tag and the content item.

[0021] In one embodiment, obtaining the at least one selected label includes: identifying a slide gesture performed relative to the user interface; and selecting between a set of predefined labels based on a direction of the slide gesture.

[0022] In one embodiment, obtaining the at least one selected tag includes: facilitating presentation of user interface elements respectively associated with a set of predefined tags via a user interface, and selecting between the set of predefined tags based on selection of one of the user interface elements.

[0023] In one embodiment, configuring the annotation job further includes identifying a set of annotators associated with the job for presenting the content item. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is an example implementation of a computing environment for facilitating the collection of labels associated with training machine learning models for medical applications.

[0025] Figure 2 is an example implementation of a job management engine for managing annotation jobs.

[0026] Figure 3 is an exemplary sequence of user interface screens associated with a noting application.

[0027] Figure 4 is another example of a user interface screen associated with a noting application.

[0028] Figure 5 is yet another example of a user interface screen associated with a notes application.

[0029] Figure 6 is an example of a user interface screen for creating an annotation job.

[0030] Figure 7 is an example of a user interface screen for assigning an annotation job to a group of annotators.

[0031] Figure 8 is an example of a user interface screen for viewing a list of annotation jobs.

[0032] Fig. 9 is an example implementation of a process for facilitating the training of a machine learning model based on labels collected via an annotation system. DETAILED DESCRIPTION

[0033] The drawings and the following description describe certain embodiments by way of example only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles described herein. Reference will now be made to several embodiments, examples of which are shown in the accompanying drawings. Whenever feasible, similar or like reference numerals may be used in the drawings and may indicate similar or like functions.

[0034] The annotation system facilitates the collection of labels for images, videos, animations (e.g., gifs), or other content items associated with training machine learning models associated with surgical applications or other medical applications. The annotation system enables an administrator to configure annotation jobs associated with training of machine learning models. The configuration of the annotation jobs controls the presentation of content items to individual participating annotators via an annotation application, and facilitates the collection of labels via a user interface of the annotation application. The annotation application enables participating annotators to provide input in a simple and efficient manner, such as by providing gesture-based input or selecting graphical elements associated with different possible labels.

[0035] Figure 1 An example embodiment of an annotation system 100 for obtaining annotations of medical images, videos, animations (eg, gifs), or other medical data is shown. The annotation system 100 includes an annotation server 110, one or more client devices 120, one or more management end devices 130, and a network 140.

[0036] The annotation server 110 includes one or more computing devices that interact with the client device 120 and the management end device 130 via the network 140 to perform various functions described herein. The annotation server 110 may include a single physical server, a group of distributed physical servers, a cloud server, one or more virtual machines or containers executed on one or more physical servers, or a combination thereof. The annotation server 110 includes at least a processor and a non-transitory computer-readable storage medium that stores instructions executed by the processor to perform the functions attributed to the annotation server 110 described herein.

[0037] In one embodiment, annotation server 110 includes user interface engine 112, job management engine 114, machine learning (ML) engine 126, content database 116, annotation database 118, and machine learning (ML) model database 124. Alternative embodiments may include different or additional components.

[0038] The job management engine 114 controls a set of jobs associated with obtaining annotations to facilitate the training of machine learning models. In general, each job involves identifying a set of training content items stored in the content database 116, obtaining a set of annotations for the training content items stored in the annotation database 118, and generating or updating a machine learning model (in the ML model database 124) based on the training content items and the associated annotations. Jobs can be created and managed based on job descriptions provided by an administrator via an administrative application 132 (described in further detail below) of the management end device 130, which indicates various parameters associated with the job.

[0039] The job description may identify the content items associated with the job directly (e.g., by associating a unique identifier of the content item with the job) or may identify the content items associated with the job based on a set of configurable characteristics. For example, the job description may identify content items for inclusion in the training set by specifying one or more types of content items (e.g., images, videos, animations (e.g., gifs), illustrations, text, etc.), one or more sources of the content items, a time or location associated with the creation of the content items, one or more characteristics of the content items (e.g., encoding type, quality, resolution, size / length, etc.), one or more tags associated with the content items, or various metadata describing the content items. In this manner, content items that meet specified criteria may be automatically added to the training set associated with a particular job based on the configuration parameters in the job description.

[0040] The job description may also specify labeling rules that include various parameters or constraints associated with obtaining annotations. For example, the job description may specify a set of predefined labels that the annotator chooses among, the number of which may vary according to the labeling rules. For example, some jobs may have labeling rules that perform binary classification, where the annotator is requested to select a label from a set of two predefined labels for each content item. Other jobs may have label labeling rules that specify multi-label classification requests, where the annotator is requested to select a label from a set of three or more predefined labels. Other jobs may be configured to allow multi-label selection, where the annotator can select any number of labels from a set of predefined labels (e.g., between zero labels and all labels) or some other predefined amount or range of labels (e.g., between zero labels and two labels from a set of four labels, select exactly two labels from a set of five labels, etc.). Other jobs may allow free-form text labeling, where the annotator is not necessarily limited to a set of predefined labels.

[0041] The job description may additionally optionally specify a set of control inputs corresponding to different ones of the predefined tags. For example, the job description may assign a specific touch screen gesture to a specific tag (e.g., swipe left, swipe right, swipe up, swipe down, etc.). In other embodiments, the control inputs may be automatically assigned.

[0042] The job description may additionally specify a configured number of judgments representing the number of labels that, once received from different annotators for a given content item, cause the content item to be added to the training set for training the machine learning model. For example, if the number of judgments is set to five, then if the content item is viewed and annotated by less than five annotators, the content item will be marked as unannotated. Once the content item receives five labels, the job management engine 114 can label the content item as annotated and can utilize the content item in the training set. In an example embodiment, the job management engine 114 can enforce a requirement that all received labels reach agreement before adding the content item to the training set. Alternatively, the job management engine 114 can enforce a predefined threshold level of agreement while still allowing a certain degree of inconsistency (e.g., 5 out of 6 labels match). In some embodiments, if the job management engine 114 observes at least a threshold level of inconsistency in the labels received from the annotators, the job management engine 114 can remove the content item (or remove the association with the job) from the content database 116.

[0043] The job description may additionally identify a group of annotators for receiving content items associated with the job for annotation. The group of annotators may be identified directly (e.g., using a user identifier of the annotator) or identified based on a set of specified criteria. For example, the job description may specify a group of annotators based on characteristics such as area of ​​expertise, experience level, location, availability, or other user data specified in a user profile of the annotator. Alternatively, the annotators may be pre-assigned to one or more groups, and the job description may identify the annotators based on the group. The job management engine 114 may then restrict participation in the job to the identified group of annotators.

[0044] The content database 116 stores content that has been annotated or that may be selected for annotation by the annotation system 100. The content may include, for example, medical images, medical videos, animations (e.g., gifs), illustrations, text-based patient data, or other medical data. In some embodiments, the content database 116 may include non-medical images, videos, animations (e.g., gifs), or other data. The content in the content database 116 may include associated metadata that describes the characteristics of the associated content generated at the time of capture, such as the type of content (e.g., image, video, animation, illustration, text, etc.), when the content was created, where the content was created, the entities involved in the creation, the license associated with the content, the file size, the encoding format, or other metadata.

[0045] Annotation database 118 stores annotations associated with content in content database 116. Annotations may include, for example, descriptive tags that describe various characteristics of the content provided by annotators, as described in further detail below.

[0046] The machine learning engine 126 trains a machine learning model based on the content items and annotations identified for the job. The machine learning engine 126 may additionally apply the machine learning model to generate predictions for unannotated content items. The machine learning engine 126 may utilize techniques such as neural networks, classification, regression, or other computer-based learning techniques. Parameters (e.g., weights) associated with the machine learning model are stored in the ML model database 124.

[0047] The UI engine 112 interfaces with the annotation application 122 and the management application 132 to allow the presentation of various information displays, controls, and content items, and processes input received from the annotation application 122 and the management application 132 as described herein. In one embodiment, the UI engine 112 may include a web server for providing the annotation application 122 and / or the management application 132 via a web interface accessible by a browser. Alternatively, the UI engine 112 may include an application server for interfacing with a locally installed version of the annotation application 122 and / or the management application 132. In one embodiment, the UI engine 112 may also enable direct access to the annotation database 118 to enable data scientists to manually review annotations and develop improvements to the machine learning process.

[0048] The client device 120 and the management device 130 each include a computer device for executing an annotation application 122 and an management application 132. The annotation application 122 facilitates the presentation of information and the collection of user input associated with presenting content items and obtaining tags from annotators via the client device 120. The annotation application 122 may also track the progress status of annotators associated with an annotation job so that a user may close the annotation application 122 and return to the same state at a later time. Thus, a user need not complete an annotation job simultaneously, but may annotate several content items at a time, or progress through an annotation job in other convenient ways. Figure 3-Figure 5 An example embodiment of a user interface associated with the annotation application 122 is provided. The management application 132 facilitates the presentation of information and the collection of input from an administrator associated with the creation or update of a job and the viewing of information associated with the job. Figure 6-Figure 8 An example embodiment of a user interface associated with management application 132 is provided.

[0049] The client device 120 and the admin device 130 may each include, for example, a mobile phone, a tablet computer, a laptop or desktop computer, or other computing device. The annotation application 122 and the admin application 132 may be executed locally on the respective client device 120, 130, or may include a web application (e.g., hosted by a remote server) accessed via a web browser. The client device 120 and the admin device 130 may each include conventional computer hardware, such as a display, an input device (e.g., a touch screen), a memory, a processor, and a non-transitory computer-readable storage medium storing instructions executed by the processor in order to perform the functions attributed to the respective device 120, 130 described herein.

[0050] The network 140 includes communication paths for communication between the client device 120, the management device 130, the annotation server 110, and other devices. The network 140 may include one or more local area networks and / or one or more wide area networks (including the Internet). The network 140 may also include one or more direct wired or wireless connections.

[0051] Figure 2 An example implementation of the job management engine 114 and its interaction with the ML engine 126 is shown, the ML engine 126 including the ML prediction engine 232 and the ML training engine 234. In this implementation, the job management engine 114 includes a cycle manager 202, an oracle (database) 204, and a data selector 206.

[0052] The ML prediction engine 232 receives an unannotated content item 212 associated with a particular job (e.g., from the content database 116) and applies the relevant machine learning model 214 associated with the job to determine a prediction metric 220 associated with a prediction of a label for the unannotated content item 212. The prediction metric may include, for example, a confidence level associated with the prediction made by the ML prediction engine 232. Alternatively, the prediction metric may include an entropy-based score or a similarity score indicating similarity between the content item and other content items that have received annotations.

[0053] The data selector 206 selects a set of selected content items 216 from the content database 116 to be annotated in relation to the job based on the prediction metric 220. If the content item 216 is included in the training set, the data selector 206 selects the content item 216 whose prediction can best contribute to improving the performance of the machine learning model 214. In an example embodiment, the selection criteria may depend on the confidence level associated with the prediction. Under this framework, if the ML prediction engine 232 predicts the label of the unannotated content item 212 with a relatively low confidence level (as indicated in the prediction metric 220), this indicates that the current ML model 214 performance for the content item 212 is relatively poor, and obtaining manual annotations for the content item 212 can therefore result in relatively significant improvements in the machine learning model 214 for other content items with similar characteristics. On the contrary, if the ML prediction engine 232 predicts the label of the unannotated content item 212 with a relatively high confidence, this indicates that the performance of the ML model 214 for the content item 212 is already relatively strong, and obtaining manual annotations for the content item 212 may not result in significant improvements. In another embodiment, the data selector 206 may select the content item 212 for annotation based on the entropy metric. Alternatively, the prediction metric 220 may include a similarity score indicating the similarity between the content item 212 and other content items that have been annotated. Here, the content items that are least similar (e.g., having a similarity score below a predefined threshold) are most likely to contribute to improving the performance of the machine learning model and may be selected by the data selector 206. In another embodiment, the data selector 206 may employ a combination of other data mining strategies or techniques.

[0054] In one embodiment, the data selector 206 evaluates the predictive metric 220 for each content item 212 separately and determines to include the content item 212 in the set of selected content items 216 if the predictive metric 220 is below a threshold level. In another embodiment, the data selector 206 evaluates a series of predictive metrics 220 for corresponding content items 212 and then selects a predefined number or percentage of content items 212 in the series for inclusion in the set of selected content items 216 corresponding to the content item 212 having the lowest relative predictive metric 220.

[0055] The oracle 204 interfaces with the annotation application 122 of the client device 120 to obtain the tags 218 of the selected content item 216 selected for annotation. The oracle 204 can control which annotators from the annotator pool have access to the selected content item 216 for annotation and which tags can be assigned to the content item based on the job description. The oracle 204 can also aggregate the results of the annotations received from the client device 120 and filter out any content items that have been marked by the annotator as containing bad data (e.g., content items in irrelevant categories, low-quality content items, etc.). The annotations received by the oracle 204 can be stored in the annotation database 118, as described above.

[0056] The ML training model 234 trains (or updates) the machine learning model 214 associated with a particular job based on the obtained labels 218 and the associated selected content items 216 received from the oracle 204. The ML training engine 234 may operate continuously to perform updates as new obtained labels 218 are received, or may execute in response to a trigger (e.g., the amount of time that has passed since the last update, the receipt of a predefined number of new labels 218, etc.).

[0057] The cycle manager 202 tracks the progress of each annotation job and determines when the job is complete. Here, the cycle manager 202 may consider a job complete when a set of predefined completion criteria associated with the job is met, such as, for example, obtaining labels 218 for at least a predefined number of content items 216, achieving at least a predefined average prediction metric 220 for predictions made by the ML prediction engine 232, detecting when the predictions of the ML prediction engine 232 stop improving, or meeting other predefined metrics associated with the job.

[0058] Figure 33 is an exemplary sequence of user interfaces presented to an annotator via an annotation application 122 for obtaining annotations of surgical media content such as video clips, images, or animations (e.g., gifs). A login screen 302 enables a user to provide credentials (e.g., a username and password) for logging into the annotation server 110. Upon receiving and verifying the login credentials, the annotation server 110 accesses a user profile associated with the user and identifies open annotation jobs for the user. The identified annotation jobs may include annotation jobs specifically assigned to the user (e.g., based on a user identifier), or may include annotation jobs that were matched to the user at the time of login. Here, annotation jobs may be matched to users based on information in the user profile (e.g., area of ​​expertise, experience level, availability to participate, etc.) and metadata associated with the job. A job list screen 304 presents the user with a list of selectable control elements 312 associated with different available jobs. Upon selecting a job (e.g., in this case, selecting "Is the surgeon idle or active"), the annotation application 122 presents the user with a job description screen 306 having a description 314 of the selected job and instructions. Here, for example, the job description screen 306 specifies that the user should "determine whether the surgeon is active or idle during the clip" and "swipe right for active, swipe left for idle." The job description screen 306 also indicates that there are two possible labels to select in this job: "idle" or "active." The user can continue the job by selecting a "start job" control element 316. The annotation application 122 then presents a sequence of annotation screens 308 that include content items 332 (in this case, video clips) for the annotator to view and mark. In this example, the annotator can select between the two labels ("idle" or "active") by selecting corresponding control elements 318-A, 318-B or by performing corresponding gestures. For example, using a touch screen device, the user can swipe left on the touch screen to select "idle" or swipe right on the touch screen to select "active." Alternatively, the user interface may allow the label to be selected through voice input or other types of input.

[0059] The user also has the option of selecting control element 320 "Not Sure" to decline to provide a label for content item 332 and jump to the next label. Annotation screen 308 additionally includes a control element for revoking the current label selection 326 or revoking all label selections 324 provided during the current session. In one embodiment, annotation screen 308 may also provide control element 328 to enable the user to mark content item 332. Marking content item 332 may automatically remove it from the job, or it may be marked for review by the job administrator. Marking content item can be used to indicate, for example, that content item 332 is not relevant to the job, is of low quality, contains occlusions, or may otherwise be unsuitable for training a machine learning model. Annotation screen 308 may also include an annotation count 330 to indicate the cumulative number of content items that the annotator has marked during the current session.

[0060] In one embodiment, the annotation application 122 can keep track of one or more jobs that a user has started, and automatically return to that job if the user exits and later restarts the application 122. This enables annotators to quickly begin annotating, interrupt when needed, and return to where they left off so that they can annotate at a convenient time.

[0061] Figure 4 An example of an annotation screen 408 associated with different jobs is shown. In this example, the job is configured to request a single label from the annotator who selects between four predefined labels 418. The annotator can select a label by selecting an associated control element. Alternatively, swipe gestures can be pre-assigned to different labels to enable selection (e.g., swipe up to the left to select "Scissors", swipe down to the left to select "Grabber", swipe up to the right to select "Stapler", and swipe down to the right to select "Cauter").

[0062] Figure 5 Another example of an annotation screen 508 associated with another job is shown. In this example, the job is configured to enable multiple tags 518 (e.g., between zero and four tags) for a single content item. The annotator can select a tag using an on-screen control element or through a combination of gestures as described above.

[0063] Figure 6 An example embodiment of a user interface screen 600 of the administrator application 132 associated with creating and / or configuring an annotation job is shown. The interface enables the job creator to provide information such as: a name for the job; a description of the job; the type of job (e.g., single label selection or multiple label selection), the number of labels, the names of a set of predefined labels, and the number of judgments.

[0064] Figure 7Another user interface screen 700 of the administrator application 132 is also shown, which enables the administrator to assign jobs to a specific group of annotators. In the illustrated embodiment, the annotators are listed by unique identifiers (e.g., user names). Alternatively, various tags that characterize the annotators (such as area of ​​expertise, availability, experience level, etc.) can be used to identify them so that the administrator can identify a group of annotators that meet the defined characteristics without having to identify them individually. In other embodiments, the annotators can be pre-assigned to one or more annotator groups, and the administrator can assign jobs to one or more groups.

[0065] Figure 8 An example of a user interface screen 800 of an administrator application 132 for viewing and editing a set of jobs is shown. The screen provides a list of jobs 802 (e.g., identified by name and / or unique identifier) ​​and various parameters associated with the jobs 804. The administrator can select a job to view additional information, delete the job, or edit parameters associated with the job.

[0066] Fig. 9 An example implementation of a process for generating or updating a machine learning model based on annotations received via an annotation application 122 is shown. The annotation server 110 configures 902 an annotation job associated with the machine learning model (e.g., based on input from a management end device 130) to specify a set of criteria, including, for example, a pool of one or more annotators, a labeling rule indicating the amount (or amount range) of labels that annotators can select, a set of predefined selectable labels, and a target judgment quantity. The annotation server 110 obtains 904 an initially unannotated content item. The annotation server 110 determines 906 whether to add the content item to the set of annotations in relation to the job. For example, the annotation server 110 may apply the machine learning model to generate a prediction for the content item and a confidence metric associated with the prediction. Then, in response to the confidence metric failing to meet a confidence threshold, the annotation server 110 adds the content item to the set of annotations. The annotation server 110 obtains 910 at least one selected label from a set of predefined selectable labels via the annotation application 122. For example, the annotation server 110 facilitates presenting the content item to at least one of the annotators in the pool via a user interface of the annotation application 122 of the client device 120. The annotation server 110 then determines 912 whether the content item satisfies the target judgment quantity based on a set of cumulative tags for the content item received from the annotator pool. In response to satisfying the target judgment quantity, the annotation server 110 updates 914 parameters of the machine learning model by using the obtained tags to retrain the machine learning model. The annotation server 110 then outputs 916 the machine learning model (e.g., by storing it in the machine learning model database 124).

[0067] The machine learning model generated by the annotation system 100 described above can be used in a variety of environments. For example, the machine learning model can be applied to preoperative, intraoperative, or postoperative surgical images or videos to automatically classify images in a surgical environment. The machine learning model can be similarly applied to other medical images or videos for the purpose of diagnosing, treating, or studying medical conditions. In other alternative embodiments, the annotation system 100 described herein can be used to obtain labels and train machine learning models associated with other types of content items that are not necessarily related to the medical field.

[0068] Embodiments of the annotation system 100 and corresponding processes may be implemented by one or more computing systems. The one or more computing systems include at least one processor and a non-transitory computer-readable storage medium that stores instructions executable by the at least one processor to perform the processes and functions described herein. The computing systems may include distributed network-based computing systems in which the functions described herein are not necessarily performed on a single physical device. For example, some implementations may utilize cloud processing and storage technology, virtual machines, or other technologies.

[0069] The foregoing description of the embodiments has been given for the purpose of illustration; the foregoing description is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed. Those skilled in the relevant art will appreciate that many modifications and variations are possible in light of the above disclosure.

[0070] Some portions of this description describe embodiments in terms of algorithms and symbolic representations of information operations. These operations, when described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent circuits, microcode, etc. In addition, it has proven convenient at times to refer to these arrangements of operations as modules without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0071] Any of the steps, operations or processes described herein can be performed or implemented with one or more hardware or software modules, either alone or in combination with other devices. Embodiments may also relate to devices for performing the operations herein. The device may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device, which is started or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a tangible, non-transient computer-readable storage medium or any type of medium suitable for storing electronic instructions, and is coupled to a computer system bus. In addition, any computing system mentioned in this specification may include a single processor or may be an architecture that uses a multi-processor design to improve computing power.

[0072] Finally, the language used in this specification is selected primarily for readability and instructional purposes, and may not be selected to describe or limit the subject matter of the present invention. Therefore, the scope of the present invention is not limited by this detailed description, but by any claims issued in an application based thereon. Therefore, the disclosure of the embodiments is intended to illustrate but not to limit the scope of the present invention, which is set forth in the appended claims.

Claims

1. A method for facilitating the training of a machine learning model for predicting features of a content item comprising an image or video associated with a surgical application, the method comprising: configuring an annotation job associated with the machine learning model to specify at least one annotator, a set of predefined selectable labels, and a number of target judgments; obtaining the content item; applying the machine learning model to generate a prediction associated with the content item and a confidence measure associated with the prediction; determining whether the confidence metric satisfies a predefined confidence threshold; In response to the confidence metric failing to satisfy the confidence threshold, adding the content item to an annotation set for receiving annotations via an annotation application; facilitating presentation of the content item to the at least one annotator via a user interface of the annotation application; obtaining, from the annotation application, at least one selected tag from the set of predefined selectable tags; determining whether the content item satisfies the target judgment quantity; as well as In response to satisfying the target number of judgments, updating parameters of the machine learning model based on the at least one selected tag and the content item.

2. The method according to claim 1, wherein: Obtaining the at least one selected tag comprises: identifying a swipe gesture performed relative to the user interface; and A selection is made between the set of predefined labels based on a direction of the slide gesture.

3. The method according to claim 1, wherein: Obtaining the at least one selected tag comprises: facilitating presentation, via the user interface, of user interface elements respectively associated with the set of predefined tags; and Based on a selection of one of the user interface elements, a selection is made between the set of predefined labels.

4. The method according to claim 1, wherein: Configuring the annotation job further includes: A labeling rule is obtained that indicates a number of selectable tags that can be selected by a single annotator for the content item.

5. The method according to claim 1, wherein: Obtaining the at least one selected tag comprises: Selection of one and only one tag from the set of predefined tags is performed via the user interface.

6. The method according to claim 1, wherein: Obtaining the at least one selected tag comprises: Selection of any number of tags from the set of predefined tags is achieved via the user interface.

7. The method according to claim 1, wherein: In response to obtaining the at least one selected tag, presentation of another content item is automatically facilitated by the noting application via the user interface of the noting application.

8. The method according to claim 1, wherein: The target judgment number includes the number of unique annotators that have provided at least one label for the content item.

9. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform steps comprising: presenting, via a user interface of the annotation application, to an annotator a content item associated with the annotation job and a set of predefined selectable tags; identifying a swipe gesture performed by the annotator via the user interface; determining at least one selected tab from the set of predefined selectable tabs based on the slide gesture; storing the selected tag in association with the content item; as well as Additional content items associated with the annotation job are presented via the user interface.

10. The non-transitory computer-readable storage medium of claim 9, wherein: Determining the at least one selected tag includes: Select only a single selected tag to be associated with the content item.

11. The non-transitory computer-readable storage medium of claim 9, wherein: Determining the at least one selected tag includes: A plurality of selected tags are selected to be associated with the content item.

12. The non-transitory computer-readable storage medium of claim 9, wherein: Determining the at least one selected tag includes: A selection is made between a first predefined label in response to performing the swipe gesture in a first direction and a second predefined label in response to performing the swipe gesture in a second direction opposite the first direction.

13. The non-transitory computer-readable storage medium of claim 9, wherein: Determining the at least one selected tag includes: Select between a first predefined label in response to performing the swipe gesture in the first direction, a second predefined label in response to performing the swipe gesture in the second direction, a third predefined label in response to performing the swipe gesture in a third direction, and a fourth predefined label in response to performing the swipe gesture in the third direction.

14. The non-transitory computer-readable storage medium of claim 9, wherein: The content item includes at least one of: an image, a video, and an animation.

15. The non-transitory computer-readable storage medium of claim 9, wherein: The instructions, when executed, further cause the one or more processors to perform steps comprising: presenting a control element to mark the content item for review by an administrator; and In response to selection of the control element, the indicia is stored in association with the content item.

16. The non-transitory computer-readable storage medium of claim 9, wherein: The instructions, when executed, further cause the one or more processors to perform steps comprising: tracking a status of the annotation job, the status recording a progress of annotations received for a set of content items associated with the annotation job; as well as In response to the annotation application being closed and reopened, the annotation job is configured based on the tracked status.

17. A method for facilitating training of a machine learning model for predicting features of a content item comprising an image or video associated with a surgical application, the method comprising: configuring an annotation job associated with the machine learning model to specify a set of predefined selectable tags for labeling a content item and a labeling rule indicating a number of selectable tags that can be selected by a single annotator for the content item; obtaining the content item; facilitating presentation of the content item via a user interface of the annotation application; obtaining, from the annotation application, at least one selected tag from the set of predefined selectable tags; as well as Parameters of the machine learning model are updated based on the at least one selected tag and the content item.

18. The method according to claim 17, wherein: Obtaining the at least one selected tag comprises: identifying a swipe gesture performed relative to the user interface; and A selection is made between the set of predefined labels based on a direction of the slide gesture.

19. The method according to claim 17, wherein: Obtaining the at least one selected tag comprises: facilitating presentation, via the user interface, of user interface elements respectively associated with the set of predefined tags; and Based on a selection of one of the user interface elements, a selection is made between the set of predefined labels.

20. The method according to claim 17, wherein: Configuring the annotation job further includes: A set of annotators associated with the job for rendering the content item is identified.

Images