Systems and methods for annotation of images for training a machine learning model

Abstract

There is provided a computer implemented method of training a ranker model using user input provided via a user interface (UI), comprising: accessing a plurality of images varying by at least one visual feature, in a plurality of iterations: presenting on a display, within the UI, at least two images selected from the plurality of images, in response to a ranking of the at least two images on a scale of the at least one visual feature made by a user via the UI, annotating the at least two images with respective annotations, and sequentially and dynamically training the ranker model using the at least two images and respective annotation, and when the plurality of iterations are terminated, providing the ranker model for ranking an input images according to the at least one visual feature.

Description

[0001] SYSTEMS AND METHODS FOR ANNOTATION OF IMAGES FOR TRAINING A MACHINE LEARNING MODEL

[0002] RELATED APPLICATION / S

[0003] This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 312,142 filed on 21 February 2022, the contents of which are incorporated herein by reference in their entirety.

[0004] FIELD AND BACKGROUND OF THE INVENTION

[0005] The present invention, in some embodiments thereof, relates to machine learning models and, more specifically, but not exclusively, to systems and methods for annotation of images for training a machine learning model.

[0006] Standard supervised approaches for training a machine learning model (e.g., neural network) are based on users manually annotating images with one or more classification labels. The machine learning model is then trained on the manually annotated images. The machine learning model outputs a predicted classification label in response to an input of a new image.

[0007] SUMMARY OF THE INVENTION

[0008] According to a first aspect, a computer implemented method of training a ranker model using user input provided via a user interface (UI), comprises: accessing a plurality of images varying by at least one visual feature, in a plurality of iterations: presenting on a display, within the UI, at least two images selected from the plurality of images, in response to a ranking of the at least two images on a scale of the at least one visual feature made by a user via the UI, annotating the at least two images with respective annotations, and sequentially and dynamically training the ranker model using the at least two images and respective annotation, and when the plurality of iterations are terminated, providing the ranker model for ranking an input images according to the at least one visual feature.

[0009] According to a second aspect, a computer implemented method of inference using a ranker model created using user input provided via a user interface (UI), comprises: feeding an input image depicting at least one visual feature into a ranker model, and obtaining as an outcome of the ranker model, a ranking of the input image according to the at least one visual feature, wherein the ranker model is trained by: accessing a plurality of images varying by the at least one visual feature, in a plurality of iterations: presenting on a display, within the UI, at least two images selected from the plurality of images, in response to a ranking of the at least two images on a scale of the at least one visual feature made by a user via the UI, annotating the at least two images with respective annotations, and sequentially and dynamically training the ranker model using the at least two images and respective annotation, and when the plurality of iterations are terminated, providing the ranker model.

[0010] According to a third aspect, a system for training a ranker model using user input provided via a user interface (UI), comprises: at least one hardware processor executing a code for: accessing a plurality of images varying by at least one visual feature, in a plurality of iterations: presenting on a display, within the UI, at least two images selected from the plurality of images, in response to a ranking of the at least two images on a scale of the at least one visual feature made by a user via the UI, annotating the at least two images with respective annotations, and sequentially and dynamically training the ranker model using the at least two images and respective annotation, and when the plurality of iterations are terminated, providing the ranker model for ranking an input images according to the at least one visual feature.

[0011] In a further implementation form of the first, second, and third aspects, further comprising: presenting, within the UI, a visual indication for each one of a set of images, wherein a distance between visual indications denotes similarity between images, and in response to a boundary marked by the user within the GUI delineating a sub-set of the set of images, defining the plurality of images according to the sub-set of the set of images included within the boundary.

[0012] In a further implementation form of the first, second, and third aspects, the visual indication for each respective image of the plurality of images is computed by: feeding the respective image into a pre-processing neural network pre-trained on a training dataset of a plurality of objects each labelled with a ground truth of an object category selected from a plurality of object categories, obtaining an activation map by extracting at least one feature layer of the pre-processing neural network, and performing a dimensionality reduction operation on the activation map for reducing the dimensionality of the activation map to two dimensions, wherein the two dimensions denote x,y coordinates presented on a display, wherein the visual indication denoting the respective image is presented on the display using the two dimensions denoting coordinates.

[0013] In a further implementation form of the first, second, and third aspects, the visual indication comprises a small and low resolution version of the respective image designed for fast loading and displaying.

[0014] In a further implementation form of the first, second, and third aspects, the plurality of iterations includes a first set of iterations comprising: presenting on the display, a pair of images selected from the plurality of images, in response to a selection made by the user via the UI of at least one of: (i) one image of the pair denoting a higher ranking, and (ii) both images of the pair are at a same ranking, annotating the pair of images with an annotation, and sequentially and dynamically training the ranker model using the pair of images and annotation.

[0015] In a further implementation form of the first, second, and third aspects, the pair of images is annotated with a single annotation indication which one of the images of the pair is higher ranked or whether the two images of the pair are similarly ranked.

[0016] In a further implementation form of the first, second, and third aspects, the ranker model is implemented as a Siamese neural network that includes a first branch and a second branch, wherein a first of the two images is fed into the first branch and a second of the at least two images is fed into the second branch, wherein weights are shared between the first and second branches such that the first and second branches have same weights, , wherein a single branch is extracted and provided as the ranker model, wherein the ranker model is designed to differentiate images with small ranked differences in the at least one visual feature.

[0017] In a further implementation form of the first, second, and third aspects, for a final two images, a number generated by the first branch is subtracted from a number generated by the second branch to obtain a result value, and the result value is fed into an activation function.

[0018] In a further implementation form of the first, second, and third aspects, the at least two images of subsequent iterations include at least one image of previous iterations.

[0019] In a further implementation form of the first, second, and third aspects, further comprising during the plurality of iterations: learning, during at least one previous iteration, at least one scale with a certain number of classes, each scale of a specific visual feature, during a current iteration: selecting images satisfying a certain scale learned during at least one previous iteration, wherein a number of the selected images corresponding to the number of classes of the certain scale, presenting within the UI, the number of the selected images arranged in a ranked order according to the certain scale, adapting the ranked order of the images of the certain scale according to user adjustment of locations of the images received via the UI, in response to the adapted ranked order, annotating the images of the adapted ranked order of the certain scale, and sequentially and dynamically training the ranker model on the annotated images of the adapted ranked order of the certain scale.

[0020] In a further implementation form of the first, second, and third aspects, adapting the ranked order comprises iteratively dragging and dropping images at different locations defining the ranked order by the user via the UI, and receiving from the user via the UI an indication that the user completed the adaptions of the ranked order. In a further implementation form of the first, second, and third aspects, further comprising identifying combinatorial pairs of images of the adapted ranked order, wherein annotating comprises annotating each combinatorial pair of images, and wherein training comprises training the ranker model on the annotated combinatorial pairs.

[0021] In a further implementation form of the first, second, and third aspects, the number of selected images corresponding to the number of classes of the certain scale presented within the UI represent an initial number, and further comprising during the current iteration: receiving from the user via the UI, an adaption of the number of classes of the certain scale presented within the UI, and dynamically updating the number of selected images presented within the UI according to the adapted number of classes.

[0022] In a further implementation form of the first, second, and third aspects, further comprising, during the iterations, receiving a selection of an image by a user via the UI, indicating that the selected image is irrelevant to the ranking of the at least one visual feature, and excluding the selected image from the training of the ranker model.

[0023] In a further implementation form of the first, second, and third aspects, further comprising: analyzing the plurality of images using the ranker model for selecting a core-set of the plurality of images representing major differences between images in terms of the at least one visual feature, wherein the at least two images are randomly selected from the core-set.

[0024] In a further implementation form of the first, second, and third aspects, the ranking of the at least two images comprises an indication that the at least two images are ranked similarly with respect to the at least one visual feature.

[0025] In a further implementation form of the first, second, and third aspects, the plurality of images depict objects of a same type that vary by the at least one visual feature.

[0026] In a further implementation form of the first, second, and third aspects, the plurality of images depict a same type of vegetable that vary by at least one visually apparent trait.

[0027] Unless otherwise defined, all technical and / or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention pertains. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of embodiments of the invention, exemplary methods and / or materials are described below. In case of conflict, the patent specification, including definitions, will control. In addition, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting. BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0028] Some embodiments of the invention are herein described, by way of example only, with reference to the accompanying drawings. With specific reference now to the drawings in detail, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of embodiments of the invention. In this regard, the description taken with the drawings makes apparent to those skilled in the art how embodiments of the invention may be practiced.

[0029] In the drawings:

[0030] FIG. 1 is a block diagram of components of a system for training a ranker model using user input provided via a user interface (UI) and / or inference using the trained ranker model, in accordance with some embodiments of the present invention;

[0031] FIG. 2 is a flowchart of a method of training a ranker model using user input provided via a UI, in accordance with some embodiments of the present invention;

[0032] FIG. 3 is a flowchart of a method of a primary phase of training the ranker model, in accordance with some embodiments of the present invention;

[0033] FIG. 4, which is a flowchart of a method of a secondary phase of training the ranker model, in accordance with some embodiments of the present invention;

[0034] FIG. 5 is a flowchart of a method of inference by a ranker model trained using user input provided via a UI, in accordance with some embodiments of the present invention;

[0035] FIG. 6 is a schematic depicting user selection of a group (i.e., subset) of images, in accordance with some embodiments of the present invention;

[0036] FIG. 7 is a schematic of an exemplary UI for training the ranker model during a primary training phase, in accordance with some embodiments of the present invention;

[0037] FIG. 8 is an exemplary UI presenting images ranked according to a scale, in accordance with some embodiments of the present invention;

[0038] FIG. 9 is a schematic depicting screen shots of an exemplary UI presenting adapted number of classes indicating curvature of cucumbers, in accordance with some embodiments of the present invention;

[0039] FIG. 10 is an exemplary UI presenting adaptation of location of images ranked according to a scale, in accordance with some embodiments of the present invention;

[0040] FIG. 11 is an exemplary UI presenting an image that is being excluded, in accordance with some embodiments of the present invention; and

[0041] FIG. 12 is schematic of an exemplary architecture of a ranker model, in accordance with some embodiments of the present invention. DESCRIPTION OF SPECIFIC EMBODIMENTS OF THE INVENTION

[0042] The present invention, in some embodiments thereof, relates to machine learning models and, more specifically, but not exclusively, to systems and methods for annotation of images for training a machine learning model.

[0043] An aspect of some embodiments of the present invention relate to systems, methods, devices, a user interface (e.g., graphical user interface (GUI)) and code instructions (stored on a memory and executable by one or more hardware processors and / or implemented as circuitry) for iteratively training a ranker model based on user rankings provided via a UI (e.g., GUI). Multiple images are accessed. The images, which may depict objects of a same type, vary by one or more visual features. For example, the images depict cucumbers that vary by amount of curvature. In multiple iterations: two or more images are selected and presented on a display within the UI. At least one of the selected images was selected and presented on the display during the preceding iteration. This creates a chain of images throughout multiple iterations, enabling the ranker model to link between subsequent rankings. The user ranks the images according to the visual feature, via the UI, for example, by indicating which image is higher ranked, and / or by dragging and dropping the images to create a ranked and ordered set of images. The images may be ranked on a scale of two or more classes, which are not necessary well defined, for example, 1-10. The ranked images are automatically annotated with respective annotations, for example, indicating relative ranking between the images presented during the current iteration. The ranker model is sequentially, iteratively, and dynamically updated using the current annotated images. The chain-linked images (i.e., including at least one image from the preceding iteration) helps the ranker model learn from and link sequential iterations. When iterations are terminated, the ranker model is provided for ranking one or more input images. Instructions may be generated based on the outcome of the ranker model, which may be automatically executed. For example, cucumbers are automatically sorted by a sorting machine based on ranking of curvature.

[0044] At least some implementations of the systems, methods, devices, UIs, and code instructions described herein address the technical problem of enhancing the user experience of a user training a ranking machine learning model. Training a ranking machine learning model is a specialized technical skill, that is therefore only accessible to a selected number of individuals specifically, such as computer programmers and / or data science experts. Other individuals, which have knowledge in other fields, are unable to quickly train their own ranking machine learning models. In order to perform the training, either help is sought from specialized individuals with this skill, or the individuals must spend considerable effort to learn this new skill. As such, training the ranking machine learning model is not accessible to a wide range of individuals, such as non data science experts.

[0045] At least some implementations of the systems, methods, devices, UIs, and code instructions described herein address the technical problem of training a ranker model for ranking images where visual traits of objects are subtle and / or difficult to categorize. For example, in the vegetable growing field, differences between grown vegetables may be subtle, but important. For example, amount of curvature of a cucumber. Straight cucumbers may be more desired over curved cucumbers, for example, since straight cucumbers may be quickly and easily cut and / or processed by machines, while curved cucumbers may not be able to be processed by such machines. Such traits may be significant in the breeding process, for example, requiring careful selection. Traits may be difficult to classify, and ranking on a scale may be used instead. For example, the amount of curvature may be best represented on a scale of 1-9, such as 1 denotes no curvature, and 9 denotes maximum curvature. A category representation of curvature may be impractical to use.

[0046] At least some implementations of the systems, methods, devices, UIs, and code instructions described herein improve the technical field of training a ranking machine learning model. Using standard approaches, the training data for a ranker model includes a list of results for a query and a relevance rating for each of those results with respect to the query. Data scientists create this training data by examining results and deciding to include or exclude each result from the data set. This is a computer programming exercise that is technically difficult to those outside the specialized field of machine learning.

[0047] At least some implementations of the systems, methods, devices, UIs, and code instructions described herein provide a solution to the above mentioned technical problem, and / or improve the technical field. The solution and / or improvement is in providing a UI and approach that enables users without specific domain knowledge in the field of machine learning to quickly train their own ranking machine learning models by themselves. For example, non data science experts, such as agronomists and / or users in other non-computer fields are able to train their own ranker model by themselves, quickly, without requiring gaining specialized skill. The solution and / or improvement is in providing an approach for training a ranker model for ranking images where visual traits of objects are subtle and / or difficult to categorize. The approach described herein is based on using user feedback to iteratively rank between a small number of images, such as a pair, or a number of images corresponding to a number of rankings of a scale. The user simply and quickly selects the highest ranked image, and / or changes the order of multiple ranked images to be “correct” as the use sees fit. Each currently presented set of images may include one or more images presented in a preceding iteration, which creates a chain-link between the images. The images are automatically annotated, optionally with an indication of relative rank, optionally between each pair of images indicating which is the higher ranked image and / or whether the two images are of the same rank. The ranker model is iteratively updated.

[0048] Before explaining at least one embodiment of the invention in detail, it is to be understood that the invention is not necessarily limited in its application to the details of construction and the arrangement of the components and / or methods set forth in the following description and / or illustrated in the drawings and / or the Examples. The invention is capable of other embodiments or of being practiced or carried out in various ways.

[0049] The present invention may be a system, a method, and / or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

[0050] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0051] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0052] Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.

[0053] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0054] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flowchart and / or block diagram block or blocks.

[0055] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0056] The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

[0057] Reference is now made to FIG. 1, which is a block diagram of components of a system for training a ranker model using user input provided via a user interface (UI) and / or inference using the trained ranker model, in accordance with some embodiments of the present invention. Reference is also made to FIG. 2, which is a flowchart of a method of training a ranker model using user input provided via a UI, in accordance with some embodiments of the present invention. Reference is also made to FIG. 3, which is a flowchart of a method of a primary phase of training the ranker model, in accordance with some embodiments of the present invention. Reference is also made to FIG. 4, which is a flowchart of a method of a secondary phase of training the ranker model, in accordance with some embodiments of the present invention. Reference is also made to FIG. 5, which is a flowchart of a method of inference by a ranker model trained using user input provided via a UI, in accordance with some embodiments of the present invention. Reference is also made to FIG. 6, which is a schematic depicting user selection of a group (i.e., subset) of images, in accordance with some embodiments of the present invention. Reference is also made to FIG. 7, which is a schematic of an exemplary UI 702 for training the ranker model during a primary training phase, in accordance with some embodiments of the present invention. Reference is also made to FIG. 8, which is an exemplary UI 802 presenting images ranked according to a scale, in accordance with some embodiments of the present invention. Reference is also made to FIG. 9, which is a schematic depicting screen shots of an exemplar UI presenting adapted number of classes indicating curvature of cucumbers, in accordance with some embodiments of the present invention. Reference is also made to FIG. 10, which is an exemplary UI 1002A-B presenting adaptation of location of images ranked according to a scale, in accordance with some embodiments of the present invention. Reference is also made to FIG. 11, which is an exemplary UI 1102 presenting an image that is being excluded, in accordance with some embodiments of the present invention.

[0058] System 100 may execute the acts of the method described with reference to FIGs. 2-11, for example, by a hardware processor(s) 102 of a computing device 104 executing code 106 A stored in a memory 106.

[0059] Computing device 104 receives images, which may be captured by imaging sensor(s) 112. The images captured by imaging sensor(s) 112 may be stored in an image repository 114, for example, data storage device 122 of computing device 104, a storage server 118, a data storage device, a computing cloud, virtual memory, and a hard disk. Alternatively or additionally, the images may be synthesized, for example, by a generative adversarial network (GAN). A group (also referred to herein as corset, or sub-set) may be selected from the available images, for example, using user interface code 122B, as described herein. Training dataset 122C is iteratively created by input provided by a user via UI 122B, defining rankings of pairs or Smultiple images which may be selected from the coreset, as described herein. Ranker model 122A is iteratively trained using iteratively updated training dataset 122C, as described herein.

[0060] Computing device 104 may be implemented as, for example, a client terminal, a virtual machine, a server, a virtual server, a computing cloud, a group of connected devices, a mobile device, a desktop computer, a thin client, a kiosk, and a mobile device (e.g., a Smartphone, a Tablet computer, a laptop computer, a wearable computer, glasses computer, and a watch computer).

[0061] Multiple architectures of system 200 based on computing device 204 may be implemented. For example: * Computing device 104 executing stored code instructions 106 A, may be implemented as one or more servers (e.g., network server, web server, a computing cloud, a virtual server) that provides services (e.g., one or more of the acts described with reference to FIG. 1 to one or more client terminals 108 over a network 110. For example, providing software as a service (SaaS) to the client terminal(s) 108, providing software services accessible using a software interface (e.g., application programming interface (API), software development kit (SDK)), providing an application for local download to the client terminal(s) 108, providing an add-on to a web browser running on client terminal(s) 108, and / or providing functions using a remote access session to the client terminals 108, such as through a web browser executed by client terminal 108 accessing a web sited hosted by computing device 108. For example, client terminals access UI code 122B running on computing device 104 via web browsers running on the client terminals, client terminals download UI code 122B for local execution (e.g., an app running on a smartphone), a plug-in that runs and / or accesses UI code 122B is installed on the web browser running on the client terminals, and / or client terminals use an API to access UI code 122B running on computing device 104. In such implementation, users using client terminals 108 select the corset of images from a global dataset of images, and / or create their own personalized ranker model 122A according to their own ranking input for their selected visual features, provided using UI code 122B, as described herein.

[0062] * Computing device 104 may be implemented as a standalone device (e.g., kiosk, client terminal, smartphone, server) that includes locally stored code instructions 106A and / or UI code 122B that implement one or more of the acts described with reference to FIGs. 2-11. The locally stored instructions may be obtained from another server (e.g., 118), for example, by downloading the code over the network, and / or loading the code from a portable storage device. In such implementation, each user uses their own computing device 104 to locally select the corset of images from a global dataset of images, and / or create ranker model 122A according to ranking input provided using UI code 122B, as described herein.

[0063] Exemplary imaging sensor(s) 112 include: RGB (red, green, blue) sensors such as CMOS and / or CCD, x-ray, multispectral, hyperspectral, visible light frequency range, near infrared (NIR) frequency range, infrared (IR) frequency range, thermal sensor, and combinations of the aforementioned. It is noted that images may be synthesized, for example, by a GAN, in which case no imaging sensor is necessarily required.

[0064] Hardware processor(s) 102 may be implemented, for example, as a central processing unit(s) (CPU), a graphics processing unit(s) (GPU), field programmable gate array(s) (FPGA), digital signal processor(s) (DSP), and application specific integrated circuit(s) (ASIC). Processor(s) 102 may include one or more processors (homogenous or heterogeneous), which may be arranged for parallel processing, as clusters and / or as one or more multi core processing units.

[0065] Memory 106 (also referred to herein as a program store, and / or data storage device) stores code instruction for execution by hardware processor(s) 102, for example, a random access memory (RAM), read-only memory (ROM), and / or a storage device, for example, nonvolatile memory, magnetic media, semiconductor memory devices, hard drive, removable storage, and optical media (e.g., DVD, CD-ROM). Memory 106 stores code 106A that implements one or more acts and / or features of the method described with reference to FIGs. 2- 11.

[0066] Computing device 104 may include a data storage device 122 for storing data, for example, ranker model(s) 122A which is iteratively created based on ranking annotations obtained from a user via UI code 122B, UI code 122B which presents a UI that includes a mechanism for a user to select a ranking of two or more images for annotation and iterative update of ranker model 122 A, and training dataset 122C which includes the images (presented to the user via the UI) and ranking annotations used to train ranker model 122 A. Data storage device 122 may be implemented as, for example, a memory, a local hard-drive, a removable storage device, an optical disk, a storage device, and / or as a remote server and / or computing cloud (e.g., accessed over network 110). It is noted that code 122A-C may be stored in data storage device 122, with executing portions loaded into memory 106 for execution by processor(s) 102.

[0067] Computing device 104 may receive images 116 (e.g., captured by imaging sensor 112) using one or more imaging interfaces 120, for example, a wire connection (e.g., physical port), a wireless connection (e.g., antenna), a local bus, a port for connection of a data storage device, a network interface card, other physical interface implementations, and / or virtual interfaces (e.g., software interface, virtual private network (VPN) connection, application programming interface (API), software development kit (SDK)).

[0068] Computing device 104 may include data interface 124, optionally a network interface, for connecting to network 110, for example, one or more of, a network interface card, a wireless interface to connect to a wireless network, a physical interface for connecting to a cable for network connectivity, a virtual interface implemented in software, network communication software providing higher layers of network connectivity, and / or other implementations.

[0069] It is noted that imaging interface 120 and data interface 124 may exist as two independent interfaces (e.g., two network ports), as two virtual interfaces on a common physical interface (e.g., virtual networks on a common network port), and / or integrated into a single interface (e.g., network interface).

[0070] Computing device 104 may communicate using network 110 (or another communication channel, such as through a direct link (e.g., cable, wireless) and / or indirect link (e.g., via an intermediary computing device such as a server, and / or via a storage device) with one or more of

[0071] • Server(s) 118, for example, to obtain images 116, obtain an updated version of code 106A and / or UI code 122B, and / or to provide the trained ranker model 122A for local execution and use by server(s) 118.

[0072] • Client terminal(s) 108, for example, when computing device 104 acts as a server providing services to create customized ranker model(s) 122 A using annotations obtained from user of client terminals 108 via UI code 122B.

[0073] • Image repository 114 that stores images 116 captured by imaging sensor(s) 112.

[0074] • Sorting device 101 and / or controller 101A, for example, for sorting objects according to ranking outcomes generated by ranker model 122 A in response to an input of the objects captured by imaging sensor(s) 112. For example, real time sorting of cucumbers into straightness bins according to a straightness ranking of each cucumber obtained from a ranker model trained on annotated images of cucumbers defining different straightness levels obtained from a user via UI code 122B, as described herein.

[0075] Sorting device 101 may be designed to automatically, manually, and / or semi- automatically sort objects, for example, vegetables. Sorting device 101 may be implemented, for example, as an assembly line of single objects or groups of objects that are sorted into different buckets and / or sorted into a ranked order. In another implementation, sorting device 101 may include a platform for storing objects, and a robotic arm for selecting individual objects.

[0076] Computing device 104 and / or client terminal(s) 108 includes or is in communication with a physical user interface 126 that includes a mechanism designed for a user to interact with UI presented on a display by executing UI code 122B. For example, enabling a user to select which image of a pair of images is higher ranked, and / or for dragging and dropping images into a ranked order, as described herein. Exemplary physical user interfaces 226 include, for example, one or more of, a touchscreen, a display, a keyboard, a mouse, and voice activated software using speakers and microphone.

[0077] Referring now back to FIG. 2, at 202, multiple images are accessed. The images may be obtained, for example, captured by one or more images sensors, such as a camera. The images may be obtained from different sources, such as from different client terminals associated with different image sensors. The images may be obtained from a single source, such as a single image sensor and / or from the same image sensor(s) (e.g., of a sorting line).

[0078] The number of images may be large, for example, over 10000, or over 100000 or over 1000000 or over 10000000 images.

[0079] Images may vary by one or more visual features. Images may depict objects of a same type that vary by one or more visual features. For example, images may depict a same type of vegetable (e.g., carrot, tomato, cucumber) that varies by one or more visually apparent traits, for example, color, length, straightness, defects, and roundness. In another example, images depict different vegetables, where vegetables of a same type vary by one or more visually apparent trait and / or the different vegetables vary by one or more visually apparent traits.

[0080] At 204, a respective visual indication may be computed for each (or most) of the images.

[0081] The visual indication for each respective image may be computed by feeding the respective image into a neural network, for example, ResNet-50, or other implementations. The neural network may be pre-trained on a training dataset of objects each labelled with a ground truth of an object category selected from multiple object categories. An activation map may be obtained by extracting one or more feature layers from the neural network. The activation map may be stored, for example, as a numpy array of dimension 2048, or other values. A dimensionality reduction operation may be performed on the activation map for reducing the dimensionality of the activation map to two dimensions (2D). For example, the 2D coordinates of a 3D reduction process may be computed from the activation map (e.g., from 2048 dimensional activations). Examples of processes for dimensionality reduction include TSN-E UMAP, and PC A. The two dimensions represent x,y coordinates when presented on a display. In an exemplary approach, the activation map is reduced to fewer dimensions (e.g., 2048 to 200), and then to 2 dimensions.

[0082] Optionally, a thumbnail is computed for each image. The thumbnail is a small and / or low resolution version of the respective image designed for fast loading and displaying.

[0083] Optionally, a unique identifier is computed for each image. The unique identifier may be computed, for example, by feeding a representation (e.g., numeric, vector, binary) representation of the image into a hashing function, for example, 128-bit MD5 hash key.

[0084] At 206, a group of the images (i.e., a sub-set of the images) is defined. The group of images may be defined according to the following exemplary approach: the visual indications (computed for the images as described with reference to 204) are presented within a UI (e.g., the GUI). The thumbnail version of each image may be presented. The visual indications (e.g., thumbnail versions thereof) are presented on the display, within the UI, using coordinates defined by the 2D representation. A distance between visual indications denotes similarity between images. The distance may be a physical distance, for example, in terms of pixels and / or millimeters. The closer the physical distance, the more similar the two images are to each other. The larger the physical image, the less similar the two images are to each other.

[0085] A boundary marked by the user within the GUI is obtained. The boundary delineates the defined group (i.e., sub-set) of the images, which are located within the boundary. The boundary may be, for example, a rectangle shape that may be resized, and / or a free-style lasso type boundary when the user draws the boundary and / or sets points of the boundary which are automatically connected. It is noted that the user is free to select the boundary, for example, to cover the entire or majority of images that are similar such as of the same type (e.g., onions), or a sub-set thereof, such as a subset of the group (e.g., onions that are of a similar size)

[0086] Since the number of images may be very large, it is not feasible for a user to simply browser through the images and select some images. The approach described herein visually and simultaneously displays the large number of images on the display, enabling the user to simultaneously evaluate the large number of images, by visually distinguishing differences between the images and / or visually presenting similarities between the images. The user delineated boundary enables selection of a part of this data, to include images that are similar to each other in at least one property. The user may delineate the boundary for selection of a portion of the images from the entire set of available images.

[0087] Referring now back to FIG. 6, UI 602 displays multiple visual indications of multiple images of different vegetables. The visual indications are thumbnails. It is visually apparent that vegetables of a same type are grouped together. Within each group of a certain type of vegetable, it is visually apparent that vegetables which have similar traits are closer together, and vegetables with less similar traits are further away. For example, tomatoes that are rounder and / or more red are closer together than tomatoes that are less round and / or less red. A boundary 604 is selected by a user. Boundary 604 may indicate one group of similar vegetables, for example, of a same type, such as onions.

[0088] Referring now back to FIG. 2, at 208, a ranker model is trained. The ranker model may be trained using a primary training stage and / or a secondary training stage. The secondary training stage may follow the primary training stage. An exemplary primary training stage is described with reference to FIG. 3. Referring now back to FIG. 3, at 302, two or more images are selected. The images may be selected from the group of images selected by the user via the UI, as described with reference to 206 of FIG. 2. When no group of images is selected, the image are selected from a defined group, for example, images selected according to tags, such as all images labelled as “vegetables” or labelled as specific vegetables such as “carrot”.

[0089] In an exemplary implementation, two images (i.e., a pair) are selected. It is noted that other number of images may be selected. In some implementations, three or more images are selected during the secondary training state.

[0090] The two images may be selected at random from the group of images. The random selection may be according to one or more parameters, for example, a minimum, range, and / or maximum statistical distance between the images indicating similarity (e.g., distance between the x,y coordinates of the images). For example, to select images that are somewhat similar to one another, images that are not so similar to one another, or perform a completely random selection without regards to similarity.

[0091] The image(s) (e.g., pair) may be selected to represent the distribution of the group of images from which they are selected. The images may be selected to represent the majority of the difference in data. A sub-group representing the distribution of the full group of images may be selected, with the image pair (or other number of images) selected randomly from the subgroup. Selection may be done, for example, by computing the PCA (or other approach) of the feature layer of each image in the group. The image(s) (e.g., pair) are selected to have a distribution that is similar to the entire population of images of the group. For example, the group of images is analyzed using the currently trained version of the ranker model. A core-set of the group of images representing major differences between images in terms of the visual feature is selected. The two (or more) images are randomly selected from the core-set.

[0092] During a subsequent iteration (e.g., as described with reference to 314), at least one of the previously presented images is selected for presentation during a next (e.g., current) iteration. In the case of a pair of images, one image of the pair selected and presented during a previous iteration, is selected and presented during the next (e.g., the current) iteration. This creates a chain-like connection of images.

[0093] At 304, the selected images are presented on a display, within the UI (e.g., within the GUI).

[0094] Optionally, instructions are generated and provided to the user, indicating what actions the user is to perform via the UI. The instructions may be, for example, presented as text in the UI, a video and / or animation playing in the UI, and / or an audio message played over speakers. The instructions describe to the user how to make a selection of an image, as described below.

[0095] At 306, a selection made by a user via the UI is obtained.

[0096] The selection is with respect to one or more visual features of the images, which may be on a scale. The scale may not yet be fully defined, being learned during the iterations, as described herein. For example, when the images are vegetables, the selection may be according to one or more of the following visual features: straightness, roundness, color tone, uniform color tone, smoothness of skin, defects, and the like.

[0097] Selection may be made, for example, by clicking on one of the images (e.g. indicating which image is higher ranked), clicking on an icon (e.g., indicating that both images have a similar ranking), pressing a number, and a voice command.

[0098] The selection indicates one of the following:

[0099] • That the selected image has a higher ranking than the other non-selected image(s) with respect to the visual feature(s). For example, the user clicks (e.g., using a mouse) and / or touches (e.g., using a touch screen) the higher ranked image. In some embodiments, that one image of the pair has a higher ranking than the other image of the pair. For example, that one cucumber depicted in one image is straighter than another cucumber depicted in the other image.

[0100] • That the selected image has a lower ranking than the other non-selected image(s) with respect to the visual feature(s). For example, the user clicks and / or touches the lower ranked image. In some embodiments, that one image of the pair has a lower ranking than the other image of the pair. For example, that one cucumber depicted in one image has a larger curvature than another cucumber depicted in the other image.

[0101] • That both (e.g., all) images have a similar ranking, without one image being ranked higher (or lower) than the others with respect to the visual feature(s). The user may click on an icon indicating “both equal”. For example, that both cucumbers depicted in the pair of images have similar curvatures.

[0102] At 308, a selection made by a user via the UI to exclude one or more images is obtained. The selection indicates that the selected image is irrelevant to the ranking of the visual feature. For example, when a tomato, which is round, appears as an image along with a straight vegetable, such as a carrot or cucumber, for ranking of straightness. In another example, when an onion appears as an image for training a ranker model of tomatoes.

[0103] The selected image(s), which are irrelevant to the ranking task, are excluded from training of the ranker model. Referring now back to FIG. 7, an indication 704 of the visual feature being ranked may be provided, for example, higher rank = less curved cucumber. The indication 704 may denote instructions provided to the user for how to select the images for training the ranker model, as described herein. UI 702 may include icons 706-710 for selecting the image, which is annotated accordingly and used to train the ranker model. Icon 706 indicates that the image of the cucumber on the left is higher ranked. Icon 708 indicates that the image of the cucumber on the right is higher ranked. Icon 710 indicates that both images of both cucumbers are equally ranked. Icon 712 indicates that that user is unable to decide. There may be an icon to exclude irrelevant images, which cannot be ranked.

[0104] Referring now back to FIG. 3, at 310, the pair (or multiple) presented images which were ranked are annotated as a set, optionally with a single annotation. The single annotation indicates which image of the pair is higher ranked or whether the two images of the pair are similarly ranked. For example, an annotation of 0 denotes that the first image is ranked higher than the second image (e.g., left image ranked higher than the right image), 1 indicates that the second image is ranked higher than the first image (e.g., right image ranked higher than the left image), and 0.5 indicates that both images are ranked equally.

[0105] Alternatively or additionally, each image is annotated according to the selection, for example, the higher ranked image is annotated with 1, and the lower ranked image is annotated with 0. When both images are equally ranked, both images are annotated with 0.5.

[0106] At 312, the ranker model is trained using the images that were presented and annotated during the current iteration. The ranker model is sequentially, dynamically, and iteratively updated, using newly presented and newly annotated images during each round. Alternatively, multiple rounds of annotation of images may occur without training the ranker model. The ranker model may then be batch trained using the annotated images obtained from the previous multiple rounds.

[0107] The ranker model may be implemented, for example, as a Siamese neural network. The Siamese neural network may include with a minus sign between two activations of the two input images. The selected architecture may be designed to enable the trained ranker model to differentiate images with small ranked differences in terms of the visual feature(s). Additional exemplary details of the Siamese neural network, and / or training and / or inference thereof, are described, for example, with reference to FIG. 12.

[0108] At 314, features described with reference to 302-312 are iterated, for iterative training of the ranker model. The iterations may be terminated, for example, when a preselected number of iterations have been performed, and / or when a performance parameter of the ranker model is reached (e.g., threshold of accuracy of ranking).

[0109] The preselected number of iterations may be obtained, for example, based on a trial and error approach, and / or based on the visually apparent differences between features. For example, subtle visual traits may require a larger number of image pairs to train the ranker model. During experiments, Inventors discovered that about 200 iterations of pairs of images may be selected for a general ranking task, with the number being adjusted higher or lower according to the specific ranking task at hand. It is noted that additional training iterations may be performed in the second phase, as described herein.

[0110] One or more scales of visual features may be learned during the iterations according to selections of the images (e.g., pairs). For example, after each training around, one or more scales may be stored, where each scale has a different number of classes.

[0111] Classes are not necessarily well defined, and may be a numerical value on the scale. For example, the scale may be, for example, 1-9, where each number is a class, for example, 5 is a class.

[0112] Scales may different between number of classes, for example, so learned scales have 2 classes of the visual feature, while other scales have 3, 5, 7, 10, or more number of classes of the visual feature. Scales may different between visual features, for example, one scale is for one visual feature (e.g., straightness), another scale for another visual feature (e.g., color), and yet another scale for a combination of two or more visual features (e.g., straightness and color). Storing the scale enables the UI to quickly present images according to the scale, as described with reference to 402 of FIG. 4.

[0113] The scale is used during the secondary training stage, as described herein.

[0114] During iterations, metadata may be generated, for example, name, and users.

[0115] At 210, the ranker model may be trained using the secondary training stage. The secondary training state may be initiated when the iterations of the primary training phase have terminated. An exemplary primary training stage is described with reference to FIG. 4.

[0116] Referring now back to FIG. 4, at 402, multiple images satisfying a certain scale of the visual features are selected. The scale(s) were learned during the primary training stage, as descried with reference to 314 of FIG. 3.

[0117] The number of selected images corresponds to the number of classes of the certain scale.

[0118] At 404, within the UI, the images selected according to the certain scale are presented. The presented image are initially arranged in a ranked order according to the certain scale. The initial ranked order is according to the previously learned rankings of the images based on the scale as learned in the primary stage.

[0119] Referring now back to FIG. 8, one or more scales may be presented in the UI. For example, as shown, two scales 804A-B are shown, presented as rows. Icon 806 indicates the number of classes in the scale, for example, 5 as shown. The number of classes may be adapted, as described with reference to 406 of FIG. 4. When the user determines that the images are correctly arranged in the ranked order of the scale, the user may move to the next scale by pressing icon 808. Icon 810 may be pressed to return to the primary training stage, for example, when the user determines that there are too many errors in the automated ranking of images.

[0120] At 406, an adaptation of the number of classes (also referred to herein as “bins”) of the certain scale is obtained. The adaptation may be performed by the use via the UI, for example, pressing up / down arrows presented in the UI, clicking a icon presented in the UI indicating the number of classes, and / or by manually typing the number of classes into a field presented on the UI.

[0121] The user may increase and / or decrease the number of classes of the scale.

[0122] At 408, features described with reference to 402-406 are iterated, for dynamically updating the number of selected images presented within the UI according to the adapted number of classes.

[0123] After each training around, one or more scales may be stored, where each scale has a different number of classes. Storing the scale enables the UI to quickly present images according to the scale, as described with reference to 402. This enables the UI to provide the user with a user experience of toggling seamlessly between the number of bins to determine what number of relevant for the scale the user envisions.

[0124] The number of classes may be increased, for example, until there is no significant, no apparent, and / or practical difference between two neighboring classes. For example, slight curvatures of cucumbers may not matter (e.g., in 10 classes), whereas no curvature (i.e., straight), mild curvature, or extreme curvature may be significant (e.g., in 3-5 classes).

[0125] Referring now back to FIG. 9, screen capture 902 depicts three classes. Screen capture 904 depicts six classes. Screen capture 906 depicts eleven classes. Screen capture 908 depicts twenty classes. Higher number of classes increase the granularity of the visual feature. It is noted that higher number of classes may be used, for example, to differentiate between combinations of visual features, for example, cucumber length and curvature.

[0126] At 410, an adjustment of the presented ranked order of the images of the certain scale at the selected number of classes, is obtained. The adjustment is performed by a user via the user interface. The adjustment may be performed by the user changing locations of the presented images.

[0127] The adjustment of the presented ranked order fine-tunes the scale.

[0128] Optionally, the adaption of the ranked order comprises iteratively dragging and dropping images at different locations by the user via the UI. The dragging and dropping defines the ranked order. Multiple dragging and dropping operations may be done, using the same and / or different images, for example, as until the scale appears well ordered to the user. The user may indicate via the UI that the user completed the adaptions of the ranked order, for example, by pressing an icon, such as “Scale OK, next examples”.

[0129] An indication of exclusion of one or more images from the scale may be obtained, for example, as described with reference to 308 of FIG. 3.

[0130] Referring now back to FIG. 10, initially as shown in UI 1002 A, the order of the images ranked according to the scale is as shown in UI 802 of FIG. 8. Arrow 1004 indicates a dragging and dropping of a certain image in order to change the ranked order of the scale. UI 1002B shows the new order of the images after implementation of the drag and drop represented by arrow 1004 shown in UI 1002 A.

[0131] Referring now back to FIG. 4, at 412, one or more image may be excluded, for example, images that do not belong anywhere in the scale and / or do not belong to any classes. The excluded images may be excluded on the basis of not contributing to an understanding of the scale. For example, images of defects and / or errors. In another example, the image being excluded does not fit the distribution of the current images. The excluded images are removed from the scale, i.e., from the presentation of images arranged according to the order of the scale. The excluded images is further excluded from training of the ranker model. Features described with reference to 308 of FIG. 3 may be combined with, and / or refer to, and / or be replaced with, features described with reference to 412 of FIG. 4.

[0132] Excluded images (e.g., as described with reference to 308 of FIG. 3 and / or 412 of FIG. 4) may be added to a list of images that are excluded during training, for example, implemented as a JSON list.

[0133] Referring now back to FIG. 11, the image is being excluded by being dragged and dropped, as represented by arrow 1104, into an icon of a trash can 1106. The image dragged and dropped into trash can 1106 is excluded from the scale, and no longer presented. The excluded image is excluded from training of the ranker model. Referring now back to FIG 4, at 414, in response to an indication that the presented order of images is ranked correctly (e.g., after adaptation of the ranked order), the images are annotated according to the adapted ranked order of the certain scale.

[0134] Combinatorial pairs of images of the adapted ranked order may be identified. For example, for the simple case of images A, B, and C, the pairs are: A-B, A-C, and B-C. The annotation may be done by annotation of each combinatorial pair of images, as described with reference to 310 of FIG. 3.

[0135] At 416, the ranker model is trained on the annotated images based on the adapted ranked order of the certain scale. The ranker model may be trained, for example, as described with reference to 312 of FIG. 3.

[0136] At 418, features described with reference to 402-416 are iterated, for iterative training of the ranker model.

[0137] The iterations may be terminated, for example, when a preselected number of iterations have been performed, and / or when a performance parameter of the ranker model is reached (e.g., threshold of accuracy of ranking).

[0138] Referring now back to FIG. 2, at 212, the trained ranker model is provided, for example, stored on a data storage device, forwarded to a remote computing device, and / or provided for execution in association with another process. The trained ranker model is provided for inference, for ranking input image(s) according to the visual feature(s).

[0139] Referring now back to FIG. 5, at 502, the trained ranking mode is accessed. The accessed trained ranker model may be selected from multiple different trained ranker models, for example, varying by object type and / or visual features being considered in the ranking.

[0140] The ranker model is trained as described with reference to FIGs. 2-4. Briefly, the ranker model is trained by accessing images varying by visual feature(s). In multiple iterations the following is performed. Presenting on a display, within a UI, at least two selected images. In response to a ranking of the images on a scale of the visual feature(s), made by a user via the UI, annotating the images with respective annotations. Sequentially and dynamically training the ranker model using the images and respective annotation. When the iterations are terminated, providing the ranker model.

[0141] At 504, one or more input images depicting visual feature(s) are fed into the ranker model. The input image(s) may be obtained by an imaging sensor. The input images depict objects and / or visual features which are considered for ranking the input image, corresponding to the selected ranker model. For example, input images depicting cucumbers and a certain amount of curvature are fed into a ranker model trained to rank cucumber images according to curvature (or lack thereof).

[0142] At 506, a ranking of the input image is obtained as an outcome of the ranker model. The ranking is according to the depicted visual feature. The ranking may be an absolute value indicating the ranking, for example, a certain class within a scale. Alternatively or additionally, the ranking may be a relative value indicting relative ranking between input images. For example, the outcome of multiple images fed into the ranker model is a ranking of the fed images, relative to one another.

[0143] At 508, instructions may be generated based on the outcome of the ranker model. The instructions may be, for example, code for execution by a processor, signals for execution by a controller and / or machine, and instruction to be followed by a human (e.g., text, video, animation, image, and audio).

[0144] The instructions may be, for example, for execution by a controller of a sorting machine, to sort the objects depicted by the input images, for example, to select the highest ranked objects and / or sort the objects into different bins based on class. For example, sort cucumbers according to curvature classes.

[0145] At 510, the instructions may be implemented, for example, automatically executed by the processor and / or controller, and / or manually implemented by a human. For example, the automatic controller sorts and / or selects the objects depicted in the input images according to the generated instructions.

[0146] Reference is now made to FIG. 12, which is schematic of an exemplary architecture of a ranker model 1202, in accordance with some embodiments of the present invention. Dataflow 1204 depicts an exemplary training process for training ranker model 1202, for example, as described herein. Dataflow 1206 depicts an exemplary process of inference using ranker model 1202, for example, as described herein.

[0147] Ranker model 1202 is based on a deep neural network architecture, which may be based, for example, on an adaptation of a RankNet model (e.g., as described with reference to From RankNet to LambdaRank to LambdaMART: An Overview, Chris J.C. Burges 2010). Ranker model 1202 is designed for sorting objects depicted in images on a scale, as described herein. After training, ranker model 1202 transforms an image within the training domain (e.g., roundness of fruits) into a rank according to the place of the image on the training scale, i.e., higher number, higher rank.

[0148] Referring now to dataflow 1204, images 1210 are fed into a pre-processing ML model 1212, optionally a neural network pre-trained on a training dataset of labelled images, for example, ResNet50 version 2 (e.g., as described with reference to Deep Residual Learning for Image Recognition, Kaiming He et al 2015), which was pre-trained on the ImageNet version 1 dataset, (e.g., obtained from TensorFlow 2 hub free repository). The result of a final layer 1214 of the pre-processing ML model, for example, 2048 float32 vectors, call activation is stored, and used in training and / or predictions.

[0149] The input data for the training is a set of image-pairs 1210, and a ground truth label, for example, a number between 0 and 1 that states which image is ranked ‘higher’ by an expert human, as described herein. The core-set created as described herein is designed to maximize the information gained during training. As described in additional detail herein, PCA is applied on the activation, reducing the dimensions from 2048 to 50, a K-means clustering process, and uniformly sampling the clustering result. Inventors discovered that the described approach was experimentally found to enhance the training speed and accuracy. The result of the core set method is a sample of image pairs for training. A chain of pairs is created, where one of the elements of each pair, is also a member of the next pair, as described herein in additional detail.

[0150] A Siamese branch 1216, is designed to pass each of the two images through a neural network (e.g., 3 layers is an exemplary implementation, and other number of layers may be used and / or other architectures of the Siamese branch 1216 may be used), while sharing the weights between a left branch 1216A into which one image is fed and a right branch 1216B into which a second image is fed. Weights are shared between the two branches 1216A-B, such that the weights are the same in both branches 1216A-B, hence the term a Siamese network 1216. It is noted that technically, there is only a single branch that is being trained, since the weights of the two branches 1216A-B are the same. During training, at 1218, each pair of images is passed through Siamese network 1216, and on the last training step using the final pair of images, the number generated by the right branch 1216B is subtracted from the number generated by the left branch 1216A (or vice versa where the number generated by the left branch is subtracted from the number generated by the right branch). The result of the subtraction 1218 is fed into a sigmoid activation 1220 (i.e., it is noted that the sigmoid is an exemplary implementation, and other activations may be used). To calculate the error, the difference between the result of the subtraction and the given result is compared, and the error is back-propagated through Siamese model 1216 for adjusting the weights thereof.

[0151] After training, a Siamese branch 1250 (e.g., the thin network) is extracted from Siamese model 1216 and used for ranking input images 1230. Input image 1230 is fed into the preprocessing ML model 1230, for example, from ResNet50 (i.e., corresponding and / or the same as pre-processing ML model 1212 described with reference to dataflow 1204). The 2048-sized vector 1234, which is obtained from pre-processing ML model 1232, is fed into trained Siamese branch 1250 and produces a number 1236 as an outcome (e.g., or other result). Outcome 1236 is used to sort images on an arbitrary scale, that they were trained on, as described herein.

[0152] The descriptions of the various embodiments of the present invention have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

[0153] It is expected that during the life of a patent maturing from this application many relevant ranker models will be developed and the scope of the term ranker model is intended to include all such new technologies a priori.

[0154] As used herein the term “about” refers to ± 10 %.

[0155] The terms "comprises", "comprising", "includes", "including", “having” and their conjugates mean "including but not limited to". This term encompasses the terms "consisting of' and "consisting essentially of .

[0156] The phrase "consisting essentially of' means that the composition or method may include additional ingredients and / or steps, but only if the additional ingredients and / or steps do not materially alter the basic and novel characteristics of the claimed composition or method.

[0157] As used herein, the singular form "a", "an" and "the" include plural references unless the context clearly dictates otherwise. For example, the term "a compound" or "at least one compound" may include a plurality of compounds, including mixtures thereof.

[0158] The word “exemplary” is used herein to mean “serving as an example, instance or illustration”. Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments and / or to exclude the incorporation of features from other embodiments.

[0159] The word “optionally” is used herein to mean “is provided in some embodiments and not provided in other embodiments”. Any particular embodiment of the invention may include a plurality of “optional” features unless such features conflict.

[0160] Throughout this application, various embodiments of this invention may be presented in a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the invention. Accordingly, the description of a range should be considered to have specifically disclosed all the possible subranges as well as individual numerical values within that range. For example, description of a range such as from 1 to 6 should be considered to have specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numbers within that range, for example, 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.

[0161] Whenever a numerical range is indicated herein, it is meant to include any cited numeral (fractional or integral) within the indicated range. The phrases “ranging / ranges between” a first indicate number and a second indicate number and “ranging / ranges from” a first indicate number “to” a second indicate number are used herein interchangeably and are meant to include the first and second indicated numbers and all the fractional and integral numerals therebetween.

[0162] It is appreciated that certain features of the invention, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the invention, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable subcombination or as suitable in any other described embodiment of the invention. Certain features described in the context of various embodiments are not to be considered essential features of those embodiments, unless the embodiment is inoperative without those elements.

[0163] Although the invention has been described in conjunction with specific embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications and variations that fall within the spirit and broad scope of the appended claims.

[0164] It is the intent of the applicant(s) that all publications, patents and patent applications referred to in this specification are to be incorporated in their entirety by reference into the specification, as if each individual publication, patent or patent application was specifically and individually noted when referenced that it is to be incorporated herein by reference. In addition, citation or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to the present invention. To the extent that section headings are used, they should not be construed as necessarily limiting. In addition, any priority document(s) of this application is / are hereby incorporated herein by reference in its / their entirety.

Claims

WHAT IS CLAIMED IS:

1. A computer implemented method of training a ranker model using user input provided via a user interface (UI), comprising: accessing a plurality of images varying by at least one visual feature; in a plurality of iterations: presenting on a display, within the UI, at least two images selected from the plurality of images, in response to a ranking of the at least two images on a scale of the at least one visual feature made by a user via the UI, annotating the at least two images with respective annotations, and sequentially and dynamically training the ranker model using the at least two images and respective annotation; and when the plurality of iterations are terminated, providing the ranker model for ranking an input images according to the at least one visual feature.

2. The computer implemented method of claim 1, further comprising: presenting, within the UI, a visual indication for each one of a set of images, wherein a distance between visual indications denotes similarity between images; and in response to a boundary marked by the user within the GUI delineating a sub-set of the set of images, defining the plurality of images according to the sub-set of the set of images included within the boundary.

3. The computer implemented method of claim 2, wherein the visual indication for each respective image of the plurality of images is computed by: feeding the respective image into a pre-processing neural network pre-trained on a training dataset of a plurality of objects each labelled with a ground truth of an object category selected from a plurality of object categories; obtaining an activation map by extracting at least one feature layer of the pre-processing neural network; and performing a dimensionality reduction operation on the activation map for reducing the dimensionality of the activation map to two dimensions, wherein the two dimensions denote x,y coordinates presented on a display,wherein the visual indication denoting the respective image is presented on the display using the two dimensions denoting coordinates.

4. The computer implemented method of claim 2, wherein the visual indication comprises a small and low resolution version of the respective image designed for fast loading and displaying.

5. The computer implemented method of claim 1, wherein the plurality of iterations includes a first set of iterations comprising: presenting on the display, a pair of images selected from the plurality of images, in response to a selection made by the user via the UI of at least one of: (i) one image of the pair denoting a higher ranking, and (ii) both images of the pair are at a same ranking, annotating the pair of images with an annotation, and sequentially and dynamically training the ranker model using the pair of images and annotation.

6. The computer implemented method of claim 5, wherein the pair of images is annotated with a single annotation indication which one of the images of the pair is higher ranked or whether the two images of the pair are similarly ranked.

7. The computer implemented method of claim 1, wherein the ranker model is implemented as a Siamese neural network that includes a first branch and a second branch, wherein a first of the two images is fed into the first branch and a second of the at least two images is fed into the second branch, wherein weights are shared between the first and second branches such that the first and second branches have same weights, , wherein a single branch is extracted and provided as the ranker model, wherein the ranker model is designed to differentiate images with small ranked differences in the at least one visual feature.

8. The computer implemented method of claim 7, wherein for a final two images, a number generated by the first branch is subtracted from a number generated by the second branch to obtain a result value, and the result value is fed into an activation function.

9. The computer implemented method of claim 1, wherein the at least two images of subsequent iterations include at least one image of previous iterations.

10. The computer implemented method of claim 1, further comprising during the plurality of iterations: learning, during at least one previous iteration, at least one scale with a certain number of classes, each scale of a specific visual feature; during a current iteration: selecting images satisfying a certain scale learned during at least one previous iteration, wherein a number of the selected images corresponding to the number of classes of the certain scale, presenting within the UI, the number of the selected images arranged in a ranked order according to the certain scale, adapting the ranked order of the images of the certain scale according to user adjustment of locations of the images received via the UI, in response to the adapted ranked order, annotating the images of the adapted ranked order of the certain scale, and sequentially and dynamically training the ranker model on the annotated images of the adapted ranked order of the certain scale.

11. The computer implemented method of claim 10, wherein adapting the ranked order comprises iteratively dragging and dropping images at different locations defining the ranked order by the user via the UI, and receiving from the user via the UI an indication that the user completed the adaptions of the ranked order.

12. The computer implemented method of claim 10, further comprising identifying combinatorial pairs of images of the adapted ranked order, wherein annotating comprises annotating each combinatorial pair of images, and wherein training comprises training the ranker model on the annotated combinatorial pairs.

13. The computer implemented method of claim 10, wherein the number of selected images corresponding to the number of classes of the certain scale presented within the UI represent an initial number, and further comprising during the current iteration: receiving from the user via the UI, an adaption of the number of classes of the certain scale presented within the UI, and dynamically updating the number of selected images presented within the UI according to the adapted number of classes.

14. The computer implemented method of claim 1, further comprising, during the iterations, receiving a selection of an image by a user via the UI, indicating that the selected image is irrelevant to the ranking of the at least one visual feature, and excluding the selected image from the training of the ranker model.

15. The computer implemented method of claim 1, further comprising: analyzing the plurality of images using the ranker model for selecting a core-set of the plurality of images representing major differences between images in terms of the at least one visual feature, wherein the at least two images are randomly selected from the core-set.

16. The computer implemented method of claim 1, wherein the ranking of the at least two images comprises an indication that the at least two images are ranked similarly with respect to the at least one visual feature.

17. The computer implemented method of claim 1, wherein the plurality of images depict objects of a same type that vary by the at least one visual feature.

18. The computer implemented method of claim 1, wherein the plurality of images depict a same type of vegetable that vary by at least one visually apparent trait.

19. A computer implemented method of inference using a ranker model created using user input provided via a user interface (UI), comprising: feeding an input image depicting at least one visual feature into a ranker model; and obtaining as an outcome of the ranker model, a ranking of the input image according to the at least one visual feature, wherein the ranker model is trained by: accessing a plurality of images varying by the at least one visual feature; in a plurality of iterations: presenting on a display, within the UI, at least two images selected from the plurality of images, in response to a ranking of the at least two images on a scale of the at least one visual feature made by a user via the UI, annotating the at least two images with respective annotations, andsequentially and dynamically training the ranker model using the at least two images and respective annotation; and when the plurality of iterations are terminated, providing the ranker model.

20. A system for training a ranker model using user input provided via a user interface (UI), comprising: at least one hardware processor executing a code for: accessing a plurality of images varying by at least one visual feature; in a plurality of iterations: presenting on a display, within the UI, at least two images selected from the plurality of images, in response to a ranking of the at least two images on a scale of the at least one visual feature made by a user via the UI, annotating the at least two images with respective annotations, and sequentially and dynamically training the ranker model using the at least two images and respective annotation; and when the plurality of iterations are terminated, providing the ranker model for ranking an input images according to the at least one visual feature.

Images