Pipeline for marking data

Abstract

Systems and methods disclosed herein provide a computer system, the computer system is configured to receive a plurality of images, select a region of at least one image defined by a bounding box, trim the selected region from the image and store the trimmed image in a folder, filter erroneously identified objects, generate a pseudo tag for remaining images, and assign a correct item name for the pseudo tag.

Description

[0001] Cross-references to related patent applications

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application Serial No. 63 / 415,898, filed on October 13, 2022, entitled “PIPELINE FOR LABELING DATA,” under 35 U.S.C. §119(e). The entire contents of U.S. Provisional Patent Application Serial No. 63 / 415,898 are incorporated herein by reference. Technical Field

[0003] Embodiments generally relate to the field of computers. Embodiments also relate to the field of machine learning. Embodiments also relate to the field of image labeling. Embodiments also relate to the field of training neural networks. Embodiments also relate to the field of computer devices and mobile devices for labeling data. Embodiments also relate to methods, systems and devices for machine learning. Embodiments also relate to methods and systems for labeling image data using computers. Background Art

[0004] With the advent of mobile devices, consumers have unprecedented access to image-generating devices. Approximately 4.7 billion photos are taken every day, a nearly three-fold increase from 2013. Experts expect this trend to continue. This has created a vast catalog of image data that continues to grow.

[0005] This library of photos provides an underutilized data source that can be used in science, engineering, business, education, and countless other applications. One of the main challenges in harnessing the power of this data is how to systematically label the data to make it more searchable. Similarly, advances in machine learning-based techniques have led to a high demand for image labeling.

[0006] There are many methods available for labeling data. For example, An image labeling service called "AI PLATFORMDATALABELING SERVICE" is provided, which is a paid service for manually labeled data sets. The prices of these services vary, but the bottleneck of such methods lies in the means of selecting, demarcating, and then labeling the images. This type of labeling is very cumbersome and unnecessarily expensive. However, it is also crucial for machine learning algorithms to have training data to function properly.

[0007] Therefore, there is a need in the art for methods and systems for pipeline data labeling, as disclosed in the embodiments described herein. Summary of the invention

[0008] The following summary is intended to help understand some unique and innovative features of the disclosed embodiments, but is not a complete description. By taking the entire specification, claims, drawings and abstract as a whole, a comprehensive understanding of the various aspects of the embodiments can be obtained.

[0009] Accordingly, one aspect of the disclosed embodiments is to provide improved image tagging methods and systems.

[0010] Another aspect of the disclosed embodiment is to provide a method, system and apparatus for labeling image data using a computer.

[0011] Another aspect of the disclosed embodiments is to provide methods, systems, and apparatus for labeling training data for machine learning.

[0012] For example, in certain embodiments, the systems and methods disclosed herein include a computer system configured to receive a plurality of images, select an area of ​​at least one image defined by a bounding box, crop the selected area from the image and store the cropped image in a folder, filter incorrectly identified objects, generate pseudo-labels for remaining images, and assign correct item names to the pseudo-labels.

[0013] In an embodiment, a system includes a computer system, the computer system further including: at least one processor; a graphical user interface; and a computer-usable medium containing computer program code, the computer-usable medium being able to communicate with the at least one processor, the computer program code including instructions executable by the at least one processor and configured to: receive a plurality of images; select a region of interest defined by a bounding box in at least one of the plurality of images; crop the selected region from the image and store the cropped image in a folder; filter incorrectly identified objects; generate pseudo-labels for the remaining images; and assign correct item names to the pseudo-labels. In an embodiment of the system, the plurality of images includes at least one of the following: an image file; a video file; and a video frame file. In an embodiment of the system, the computer program code including instructions executable by the at least one processor is further configured to identify any image in the plurality of images that is missing a bounding box. In an embodiment of the system, the computer program code including instructions executable by the at least one processor is further configured to generate an annotation file, the annotation file corresponding to the plurality of images received. In an embodiment, the folder further includes a folder name, the folder name corresponding to an object. In an embodiment of the system, the cropped images in the folder follow a file naming convention. In an embodiment of the system, the file naming convention includes a file name of the type [original image name]-[line number in the annotation file]. In an embodiment of the system, the computer program code including instructions executable by the at least one processor is also configured to sort the cropped images by file size. In an embodiment of the system, the computer program code including instructions executable by the at least one processor is also configured to sort the cropped images by file name. In an embodiment of the system, the computer program code including instructions executable by the at least one processor is also configured to train a labeling neural network, wherein the trained labeling neural network is used to generate the pseudo-labels for the remaining images. In an embodiment of the system, the computer program code including instructions executable by the at least one processor is also configured to train a classification neural network, wherein the trained classification neural network is used to assign the correct project name to the pseudo-label.

[0014] In another embodiment, a method includes: receiving a plurality of images; selecting a region of interest defined by a bounding box in at least one of the plurality of images; cropping the selected region from the image and storing the cropped image in a folder; filtering incorrectly identified objects; generating pseudo-labels for the remaining images; and assigning the pseudo-labels with correct project names. In an embodiment, the method also includes identifying any image in the plurality of images that is missing a bounding box. In an embodiment, the method also includes generating an annotation file, the annotation file corresponding to the plurality of images received. In an embodiment, the method also includes sorting the cropped images by file size; and removing the cropped images from the incorrect folder. In an embodiment, the method also includes sorting the cropped images by file name; and removing the cropped images from the incorrect folder. In an embodiment, the method also includes training a labeling neural network, wherein the trained labeling neural network is used to generate the pseudo-labels for the remaining images. In an embodiment, the method also includes training a classification neural network, wherein the trained classification neural network is used to assign the pseudo-labels with the correct project names.

[0015] In another embodiment, a system includes a computer system, the computer system further including: at least one processor; a graphical user interface; and a computer-usable medium containing computer program code, the computer-usable medium being able to communicate with the at least one processor, the computer program code including instructions executable by the at least one processor and configured to: receive a plurality of images; select a region of interest defined by a bounding box in at least one of the plurality of images; crop the selected region from the image and store the cropped image in a folder; generate an annotation file, the annotation file corresponding to the plurality of images received; sort the cropped images by file size; remove the cropped images from an erroneous folder; sort the cropped images by file name; remove the cropped images from an erroneous folder; generate pseudo-labels for the remaining images using a labeling neural network; and assign correct project names to the pseudo-labels using a classification neural network. In an embodiment of the system, the computer program code including instructions executable by the at least one processor is further configured to: train a labeling neural network, wherein the trained labeling neural network is used to generate the pseudo-labels for the remaining images; and train a classification neural network, wherein the trained classification neural network is used to assign the correct project names to the pseudo-labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Like reference numbers in the accompanying drawings, which are incorporated into and constitute a part of the specification, refer to identical or functionally similar elements throughout the various views and further illustrate the embodiments and, together with the detailed description, serve to explain the embodiments disclosed herein.

[0017] Figure 1 depicts a block diagram of a computer system implemented in accordance with the disclosed embodiments;

[0018] Figure 2 depicts a graphical representation of a network of data processing devices in which aspects of embodiments of the present invention may be implemented;

[0019] Figure 3 Depicted is a diagram for guiding a user according to an example embodiment. Figure 1 a computer software system for the operation of the data processing system shown;

[0020] Figure 4A depicts steps associated with a method of marking an image according to disclosed embodiments;

[0021] Figure 4B depicts steps associated with a method of training a classifier according to disclosed embodiments;

[0022] Figure 5A depicts steps associated with a method of sorting and filtering image data in accordance with disclosed embodiments;

[0023] Figure 5B depicts steps associated with a method of sorting and filtering image data in accordance with disclosed embodiments;

[0024] Figure 6 depicts a block diagram of a system for labeling an image in accordance with disclosed embodiments;

[0025] Fig. 7A depicts exemplary file names according to disclosed embodiments;

[0026] Figure 7B depicts an exemplary file folder in accordance with the disclosed embodiments;

[0027] Figure 8 depicts a process for sorting and filtering file data in accordance with disclosed embodiments;

[0028] Fig. 9 Depicts exemplary steps in a computer-implemented method for labeling an image in accordance with disclosed embodiments; and

[0029] Fig.10Depicted are exemplary steps in a computer-implemented method for training a neural network in accordance with disclosed embodiments. DETAILED DESCRIPTION

[0030] The details described below are intended to be exemplary and are provided to illustrate one or more embodiments and are not intended to limit the scope thereof.

[0031] Such exemplary embodiments will be described more fully below, including with reference to the accompanying drawings showing exemplary embodiments. The systems and methods disclosed herein may be embodied in various ways and should not be construed as being limited to the embodiments set forth herein. The description is provided to make the present disclosure thorough and complete and to fully convey the scope of the embodiments to those skilled in the art. The same reference numerals may refer to the same elements throughout.

[0032] The terms used herein are only used to describe specific embodiments and are not intended to be limiting. As used herein, singular forms such as "a", "an" and "the" are intended to include plural forms as well, unless the context clearly indicates otherwise. Similarly, the terms "include" and / or "comprise" used herein specify the presence of the features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of other features, integers, steps, operations, elements, components and / or groups thereof.

[0033] Throughout the specification and claims, terms may have nuanced meanings that are suggested or implied in the context beyond their explicitly stated meanings. Likewise, the phrase "in one embodiment" as used herein does not necessarily refer to the same embodiment, and the phrase "in another embodiment" as used herein does not necessarily refer to a different embodiment. For example, the claimed subject matter is intended to include all or part of a combination of the example embodiments.

[0034] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It should also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with that in the context of the relevant technology and should not be interpreted as an idealized or overly formal meaning unless explicitly defined herein.

[0035] It is contemplated that any embodiment discussed in this specification can be implemented with respect to any method, kit, reagent or composition of the invention, and vice versa. In addition, the compositions of the invention can be used to implement the methods of the invention.

[0036] It should be understood that the specific embodiments described herein are shown only by way of illustration and are not intended to limit the present invention. Key features may be employed in various embodiments without departing from the scope of the present invention. Those skilled in the art will recognize or be able to determine multiple equivalents of the specific procedures described herein using only routine experiments. Such equivalents are considered to be within the scope of the present invention and are covered by the claims.

[0037] In the claims and / or the specification, the word "a" and "an" when used in conjunction with the term "comprising" may mean "one", but is also consistent with the meaning of "one or more", "at least one", and "one or more than one". The term "or" used in the claims is used to mean "and / or" unless it is explicitly stated to refer only to alternatives or the alternatives are mutually exclusive, although the present disclosure supports a definition that refers only to alternatives and "and / or". Throughout this application, the term "about" is used to indicate that a value includes the variation of errors inherent in the device, the method used to determine the value, or the variation that exists among the subjects of study.

[0038] As used in this specification and claims, the words "comprising" (and any form of including, such as "comprise" and "comprises"), "having" (and any form of having, such as "have" and "has"), "including" (and any form of containing, such as "includes" and "include"), or "containing" (and any form of containing, such as "contains" and "contain") are inclusive or open-ended and do not exclude additional, unrecited elements or method steps.

[0039] As used herein, the term "or combinations thereof" refers to all permutations and combinations of the items listed before the term. For example, "A, B, C, or combinations thereof" is intended to include at least one of: A, B, C, AB, AC, BC, or ABC, and also BA, CA, CB, CBA, BCA, ACB, BAC, or CAB if the order is important in a particular context. Continuing with this example, combinations containing one or more items or terms repeated, such as BB, AAA, AB, BBC, AAABCCCC, CBBAAA, CABABB, and the like, are expressly included. It will be understood by those skilled in the art that there is generally no limit to the number of items or terms in any combination unless otherwise apparent from the context.

[0040] In view of the present disclosure, all compositions and / or methods disclosed and claimed herein can be made and executed without undue experimentation. Although the compositions and methods of the present invention have been described according to preferred embodiments, it will be apparent to those skilled in the art that variations may be made to the compositions and / or methods and the steps or sequence of steps of the methods described herein without departing from the concept, spirit and scope of the present invention. All such similar substitutions and modifications apparent to those skilled in the art are deemed to be within the spirit, scope and concept of the present invention as defined by the appended claims.

[0041] Figures 1 to 3 is an exemplary diagram of a data processing environment in which embodiments of the present invention may be implemented. It should be understood that Figures 1 to 3 It is merely exemplary and is not intended to assert or imply any limitation with respect to aspects of the disclosed embodiments or the environments in which the embodiments may be implemented. Many modifications to the depicted environments may be made without departing from the spirit and scope of the disclosed embodiments.

[0042] Figure 1 A block diagram of a computer system 100 that executes programming for implementing portions of the methods and systems disclosed herein is shown. A computing device in the form of a computer 110 configured to interface with sensors, peripherals, and other elements disclosed herein may include one or more processing units 102, a memory 104, a removable storage device 112, and a non-removable storage device 114. A processor or processing unit 102 as used herein refers to one or more processors that perform the functions described, or to a plurality of processors that collectively perform the desired functions among themselves, whereby the functions may be divided among one or more processors to achieve the desired results. In some embodiments, the processing unit 102 may include one or more GPUs. The memory 104 may include a volatile memory 106 and a non-volatile memory 108. The computer 110 may include or have access to a computing environment that includes various temporary and non-temporary computer-readable media, such as the volatile memory 106 and the non-volatile memory 108, the removable storage device 112, and the non-removable storage device 114. Computer storage includes, for example, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD ROM), digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium capable of storing computer readable instructions and data (including image data).

[0043] The computer 110 may include or have access to a computing environment including an input 116, an output 118, and a communication connection 120. The computer may operate in a networked environment, using the communication connection 120 to connect to one or more remote computers, remote sensors, detection devices, handheld devices, multifunction devices (MFDs), mobile devices, tablet devices, mobile phones, smart phones, or other such devices. The remote computers may also include personal computers (PCs), servers, routers, network PCs, RFID-enabled devices, peer devices, or other common network nodes, etc. The communication connection may include a local area network (LAN), a wide area network (WAN), a Bluetooth connection, or other network. This functionality is described below with respect to Figure 2 A more detailed description is given in the related description.

[0044] The output 118 is most commonly provided as a computer monitor, but may include any output device. The output 118 and / or the input 116 may include a data collection device associated with the computer system 100. In addition, the input 116, which typically includes a computer keyboard and / or a pointing device (e.g., a computer mouse, a computer trackpad, etc.), allows a user to select and instruct the computer system 100. The output 118 and the input 116 may be used to provide a user interface. The output 118 may be used as a display for displaying data and information to a user, as well as for interactively displaying a graphical user interface (GUI) 130.

[0045] Note that the term "GUI" generally refers to an environment that presents programs, files, options, and the like with the aid of icons, menus, and dialog boxes that are graphically displayed on a computer monitor screen. A user can interact with the GUI by directly touching the screen and / or pointing and clicking with a user input device 116 (e.g., a pointing device such as a mouse and / or a keyboard) to select and activate these options. Specific items can function in the same manner to the user in all applications because the GUI provides standard software routines (e.g., module 125) to manipulate these elements and report the user's actions. As described below, the GUI can also be used to display an electronic service image frame.

[0046] Computer readable instructions (e.g., program modules or nodes 125, which may represent other modules or nodes described herein) are stored on computer readable media and can be executed by processing unit 102 of computer 110. Program modules or nodes 125 may include computer applications. Hard disks, CD-ROMs, RAM, flash memory, and USB drives are just some examples of items that include computer readable media.

[0047] Figure 2A graphical representation of a network of data processing systems 200 in which aspects of the present invention may be implemented is depicted. Network data processing system 200 is a network of computers or other devices (e.g., mobile phones, smart phones, sensors, detection devices, etc.) in which embodiments of the present invention may be implemented. Note that system 200 may be implemented in the context of software modules such as program modules 125. System 200 includes a network 202 in communication with one or more clients 210, 212, and 214, and an external device 205, which may be a computer, camera, or other such device. Network 202 may also communicate with one or more RFID and / or GPS enabled devices or sensors, neural network 204, server 206, and storage device 208. Network 202 is a medium that may be used to provide communication links between various devices and computers connected together within a networked data processing system (e.g., computer system 100). Network 202 may include connections such as wired communication links, various types of wireless communication links, fiber optic cables, quantum or quantum encryption or quantum teleportation networks, and the like. The network 202 may communicate with one or more servers 206, one or more external devices (e.g., RFID and / or GPS enabled devices) or neural networks 204, and memory storage units (e.g., memory or database 208). It should be understood that the RFID and / or GPS enabled devices or neural networks 204 may be embodied as a module on a mobile device, cell phone, tablet device, monitoring device, detector device, sensor microcontroller, controller, receiver, transceiver, or other such device.

[0048] In the depicted example, RFID and / or GPS enabled devices, neural network 204, server 206, and clients 210, 212, and 214 are connected to network 202 along with storage unit 208. Clients 210, 212, and 214 may be, for example, personal computers or network computers, handheld devices, mobile devices, tablet devices, smart phones, personal digital assistants, microcontrollers, recording devices, MFDs, etc. Figure 1 The illustrated computer system 100 may be, for example, a client, such as client 210 and / or 212 .

[0049] Depending on design considerations, computer system 100 may also be implemented as a server, such as server 206. In the depicted example, server 206 provides data such as boot files, operating system images, applications, and application updates to clients 210, 212, and / or 214. In this example, clients 210, 212, and 214, RFID and / or GPS enabled devices, and neural network 204 are clients to server 206. Network data processing system 200 may include additional servers, clients, and other devices not shown. Specifically, a client may connect to any member of a network of servers that provide equivalent content.

[0050] In the depicted example, network data processing system 200 is the Internet, with network 202 representing a worldwide collection of networks and gateways that use the Transmission Control Protocol / Internet Protocol (TCP / IP) suite of protocols to communicate with one another. At the heart of the Internet is a backbone of high-speed data communication lines that route data and messages between major nodes or host computers consisting of thousands of commercial, government, educational, and other computer systems. Of course, network data processing system 200 may also be implemented as a variety of different types of networks, such as an intranet, a local area network (LAN), or a wide area network (WAN). Figure 1 and Figure 2 This is merely an example and is not intended to be an architectural limitation for different embodiments of the present invention.

[0051] Figure 3 A software system 300 is illustrated that can be used to direct a data processing system (eg, Figure 1 The software application 305 may be stored in Figure 1 104, on removable storage 112, or on non-removable storage 114, and typically includes and / or is associated with a kernel or operating system 310 and a shell or interface 315. One or more applications (e.g., modules or nodes 125) can be "loaded" (i.e., transferred from removable storage 112 to memory 104) for execution by data processing system 100. Data processing system 100 can receive user commands and data through user interface 315, which can include input terminals 116 and output terminals 118 accessible to user 320. Computer system 100 can then act on these inputs according to instructions from operating system 310 and / or software applications 305 and any software modules 125 thereof.

[0052] Generally speaking, program modules (e.g., module 125) may include, but are not limited to, routines, subroutines, software applications, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types and instructions. In addition, those skilled in the art will appreciate that elements of the disclosed methods and systems may be practiced using other computer system configurations, such as handheld devices, mobile phones, smart phones, tablet devices, multiprocessor systems, printers, copiers, fax machines, multi-function devices, data networks, microprocessor-based or programmable consumer electronics, networked personal computers, minicomputers, mainframe computers, servers, medical devices, medical equipment, etc.

[0053] Note that the term "module" or "node" as used in this article can refer to a collection of routines and data structures that perform a specific task or implement a specific abstract data type. A module can consist of two parts: an interface, which lists the constants, data types, variables, and routines that can be accessed by other modules or routines; and an implementation, which is usually private (accessible only by that module) and contains the source code that actually implements the routines in the module. The term "module" can also simply refer to an application, such as a computer program designed to assist in performing a specific task (such as word processing, accounting, inventory management, etc.), or a hardware component designed to equivalently assist in performing a task.

[0054] An interface 315 (e.g., a graphical user interface 130) may be used to display the results, after which a user 320 may provide additional input or terminate a particular session. In some embodiments, the operating system 310 and the GUI 130 may be implemented in a "Windows" system environment. Of course, it will be appreciated that other types of systems are possible. For example, in addition to the traditional "Windows" system, other operating systems may be used for the operating system 310 and the interface 315, such as a real-time operating system (RTOS) that is more commonly used in wireless systems. The software application 305 may include, for example, a module 125, which may include instructions for performing steps or logical operations such as those shown and described herein.

[0055] The following description is directed to embodiments of the present invention, which may be implemented in the context of, or require the use of, a data processing system, such as computer system 100, which incorporates Figures 1 to 3, and the program modules 125 and the data processing system 200 and the network 202 shown in . However, the present invention is not limited to any particular application or any particular environment. On the contrary, those skilled in the art will find that the system and method of the present invention can be advantageously applied to various systems and application software, including database management systems, word processors, etc. In addition, the present invention can be implemented on a variety of different platforms, including Windows, Macintosh, UNIX, LINUX, Android, Arduino, etc. Therefore, the description of the following exemplary embodiments is for illustrative purposes only and is not to be considered as limiting.

[0056] In embodiments disclosed herein, systems, methods, and apparatus may include aspects for training deep neural networks for applications such as object detection, segmentation, and 3D object detection. This approach requires a large amount of accurately labeled data. Generally speaking, a labeler labels an image by drawing a bounding box / polygon / 3D bounding box / 3D polygon, etc. around the object to be detected and assigns a name from a category to the object.

[0057] The disclosed system and method is intended to label image data quickly and efficiently. This can be achieved by drawing the correct bounding box / polygon / 3D bounding box / 3D polygon, etc. around the object. Next, in the review step, the labeling results are reviewed using a review software module. The image is cropped and all cropped images identified in the labeling and review process are placed into a folder. Using the disclosed sorting method, the reviewer can easily find any incorrectly saved cropped images.

[0058] Figure 4A A method 400 of labeling data according to the disclosed embodiments is illustrated. The method 400 begins at step 402. As shown at step 404, a labeling task is assigned and then accepted. Typically, this includes the submission of a batch of one or more images as image data files, and an indication of labeling criteria for the images. In certain embodiments, the labeling task may include labeling an array of images, wherein one or more objects in the array of photos are to be identified. For example, the labeling task may include an array of 20 photos, video frames, or videos, and a request to identify or label an object such as "bird" in the image data containing the array of images, videos, or video frames.

[0059] Therefore, as a preliminary step of method 400, project data may be collected. For example, this may include information indicating the objects to be identified in the image and providing the image itself. For example, the number of images "I" may be provided by a client or user. If the total number of possible labels L is known, the total number may be provided by the following equation (1):

[0060] Total number of tasks = I*L(1)

[0061] Therefore, if the number of images I is 10 and the total number of possible labels L is 500, the total number of tasks required is 5000. In some embodiments, various aspects of the labeling task can also be specified. For example, Table 1 illustrates options indicating the data type, task purpose, and bounding box type, as shown below:

[0062] Table 1

[0063] Data Types Task Choose a style image Bounding Box Bounding Box image segmentation segmentation image Spin Box Bounding Box image Polygon / Polyline Polygon video Object Tracking Bounding Box

[0064] Next at step 406, a bounding box / polygon / 3D boundary or other such boundary line, such as a box / 3D polygon, may be drawn around the object of interest. In some embodiments, this may be done by a human labeler whose task is to add the bounding box to the image. In other embodiments, an existing model may be used to provide the bounding box / polygon / 3D bounding box / 3D polygon instead of using a human labeler. For example, if a model is able to label one category of an image (e.g., a person in the image), the model may also be used to label other similar categories (e.g., monkeys) if necessary.

[0065] In some embodiments, the bounding box can be drawn by a person using a bounding box drawing application and an associated GUI provided by a marking software module, as further described herein. The system automatically assigns a first marking item to the object and saves the result to an annotation file. Please note that the traditional view is to assign the correct project name to the object. However, if there are 100 projects / objects, the marker needs to browse all 100 projects to find the correct name, which is very time-consuming. Therefore, in the current method, the first project name or label is assigned to the object, regardless of whether this is the correct project name.

[0066] Once the initial steps are completed, the review process can begin. In the review step 408, the image tags can be reviewed to ensure that all images have no missing bounding boxes / polygons / 3D bounding boxes / 3D polygons. This step is to ensure that bounding boxes have been properly added to all files where appropriate.

[0067] Next, at step 410, the reviewer module may be used to crop objects from the image and place them in corresponding folders named after the corresponding projects. In some embodiments, a file in the system (optionally named "classes.txt" (or other such name)) contains the names of all projects. In some embodiments, aspects of this step may be accomplished by a human using a cropping application and associated GUI provided by the reviewer module.

[0068] When saving the cropped image of the object, the review module can use a naming convention. The following is an exemplary naming convention, but it should be understood that other naming conventions can be used. In some embodiments, an exemplary naming convention includes: [original image name]-[line number in annotation file].jpg.

[0069] Fig. 7A and Figure 7B An example illustrating this situation is, Fig. 7A An array of image data files 704 having file names 702 is shown, Figure 7B Folder 750 is illustrated, where name 752 corresponds to the project name. Note that folder name 752 can be related to the objects identified in the initial step, as those aspects of the image that need to be identified. As shown in the exemplary annotation file, each line can have 5 numbers. The first is the index number of the project. The index number is used to find the project name in the classes.txt file. The remaining 4 numbers give information about the location of the bounding box in the image. When cropping an image, the reviewer module will read each line and use the bounding box information to crop the project image from the original image. Therefore, the name of the original image (provided by the customer or user) is the name given by the customer or user. However, for the cropped image, the name is nominally assigned as: [original image name]_[corresponding line number in the annotation file].jpg.

[0070] At step 412, the images in each folder are sorted. In an embodiment, the images in each folder may be sorted using Figure 5A and / or Figure 5B The sorting method shown in sorts the cropped images in various folders so that misplaced cropped images or “false positive bounding boxes” can be more easily identified at step 412 .

[0071] A false positive bounding box as used in this article refers to a bounding box in the original image that has the name of the item, but there is no object or many objects in the bounding box. In such examples, the cropped image is a false positive image or a false positive bounding box.

[0072] In some aspects, such as Figure 5A As shown, the cropped images in each folder can be sorted by the size of the cropped images at 502. False positive images usually have different sizes, for example, they are either very large or very small compared to other correctly labeled images. Sorting by size will move those images that are likely to be wrongly labeled to the corresponding beginning or end of the sorted images. This makes it very easy to identify such erroneously identified images at step 504. Once false positive images are detected, they are easily deleted because they are either at the bottom or at the top in the sorting.

[0073] Next, if Figure 5B As shown, the images in the folder can be sorted by name at 506. At step 508, the images after the cropping of the nth row are filtered out and sorted by name. For example, in many cases, the images from the client are from videos and are named: 1.jpg, 2.jpg, etc. Because they are from videos, 1.jpg is similar to 2.jpg. In this case, the objects in the nth row of 1.jpg and the nth row of 2.jpg are the same in many cases. By doing this, it can be easily seen that the misplaced objects are grouped together, as shown at 510, and it is very convenient to move the entire misplaced group to the correct folder.

[0074] For example, Figure 8 An exemplary file annotation 802 is provided. As shown, in the annotation file 802, each line has 5 numbers. The first is the index number of the item. The index number is used to find the item name in the classes.txt file. The remaining 4 numbers give information about the location of the bounding box in the image. When cropping an image, the regenerator module can read each line and use the bounding box information to crop the item image from the original image.

[0075] Next, at step 414, the regenerator module can correct the annotation file of the original image. This is done iteratively. As shown, the original annotation file contains several lines. Each line corresponds to the information of the project: the name and the bounding box position. The bounding box position may be correct, but the name may not be correct. If the name is incorrect, the corresponding cropped image will be placed in the wrong folder. In the review step, the reviewer can move the cropped image to the correct folder. The program module can then correct the annotation file and read all images in all folders. When it sees the cropped image in the folder, it can collect the following information: the original image name, the line number, and the correct project name (i.e., the folder name). The module will then replace the project name in the corresponding line of the corresponding annotation file until all annotation files are correct.

[0076] Depending on the task (e.g., object detection, segmentation, 3D object detection, 3D segmentation), an object detection / segmentation neural network can be trained using the correct labeled data, such as Figure 4B As further shown. For example, the YoloV3 framework can be used to train the Yolo object detection neural network. In other embodiments, other methods can be used.

[0077] exist Figure 4BIn the illustrated embodiment, method 450 begins at 452. At step 454, an object detection neural network may be trained using the original images and the corresponding annotation files. Using this training, the object detection neural network may be trained to recognize bounding boxes and names of items. This is understood as a training process. Upon completion of the training process, the trained neural network may recognize the location of the bounding boxes and may recognize the corresponding names of a set of images.

[0078] In addition, using the cropped images, a classification neural network can be trained at step 460 and used at step 462 to assign the correct item name to the previously added tags. In order to train the object detection and image classification neural networks, a graphics processing unit (GPU) server and associated computer architecture can be used. In an embodiment, the classification neural network can be given a series of folders, each of which contains a large number of cropped images. During the training process, the classification neural network learns what image corresponds to what name (the name of the folder). Once the training process is complete, a trained classification neural network is available. The trained classification neural network can be used to predict the name of the cropped image (e.g., the folder name). The classification neural network can then be used to correct the folder of the cropped image by moving the cropped image to the correct folder.

[0079] Training a classification neural network may typically include: loading and normalizing the training and test datasets, defining a convolutional neural network, defining a loss function, training the network on the training data, and testing the network on the test data.

[0080] Fig.10 A method for training an object detection and / or image classification neural network according to the disclosed embodiments is illustrated, as shown in step 460. It should be understood that the steps listed in method 1000 are exemplary and other training methods are possible.

[0081] At step 1005, the computer system can be set up and the associated GPU can be selected and configured for optimal performance. Likewise, computing hardware such as RAM and memory storage can be configured. Next, at step 1010, an operating system and necessary drivers can be installed on the computer system. The driver can be selected based on the GPU associated with the computer system.

[0082] At step 1015, any libraries or toolkits required by the neural network may be installed. In some embodiments, this may include a GPU library for a specific model associated with the neural network being trained (e.g., a deep learning neural network).

[0083] Next, at step 1020, a machine learning framework may be installed on the GPU-based computer system. A deep learning framework may be selected to be compatible with the object detection task / image classification task. Once the machine learning framework is installed, an object detection / image classification library may be selected at step 1025. The object detection / image classification library should be selected to match the machine learning framework. At step 1030, a training data set may be prepared. This may require formatting to make the training data set compatible with the selected library.

[0084] The model is now ready for training at step 1035. This step may include configuring parameters associated with the model and training the model for the desired application (e.g., object detection or image classification). Once the model is trained, at step 1040, the model performance may be checked to ensure that the convergence of the model is acceptable. The trained model is then ready for deployment at 1045.

[0085] Once both neural networks are trained, processing can proceed to labeling the remaining images. First, for any remaining images, the object detection neural network can identify the location of the bounding box at step 456, and can identify the corresponding name. This information is saved as an annotation file, but it is only a pseudo-annotation file because it requires review by the reviewer. Figure 4A At step 416 , pseudo labels for the remaining images are generated.

[0086] Pseudo labels can then be used to check if there are any missing labels, and existing labels can be adjusted as needed at step 418. Once the pseudo labels are generated. The labeler can check if the bounding boxes have been correctly labeled. There is no need to check if the names of the bounding boxes are correct. If there are any missing bounding boxes, the labeler will add a bounding box and assign its name to the first name of the item. Once the labelers have completed their work, the reviewer will take over the task. The image can be cropped and placed in the corresponding folder. Then, a classification neural network can be used at step 420 to correct the folder of the cropped image by moving the cropped image to the correct folder.

[0087] Finally, the method includes manually checking the folders of cropped images. If any cropped images are misplaced, they can be moved to the correct folder. This is similar to step 412. Finally, as shown in step 414, the annotation file of the original image is regenerated using the regenerator module. The method ends at 422.

[0088] These steps can then be repeated, wherein the training neural network step can be repeated to obtain a better neural network to perform future tasks. For example, assume that a collection of 100,000 images is provided. For example, further assume that 10,000 images are selected. The object detection neural network can be trained using these 10,000 images, and the classification neural network can be trained using corresponding cropped images from the 10,000 images. According to the disclosed method, all 100,000 images can be labeled. The two neural networks are trained again using all the images and the corresponding cropped images, respectively, to improve the neural networks.

[0089] Figure 6 1 illustrates aspects of a computer system 600 and associated computer system architecture that can be used to implement the methods disclosed herein. It should be understood that the computer system 600 may include, for example, Figures 1 to 3 600. It should also be understood that the computer system may include multiple computer systems, each configured to provide one or more of the functions described herein. Computer system 600 may include I / O components, including but not limited to camera 602, touch screen interface / display 604, speaker 606, and microphone 608. I / O components may also include a keyboard, mouse, or other such input / output hardware.

[0090] The computer system 600 may include a labeling module 610 implemented as computer hardware or software. The labeling module 610 may include a bounding box drawing application 612 and an initial labeling module 614, which automatically assigns a first item to an object and saves the result to an annotation file. The labeling module 610 may be used to implement an initial labeling step, including but not limited to drawing a bounding box around an object in an image.

[0091] The computer system 600 may also include an inspector module 616 that includes a cropping application 618. In some implementations, the cropping application may include a software program configured to crop images and place them in corresponding folders 620 using names from a predefined file (e.g., "Classes.txt").

[0092] The computer system 600 may also include a regenerator module 622. The regenerator module may include a software program configured to regenerate the annotation file 624 for the original image. Figure 8 An example of this process is shown. Figure 8As shown, the image data file 802 includes a file name 804 and a line number 806. The image data file is sent to a corresponding folder 620. The regenerator module then creates an annotation file 624, which includes an entry 808 of the image data file 802, which includes a line 810, an index number 812 for finding the name of the item in the classes.txt file, and the position of the bounding box in the image represented by the last four numbers 814.

[0093] The computer system may also include a labeling neural network 626 and a classification neural network 628 .

[0094] Fig. 9 Example of using Figure 6 The exemplary method implemented by the computer system shown. The method starts at 902.

[0095] At step 904 , a bounding box drawing application 612 may be used to draw a bounding box around the object of interest in each data file in the data file array associated with the object detection task, and the bounding box may include a standard bounding box / polygon / 3D bounding box / 3D polygon, etc.

[0096] At step 906 , the computer system automatically assigns the first item to the object using the initial tagging application 614 and saves the result to an annotation file.

[0097] Next at step 908, each data file in the data file array is checked to ensure that they have no missing bounding boxes.

[0098] At step 910, the cropping application 618 may crop the objects of interest from the image and place them into corresponding folders 620. The corresponding folders may use a naming convention such that the folder name corresponds to the name of the object of interest. The cropped image may be saved according to the following exemplary naming convention: [original name]-[line number in the annotation file].jpg. This exemplary naming convention may be modified in other embodiments.

[0099] The cropped images in each folder are then sorted so that misplaced cropped images are more easily identified. At step 912, the cropped images in each folder are sorted by the size of the cropped images. The size of the images is usually quite standard. Sorting by file size makes it easy to identify abnormal files that are either unexpectedly large or unexpectedly small. At step 914, images that incorrectly identify objects are removed.

[0100] Next, at step 916, the cropped images in each folder are sorted by name. In many cases, the image data files may be videos and are therefore named in a sequential manner; for example, such as: "1.jpg", "2.jpg", etc. In this case, it should be expected that image data file 1.jpg is highly similar to image data file 2.jpg. Therefore, in most cases, the objects in the nth row of 1.jpg are the same as the nth row of 2.jpg. When sorting by name, misplaced objects are easily identified. At step 918, any misplaced cropped image files can be moved to the correct folder.

[0101] At step 920, the regenerator module 622 is used to regenerate the annotation files of the original images. These annotation files are correct. Figure 8 An example of this processing is illustrated.

[0102] At step 922, depending on the task (e.g., object detection, segmentation, 3D object detection, 3D segmentation), the labeling neural network 626 may be trained using the correct labeled data.

[0103] At step 924, pseudo labels for the remaining images are generated using the trained labeling neural network 626, which may be an object detection neural network.

[0104] At step 926, the pseudo-labels are used to check if any labels are missing and adjust the existing labels if necessary. As mentioned above, at this stage, it is not important whether the project name is correct.

[0105] At step 928, the sorted cropped images are used to train the classification neural network 628. At step 930, the classification neural network 628 is then used to assign the correct project name to the label added at step 926. In some cases, manual adjustments may also be required (such as steps 912 to 918). Finally, an annotation file is generated in step 920.

[0106] It should be understood that the process can then be repeated from step 910 to step 920 as needed to reach convergence. On this basis, steps 922 and 924 can be repeated as needed to improve the labeling neural network 626 for future tasks. The method ends at 932.

[0107] The embodiments disclosed herein greatly improve the tagging speed, increase the speed and accuracy of tagging tasks, and make the training of taggers easier.

[0108] It will be appreciated that variations of the above-disclosed and other features and functions, or alternatives thereof, may be desirably combined into many other different systems or applications.

[0109] In an embodiment, a system includes a computer system, which further includes: at least one processor and / or at least one GPU; a graphical user interface; and a computer-usable medium containing computer program code, which is capable of communicating with the at least one processor, and the computer program code includes instructions that can be executed by the at least one processor and is configured to: receive multiple images; select a region of interest defined by a bounding box in at least one of the multiple images; crop the selected region from the image and store the cropped image in a folder; filter incorrectly identified objects; generate pseudo-labels for the remaining images; and assign correct item names to the pseudo-labels.

[0110] In an embodiment, the plurality of images includes at least one of: an image file; a video file; and a video frame file.

[0111] In an embodiment of the system, the computer program code comprising instructions executable by the at least one processor is further configured to identify any image in the plurality of images that has a missing bounding box.

[0112] In an embodiment of the system, the computer program code comprising instructions executable by the at least one processor is further configured to generate an annotation file corresponding to the received plurality of images.

[0113] In an embodiment of the system, the folder further comprises a folder name, the folder name corresponding to the object. In an embodiment of the system, the cropped images in the folder follow a file naming convention. In an embodiment, the file naming convention comprises a file name of the type [original image name]-[line number in the annotation file].

[0114] In an embodiment of the system, the computer program code comprising instructions executable by the at least one processor is further configured to sort the cropped images by file size. In an embodiment of the system, the computer program code comprising instructions executable by the at least one processor is further configured to sort the cropped images by file name.

[0115] In an embodiment of the system, the computer program code comprising instructions executable by the at least one processor is further configured to train a labeling neural network, wherein the trained labeling neural network is used to generate the pseudo labels for the remaining images.

[0116] In an embodiment of the system, the computer program code comprising instructions executable by the at least one processor is further configured to train a classification neural network, wherein the trained classification neural network is used to assign the correct item name to the pseudo label.

[0117] In another embodiment, a method includes: receiving a plurality of images; selecting a region of interest defined by a bounding box in at least one of the plurality of images; cropping the selected region from the image and storing the cropped image in a folder; filtering incorrectly identified objects; generating pseudo-labels for remaining images; and assigning correct item names to the pseudo-labels.

[0118] In an embodiment, the method further comprises identifying any images in the plurality of images that are missing a bounding box.

[0119] In an embodiment, the method further comprises generating an annotation file corresponding to the received plurality of images.

[0120] In an embodiment, the method further comprises sorting the cropped images by file size; and removing the cropped images from the wrong folder. In an embodiment, the method further comprises sorting the cropped images by file name; and removing the cropped images from the wrong folder.

[0121] In an embodiment, the method further comprises training a labeling neural network, wherein the trained labeling neural network is used to generate the pseudo-labels for the remaining images. In an embodiment, the method further comprises training a classification neural network, wherein the trained classification neural network is used to assign the correct project names to the pseudo-labels. The GPU server may be used to train the neural network.

[0122] In an embodiment, a computer system can be set up and an associated GPU can be selected and configured for optimal performance. Similarly, computing hardware such as RAM and memory storage devices can be configured. An operating system and necessary drivers can be installed on the computer system. Drivers can be selected based on the GPU associated with the computer system. Any libraries or toolkits required for neural networks can be installed. In some embodiments, this can include a GPU library for a specific model associated with the neural network being trained (e.g., a deep learning neural network). Next, a machine learning framework can be installed on a GPU-based computer system. A deep learning framework can be selected to be compatible with object detection tasks / image classification tasks. Once the machine learning framework is installed, an object detection / image classification library can be selected. An object detection / image classification library should be selected to match the machine learning framework. A training data set can be prepared. This may require formatting to make the training data set compatible with the selected library. The model is now ready for training. This step can include configuring parameters associated with the model and training the model for the desired application (e.g., object detection or image classification). Once the model is trained, the model performance can be checked to ensure that the convergence of the model is acceptable. Then, the trained model is ready for deployment.

[0123] In another embodiment, a system includes a computer system, which further includes: at least one processor; a graphical user interface; and a computer-usable medium containing computer program code, which is capable of communicating with the at least one processor, and the computer program code includes instructions that can be executed by the at least one processor and is configured to: receive multiple images; select a region of interest defined by a bounding box in at least one image of the multiple images; crop the selected region from the image and store the cropped image in a folder; generate an annotation file, the annotation file corresponding to the received multiple images; sort the cropped images by file size; remove the cropped images from the wrong folder; sort the cropped images by file name; remove the cropped images from the wrong folder; generate pseudo-labels for the remaining images using a labeling neural network; and assign correct item names to the pseudo-labels using a classification neural network.

[0124] In an embodiment of the system, the computer program code comprising instructions executable by the at least one processor is further configured to: train a labeling neural network, wherein the trained labeling neural network is used to generate the pseudo-labels for the remaining images; and train a classification neural network, wherein the trained classification neural network is used to assign the correct item names to the pseudo-labels.

[0125] It is to be understood that various presently unforeseen or unanticipated alternatives, modifications, variations or improvements therein may be subsequently made by those skilled in the art which are also intended to be encompassed by the following claims.

Claims

1. A system, comprising: The computer system further comprises: at least one processor; Graphical user interface; and A computer usable medium containing computer program code capable of communicating with the at least one processor, the computer program code comprising instructions executable by the at least one processor and configured to: receiving a plurality of images; selecting a region of interest defined by a bounding box in at least one of the plurality of images; cropping a selected region of interest from the image and storing the cropped image in a folder; Filtering out incorrectly identified objects; Generate pseudo labels for the remaining images; and Assign the correct project name to the pseudo-label.

2. The system according to claim 1, wherein: The plurality of images includes at least one of the following: Image files; Video files; as well as Video frame file.

3. The system according to claim 1, wherein: The computer program code comprising instructions executable by the at least one processor is further configured to: Any images in the plurality of images that have missing bounding boxes are identified.

4. The system according to claim 1, wherein: The computer program code comprising instructions executable by the at least one processor is further configured to: An annotation file is generated, the annotation file corresponding to the received plurality of images.

5. The system according to claim 1, wherein: The folder also includes: The folder name that corresponds to the object.

6. The system according to claim 5, wherein: The cropped images in the folder follow a file naming convention.

7. The system according to claim 6, wherein: The file naming conventions include: File name of the type [original image name]-[line number in the annotation file].

8. The system according to claim 1, wherein: The computer program code comprising instructions executable by the at least one processor is further configured to: Sort the cropped images by file size.

9. The system according to claim 1, wherein: The computer program code comprising instructions executable by the at least one processor is further configured to: Sort the cropped images by file name.

10. The system according to claim 1, wherein: The computer program code comprising instructions executable by the at least one processor is further configured to: A labeling neural network is trained, wherein the trained labeling neural network is used to generate the pseudo labels for the remaining images.

11. The system according to claim 1, wherein: The computer program code comprising instructions executable by the at least one processor is further configured to: A classification neural network is trained, wherein the trained classification neural network is used to assign the correct item name to the pseudo label.

12. A method comprising: receiving a plurality of images; selecting a region of interest defined by a bounding box in at least one of the plurality of images; cropping a selected region from the image and storing the cropped image in a folder; Filtering out incorrectly identified objects; Generate pseudo labels for the remaining images; as well as Assign the correct project name to the pseudo-label.

13. The method according to claim 12, further comprising: Any images in the plurality of images that have missing bounding boxes are identified.

14. The method according to claim 12, further comprising: An annotation file is generated, the annotation file corresponding to the received plurality of images.

15. The method according to claim 12, further comprising: Sort the cropped images by file size; as well as Remove cropped images from wrong folder.

16. The method according to claim 12, further comprising: Sort the cropped images by file name; as well as Remove cropped images from wrong folder.

17. The method according to claim 12, further comprising: A labeling neural network is trained, wherein the trained labeling neural network is used to generate the pseudo labels for the remaining images.

18. The method according to claim 12, further comprising: A classification neural network is trained, wherein the trained classification neural network is used to assign the correct item name to the pseudo label.

19. A system, comprising: The computer system further comprises: at least one processor and at least one GPU; Graphical user interface; and A computer usable medium containing computer program code capable of communicating with the at least one processor, the computer program code comprising instructions executable by the at least one processor and configured to: receiving a plurality of images; selecting a region of interest defined by a bounding box in at least one of the plurality of images; cropping a selected region from the image and storing the cropped image in a folder; generating an annotation file corresponding to the received plurality of images; Sort the cropped images by file size; Remove cropped images from wrong folders; Sort the cropped images by file name; Remove cropped images from wrong folders; Generate pseudo labels for the remaining images using a labeling neural network; and A classification neural network is used to assign the correct item names to the pseudo labels.

20. The system of claim 19, wherein: The computer program code comprising instructions executable by the at least one processor is further configured to: training a labeling neural network, wherein the trained labeling neural network is used to generate the pseudo labels for the remaining images; and A classification neural network is trained, wherein the trained classification neural network is used to assign the correct item name to the pseudo label.

Images