Student Network Education for End-to-End Semi-Supervised Object Detection

Abstract

A system and method for end-to-end semi-supervised object detection is provided. The system generates an input batch by taking labeled and unlabeled images from an image dataset and applying an image transform to the images. The system further generates a first result for each image of the input batch by applying a teacher neural network to the input batch. For objects in unlabeled images of the batch, the first result includes candidate bounding boxes and scores for the boxes. The system determines a threshold score based on the scores to select a foreground bounding box from the candidates. The system generates a second result by applying a student neural network to the unlabeled images and calculates a training loss for the input batch based on the foreground bounding boxes and the second result. The system trains the student neural network based on the training loss.

Description

[Technical Field]

[0001] [Incorporation through cross-reference / citation to related applications]

[0001] This application claims priority to U.S. Patent Application No. 18 / 159,492, filed with the U.S. Patent and Trademark Office on 25 January 2023, which claims priority to U.S. Provisional Patent Application No. 63 / 268,863, filed on 4 March 2022, the entire contents of which application are incorporated herein by reference.

[0002]

[0002] Various embodiments of the present disclosure relate to neural networks and object detection. More specifically, various embodiments of the present disclosure relate to systems and methods for training student networks for end-to-end semi-supervised object detection. [Background technology]

[0003]

[0003] Advances in the fields of computer vision and artificial intelligence have led to the development of various types of neural networks (or models) for various applications such as object detection. Typically, the goal of object detection is to identify and locate objects associated with a specific class label from still image or video data. The location of an object in an image can be indicated by a bounding box superimposed on the image. Recently, neural network models have been used for object detection. Such models are trained on a training dataset that can include multiple images associated with each object class. For example, if a neural network is to be trained for object detection (e.g., dogs), the training dataset may include several images of the object, a class label, and the coordinates of a bounding box that can be placed around the object. Often, the images in the dataset are manually annotated by a person. For example, a person may label each image with a class label and annotate the image with a bounding box that includes the coordinates of the bounding box. In some cases, there may be a small number of examples of labeled images for a particular class. In such cases, semi-supervised learning (SSL) may be used. SSL takes advantage of the potential of unlabeled data to facilitate model training when a large annotated dataset is not available. While SSL methods have been successfully applied to image classification and object detection tasks, the complexity of object detector architecture design is hindering the transition of existing semi-supervised techniques from image classification to object detection.

[0004]

[0004] Those skilled in the art will be able to see the limitations and disadvantages of conventional methods by comparing the described system with some aspects of the disclosure shown with reference to the drawings in the remainder of this application. [Overview of the project] [Problems that the invention aims to solve]

[0005]

[0005] A system and method for educating a student network for end-to-end semi-supervised object detection is provided, substantially shown in at least one figure and / or described in relation to these figures, and more fully shown in the claims.

[0006]

[0006] These and other features and advantages of the Disclosure can be understood by considering the following detailed description of the Disclosure with reference to the accompanying drawings, which indicate the same elements throughout by the same reference numerals. [Brief explanation of the drawing]

[0007] [Figure 1] This figure shows a network environment for training a student network for end-to-end semi-supervised object detection, according to one embodiment of the present disclosure. [Figure 2] This is an exemplary block diagram of a system for training a student network for end-to-end semi-supervised object detection, according to one embodiment of the present disclosure. [Figure 3] This figure shows an exemplary architecture of a teacher-student framework for end-to-end semi-supervised object detection according to one embodiment of the present disclosure. [Figure 4] This flowchart shows an exemplary method for training a student network for end-to-end semi-supervised object detection according to one embodiment of the present disclosure. [Modes for carrying out the invention]

[0008]

[0011] The implementations described below can be found in the disclosed systems and methods for training student networks for end-to-end semi-supervised object detection. Object detection can be defined as the task of detecting instances of a particular class of objects in an image or video. In some cases, object detection further includes another task of generating bounding boxes around the detected objects. Object detection has applications in various fields such as autonomous vehicles, unmanned aerial vehicles (UAVs), mobile phone video surveillance, and image search systems. As an example, object detection is used in advanced driver-assistance systems (ADAS) to improve traffic safety by enabling vehicles to detect lane markings or pedestrians.

[0009]

[0012] This disclosure provides a system capable of extracting labeled and unlabeled images from an image dataset and generating an input batch by applying a set of image transformations to the labeled and unlabeled images. The system can further generate results for each image in the input batch by applying a teacher neural network to the input batch. The teacher neural network may be a network pre-trained for an object detection task, and the results for objects in the first unlabeled image of the input batch may include a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes. For an object, the system can determine a threshold score based on the set of scores, and based on the threshold score, can select a foreground bounding box from the set of candidate bounding boxes. The system can further generate results including bounding box predictions for the object by applying a student neural network to the first unlabeled image. The student neural network may be an untrained network to be trained for an object detection task. Based on the foreground bounding box and bounding box predictions, the system can calculate a training loss for the input batch, and based on the training loss, can retrain the student neural network for the object detection task.

[0010]

[0013] Recently, object detection tasks are achieved by using a neural network model (or multiple neural network models) that has been pre-trained for detecting one or more objects. To train a neural network model (or multiple neural network models), it is necessary to generate a dataset containing multiple training samples. Each training sample can contain at least one image of each of the one or more objects to be detected. Furthermore, each training sample can contain a class label associated with the object in the corresponding image, and the coordinates of the bounding box containing the object in the corresponding image.

[0011]

[0014] Generating a dataset requires collecting a large number of image examples (e.g., thousands) of each object. Typically, this collection is done manually from various sources and is a tedious task. In some cases, there may be a limited number of labeled image examples for a particular object class. In such cases, semi-supervised learning (SSL) may be used. SSL leverages the potential of unlabeled data to facilitate model training when large annotated datasets are unavailable. While SSL methods have been successfully applied to image classification and object detection tasks, the complexity of object detector architecture design hinders the transition of existing semi-supervised techniques from image classification to object detection.

[0012]

[0015] The present disclosure provides a neural network framework that can be based on semi-supervised learning for object detection. Specifically, in semi-supervised learning, unannotated (or unlabeled) data is used to facilitate the model learning of a neural network with limited annotated (or labeled) data. The present disclosure can use a teacher-student framework that performs pseudo-labeling on unlabeled images and uses these pseudo-labels along with some labeled images for each iteration to train a detector (i.e., the student neural network). The teacher-student framework includes a teacher neural network and a student neural network. The teacher neural network can be a network pre-trained for an object detection task, and the student neural network can be an untrained network that may need to be trained for the object detection task. The student neural network is trained for the object detection task based on a training loss calculated from the individual predictions of the teacher neural network and the student neural network.

[0013]

[0016] The present disclosure can be used in scenarios where the number of labeled images is limited (e.g., 1%) compared to the amount of unlabeled images in a dataset. Thus, the present disclosure can significantly reduce the human labor that may be required to generate a dataset and label the images in the dataset.

[0014]

[0017] The present disclosure proposes a new update mechanism composed of an Exponential Moving Average (EMA) and an Exponential Adaptive Difference Moving Average (E-ADMA) to update a teacher neural network from a student neural network during training time. The present disclosure also provides two new loss functions for classification, called background similarity loss function and foreground-background dissimilarity loss function, which can utilize the background / foreground predictions of the teacher neural network and the student neural network to improve the classification performance. The present disclosure can also disclose a jitter-bagging module that can help refine the bounding box prediction of the neural network. The present disclosure also proposes a new adaptive threshold mechanism for obtaining an optimal bounding box for classification and regression tasks.

[0015]

[0018] FIG. 1 is a diagram showing a network environment for training a student network for end-to-end semi-supervised object detection according to an embodiment of the present disclosure. Referring to FIG. 1, a diagram of a network environment 100 is shown. The network environment 100 includes a system 102. The system 102 includes a circuit 104 and a memory 106. The memory 106 can include, for example, a teacher neural network 108 and a student neural network 110. Referring to FIG. 1, a display device 112, a server 114, and a communication network 116 are further shown. As an example, an image dataset 118 and an input batch 120 are also shown.

[0016]

[0019] System 102 may include preferred logic, circuitry, and interfaces that can be configured to train a student neural network 110 for an object detection task. The object detection task may be a semi-supervised machine learning task in which the number of training examples (e.g., fewer than 4-5 images) is small for the target object class that the student neural network 110 needs to be trained on. Examples of System 102 include, but are not limited to, computing devices, mainframe machines, servers, computer workstations, game devices, and / or consumer electronic (CE) devices.

[0017]

[0020] Circuit 104 may include preferred logic, circuits, and interfaces that can be configured to execute program instructions associated with different operations to be performed by system 102. Circuit 104 can be implemented based on several processor technologies known in the art. Examples of processor technologies include, but are not limited to, central processing units (CPUs), x86-based processors, reduced instruction set computer (RISC) processors, application-specific integrated circuit (ASIC) processors, composite instruction set computer (CISC) processors, graphics processing units (GPUs), coprocessors (such as inference accelerators or artificial intelligence (AI) accelerators), and / or combinations thereof.

[0018]

[0021] Memory 106 may include preferred logic, circuitry, and / or interfaces that can be configured to store program instructions to be executed by circuitry 104. Memory 106 may also store the teacher neural network 108 and the student neural network 110. In at least one embodiment, memory 106 may also store the input batch 120 and intermediate or final results obtained from the teacher neural network 108 and the student neural network 110. Examples of implementations of memory 106 include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), hard disk drives (HDDs), solid-state drives (SSDs), CPU caches, and / or secure digital (SD) cards.

[0019]

[0022] Each of the teacher neural network 108 and the student neural network 110 can be a computational network or system of artificial neurons arranged in multiple layers. The multiple layers of the corresponding neural network can include an input layer, one or more hidden layers, and an output layer. Each of the multiple layers can include one or more nodes (or artificial neurons). The outputs of all nodes in the input layer can be coupled to at least one node in the (one or multiple) hidden layers. Similarly, the input of each hidden layer can be coupled to the output of at least one node in the other layers of the corresponding neural network. The output of each hidden layer can be coupled to the input of at least one node in the other layers of the corresponding neural network. The (one or multiple) nodes in the final layer can receive input from at least one hidden layer and output a result. The number of layers and the number of nodes in each layer can be determined from the hyperparameters of the corresponding neural network. Such hyperparameters can be set before or after training the corresponding neural network on the training dataset.

[0020]

[0023] Each node in a corresponding neural network can correspond to a mathematical function (e.g., a sigmoid function or a normalized linear unit) with a set of parameters that can be tuned during network training. These parameters may include, for example, weight parameters and regularization parameters. Each node can use the mathematical function to compute an output based on one or more inputs from nodes in other layers (e.g., previous layers) of the corresponding neural network. All or some nodes in a corresponding neural network can correspond to the same or different mathematical functions.

[0021]

[0024] In training a corresponding neural network, one or more parameters of each node in the corresponding neural network can be updated based on whether the output of the final layer for a given input (from the training dataset) matches the correct result based on the loss function for the corresponding neural network. The above process can be repeated for the same or different inputs until the minimum value of the loss function is reached and the training error is minimized. Several training methods, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, and metaheuristics, are known in the art.

[0022]

[0025] The teacher neural network 108 may include electronic data that can be implemented, for example, as a software component of an application executable on system 102. The teacher neural network 108 may rely on libraries, external scripts, or other logic / instructions executed by a processing device such as circuit 104. The teacher neural network 108 may include code and routines configured to enable a computing device such as circuit 104 to perform one or more actions for object detection. In addition, or alternatively, the teacher neural network 108 may also be implemented using hardware including a processor, a microprocessor (e.g., performing or controlling one or more actions), a field-programmable gate array (FPGA), a coprocessor (e.g., an inference accelerator), or an application-specific integrated circuit (ASIC). Alternatively, in some embodiments, the neural network may be implemented using a combination of both hardware and software.

[0023]

[0026] Similar to the teacher neural network 108, the student neural network 110 may include electronic data that can be implemented, for example, as a software component of an application executable on system 102. The student neural network 110 may rely on libraries, external scripts, or other logic / instructions executed by a processing device such as circuit 104. The student neural network 110 may include code and routines that can be configured to enable a computing device such as circuit 104 to perform one or more actions for object detection. In addition, or alternatively, the student neural network 110 may also be implemented using hardware including a processor, a microprocessor (e.g., performing or controlling the execution of one or more actions), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). Alternatively, in some embodiments, the neural network may be implemented using a combination of hardware and software.

[0024]

[0027] In one embodiment, the teacher neural network 108 may be a neural network that has been pre-trained for the object detection task. On the other hand, the student neural network 110 may be an untrained network that may need to be trained for the object detection task.

[0025]

[0028] Examples of the teacher neural network 108 and the student neural network 110 include, but are not limited to, deep neural networks (DNNs), convolutional neural networks (CNNs), region-based convolutional neural networks (R-CNNs), Fast R-CNNs, Faster R-CNNs, artificial neural networks (ANNs), (You Only Look Once) YOLO networks, CNNs + ANNs, fully connected neural networks, and / or combinations of such networks. In certain embodiments, the teacher neural network 108 and / or the student neural network 110 may be based on a hybrid architecture of multiple deep neural networks (DNNs).

[0026]

[0029] The display device 112 may include preferred logic, circuitry, and interfaces that can be configured to display labeled and unlabeled images from the image dataset 118. In one embodiment, the display device 112 may be configured to display an input batch 120 that can be generated by applying a set of image transformations to the labeled and unlabeled images. The display device 112 can be used to view the execution status of operations related to the training of the student neural network 110. The display device 112 can be implemented through several known technologies, such as liquid crystal display (LCD) displays, light-emitting diode (LED) displays, plasma displays, or organic LED (OLED) display technologies, or at least one of other display devices, but is not limited to the following. According to one embodiment, the display device 112 may mean a display screen for a head-mounted device (HMD), a smart glasses device, a see-through display, a projection display, an electrochromic display, or a transparent display.

[0027]

[0030] Server 114 may include suitable logic, circuitry, interfaces, and / or code that can be configured to store the image dataset 118. Server 114 may also be configured to store the input batch 120 and the results associated with the neural network. According to one embodiment, Server 114 can be implemented as a cloud server and can perform operations through web applications, cloud applications, HTTP requests, repository operations, file transfers, etc. Other implementation examples of Server 114 include, but are not limited to, media servers, database servers, file servers, web servers, application servers, mainframe servers, or cloud computing servers.

[0028]

[0031] In at least one embodiment, the server 114 can be implemented as multiple distributed cloud-based resources using some techniques well known to those skilled in the art. Those skilled in the art will understand that the scope of this disclosure is not limited to implementing the server 114 and system 102 as two separate entities. In certain embodiments, the functionality of the server 114 can be incorporated into system 102, in whole or at least in part, without departing from the scope of this disclosure.

[0029]

[0032] The communication network 116 may include a communication medium through which the system 102, the display device 112, and the server 114 can communicate with each other. The communication network 116 may include either a wired connection or a wireless connection. Examples of the communication network 116 include, but are not limited to, the internet, a cloud network, a cellular or wireless mobile network (such as Long-Term Evolution and 5G New Radio), a Wireless Fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), or a metropolitan area network (MAN). Various devices in the network environment 100 may be configured to connect to the communication network 116 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, Multihop Communication, Wireless Access Point (AP), Device-to-Device Communication, Cellular Communication Protocol, and Bluetooth (BT) Communication Protocol.

[0030]

[0033] The image dataset 118 can correspond to a set of one or more object instances and may include a set of labeled images 118A and a set of unlabeled images 118B. Each image in the image dataset 118 may contain at least one object. An object can be a living object or an inanimate object. Living objects may possess the properties or characteristics of a living organism, while inanimate objects may lack such characteristics. Examples of living objects include humans, birds, and animals. Examples of inanimate objects include rocks, chairs, and vehicles.

[0031]

[0034] Each image in the set of labeled images 118A can be labeled (or annotated) with the name of the object contained in the corresponding image. For example, if the image is of a dog, it can be labeled as a dog. In one embodiment, the image can be further labeled with the coordinates of the bounding box containing the object. Each unlabeled image 118B does not have to include labels for the (one or multiple) objects contained in the image.

[0032]

[0035] During operation, circuit 104 can extract labeled and unlabeled images from the image dataset 118. To extract such images, circuit 104 can randomly sample the image dataset 118 using a sampling ratio. For example, the sampling ratio can be set to 0.2 for unlabeled images and 0.5 for labeled images.

[0033]

[0036] Circuit 104 can generate an input batch (e.g., input batch 120) by applying a set of image transformations to labeled and unlabeled images. In one embodiment, the set of image transformations may include a first subset of image transformations that can be associated with a first data enhancement type (i.e., weak data enhancement) and a second subset of image transformations that can be associated with a second data enhancement type (i.e., strong data enhancement). The second data enhancement type may differ from the first data enhancement type. For example, the first subset of image transformations may include image inversion, image shift, etc. The second subset of image transformations may include one or more of the following: image rotation, blurring, contrast change, shear, masking of one or more areas of an image, jitter addition, random noise addition, etc.

[0034]

[0037] After the input batch 120 is generated, images from the input batch 120 are supplied to the teacher neural network 108 and the student neural network 110. For each image in the input batch 120, the circuit 104 can generate a first result (i.e., a supervised or unsupervised object detection result). The first result can be generated by applying the teacher neural network 108 to the images in the input batch 120. As described above, the teacher neural network 108 can be a network pre-trained for the object detection task. For an object in the first unlabeled image of the input batch 120, the first result may include a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes. Each score in the set of scores may correspond to a confidence score associated with the corresponding candidate bounding box. Specifically, the confidence score may indicate the likelihood that an object exists within the corresponding bounding box.

[0035]

[0038] According to one embodiment, the set of scores may include foreground scores and background scores. The foreground scores may relate to the foreground bounding boxes of the set of candidate bounding boxes, and the background scores may relate to the background bounding boxes of the set of candidate bounding boxes. Circuit 104 can determine a threshold score based on the foreground scores and background scores. Details regarding the determination of the threshold score are shown, for example, in Figure 3.

[0036]

[0039] Threshold selection can be performed to adaptively filter out bounding boxes that are not part of the foreground. In one embodiment, system 102 may be configured to apply a non-maximum suppression operation to a set of candidate bounding boxes to extract a subset of candidate bounding boxes from that set. Circuit 104 may select foreground bounding boxes from the subset of candidate bounding boxes based on the determined threshold score. Foreground bounding boxes may be selected to be used as ground truth for student neural network 110. Circuit 104 may produce a second result by applying student neural network 110 to a first unlabeled image. The second result may include bounding box predictions for objects. As described above, student neural network 110 may be an untrained network that may need to be trained for an object detection task. Circuit 104 may calculate a training loss for input batch 120 based on the selected foreground bounding boxes and bounding box predictions. The calculated training loss may include a loss component for each image in input batch 120. Circuit 104 can train the student neural network 110 on an object detection task based on the training loss. Details about training the student neural network 110 are shown, for example, in Figure 3.

[0037]

[0040] Figure 2 is an exemplary block diagram of a system for training a student network for end-to-end semi-supervised object detection according to one embodiment of the present disclosure. The description of Figure 2 will be made in relation to the elements of Figure 1. Referring to Figure 2, a block diagram 200 of the system 102 of Figure 1 is shown. This system includes circuitry 104, memory 106, a teacher neural network 108, a student neural network 110, a display device 112, an input / output (I / O) device 202, a network interface 204, and an inference accelerator 206.

[0038]

[0041] The I / O device 202 may include suitable logic, circuitry, and / or interfaces that can be configured to receive one or more inputs and / or render information generated by system 102. The I / O device 202 may include various input and output devices and can be configured to communicate with different operating components of system 102. Examples of the I / O device 202 include, but are not limited to, a touchscreen, keyboard, mouse, joystick, microphone, and display device (such as display device 112).

[0039]

[0042] The network interface 204 may include preferred logic, circuitry, interfaces, and / or code that can be configured to establish communication between the system 102, the display device 112, and the server 114 via the communication network 116. The network interface 204 may be configured to implement known technologies that support wired or wireless communication. Examples of the network interface 204 include, but are not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identification module (SIM) card, and / or a local buffer.

[0040]

[0043] The network interface 204 can be configured to communicate with networks such as the Internet, intranets, and / or cellular telephone networks, wireless local area networks (WLANs), personal area networks, and / or metropolitan area networks (MANs) via offline and online wireless communication. Wireless communication may use any of several communication standards, protocols, and technologies, including Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), LTE, 5G New Radio, Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (IEEE 802.11, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or any other IEEE 802.11 protocol), Voice over Internet Protocol (VoIP), Wi-MAX, Internet of Things (IoT) technology, Machine Type Communication (MTC) technology, Email Protocol, Instant Messaging, and / or Short Message Service (SMS).

[0041]

[0044] The inference accelerator 206 may include suitable logic, circuitry, interfaces, and / or code that can be configured to act as a coprocessor for circuit 104 to accelerate computations related to the operation of the teacher neural network 108 and / or the student neural network 110. For example, the inference accelerator 206 can accelerate computations on system 102 so that the first and second results are produced in less time than would normally take without the inference accelerator 206. The inference accelerator 206 can implement various acceleration techniques, such as parallelizing some or all of the operation of the teacher neural network 108 and the student neural network 110. The inference accelerator 206 can be implemented as software, hardware, or a combination thereof. Implementations of the inference accelerator 206 include, but are not limited to, GPUs, tensor processing units (TPUs), neuromorphic chips, vision processing units (VPUs), field-programmable gate arrays (FPGAs), reduced instruction set computer (RISC) processors, application-specific integrated circuit (ASIC) processors, composite instruction set computer (CISC) processors, microcontrollers, and / or combinations thereof.

[0042]

[0045] Figure 3 shows an exemplary architecture of a teacher-student framework for end-to-end semi-supervised object detection according to one embodiment of the present disclosure. The description of Figure 3 will be made in relation to the elements of Figures 1 and 2. Referring to Figure 3, Figure 300 shows the teacher-student framework 302. The teacher-student framework 302 may include a teacher neural network 304 and a student neural network 306. The teacher neural network 304 may be a network that has been pre-trained for the object detection task, and the student neural network 306 may be an untrained network that may need to be trained for the object detection task. Referring to Figure 3, labeled images 308 and unlabeled images 310 are further shown.

[0043]

[0046] At any given moment in time, the circuit 104 can retrieve labeled images 308 and unlabeled images 310 from the image dataset 118. The image dataset 118 may include a set of labeled images 118A and a set of unlabeled images 118B. In one embodiment, the circuit 104 can randomly sample the image dataset 118 using a sample ratio to retrieve labeled images 308 and unlabeled images 310. For example, the sample ratio for the set of labeled images 118A may be 0.5, and the sample ratio for the set of unlabeled images 118B may be 0.2. As shown in the figure, for example, labeled image 308 may be an image of an animal 308A in a forest. Labelled image 308 includes a bounding box 308B around the animal 308A. The animal 308A may correspond to an object. In one embodiment, a student neural network 110 may need to be trained for the detection of the animal 308A.

[0044]

[0047] System 102 can generate an input batch 312 based on the extraction of labeled images 308 and unlabeled images 310 from the image dataset 118. The input batch 312 can be generated by applying a set of image transformations to the labeled images 308 and unlabeled images 310. In one embodiment, the set of image transformations may include a first subset of image transformations associated with a first data augmentation type. The set of image transformations may also include a second subset of image transformations associated with a second data augmentation type. The second data augmentation type may differ from the first data augmentation type. The first data augmentation type may mean a weak data augmentation technique, while the second data augmentation type may mean a strong data augmentation technique.

[0045]

[0048] As an example, and not an exhaustive list, a first subset of image transformations may include image flipping and image shifting operations. Image flipping can correspond to rotating an image along a horizontal or vertical axis. Image shifting can correspond to shifting pixels in an image to new positions within the image. A second subset of image transformations may include one or more of the following: image rotation, blurring, contrast change, shearing, masking one or more areas of an image, adding jitter, or adding random noise. Image rotation can correspond to rotating an image clockwise or counterclockwise by a specific angle. Blurring can correspond to adding a Gaussian blur to an image to reduce the sharpness of (one or more) objects within the image. Contrast change can correspond to changing the contrast of an image. Similarly, shearing can correspond to dividing an image into parts and separating those parts by a specific distance (with respect to pixel coordinates). Masking one or more areas of an image can correspond to hiding one or more areas of an image. The jitter addition operation can correspond to the addition of additional horizontal lines in the corresponding image. In some embodiments, a second subset of image transformations may include a first subset of image transformations.

[0046]

[0049] The input batch 312 may include a first unlabeled image 312A, a second unlabeled image 312B, and a labeled image 312C. 2 312 unlabeled images B This can be associated with the second data augmentation type (i.e., strong data augmentation). On the other hand, 1 312 unlabeled images AThe labeled image 312C can be associated with a first data augmentation type (i.e., weak data augmentation). Specifically, the first unlabeled image 312A can be generated by applying at least one operation from a first subset of image transformations to the unlabeled image 310. The second unlabeled image 312B can be generated by applying at least one operation from a second subset of image transformations to the unlabeled image 310. Similarly, the labeled image 312C can be generated by applying at least one operation from a first subset of image transformations to the labeled image 308. As shown in the figure, for example, the first unlabeled image 312A and the labeled image 312C can be generated by applying an image inversion operation to the unlabeled image 310 and the labeled image 308, respectively. The second unlabeled image 312B can be generated by applying an image inversion operation and a masking operation to a specific portion of the unlabeled image 310.

[0047]

[0050] When generating the input batch 312, the system 102 can be configured to apply the training neural network 304 to each image in the input batch 312. As described above, the training neural network 304 can be a network pre-trained for the object detection task. Based on applying the training neural network 304 to each image in the input batch 312, the system 102 can generate a first result for each input. The first result for an object (i.e., a player) in the first unlabeled image 312A of the input batch 312 may include a set of candidate bounding boxes for the object (and / or other foreground or background objects) and a set of scores corresponding to the set of candidate bounding boxes. Similarly, the first result for an object (i.e., a player) in the second unlabeled image 312B of the input batch 312 may include a first set of candidate bounding boxes for the object (and / or other foreground or background objects) and a first set of scores corresponding to the first set of candidate bounding boxes. Furthermore, the first result for an object (i.e., an animal 308A) in the third labeled image 312C of the input batch 312 may include a second set of candidate bounding boxes for the object (and / or other foreground or background objects) and a second set of scores corresponding to the second set of candidate bounding boxes.

[0048]

[0051] In one embodiment, the set of candidate bounding boxes may include foreground bounding boxes and background bounding boxes. The foreground bounding box can be treated as a prediction of a region of interest (ROI) that contains at least some or all of the objects in the first unlabeled image 312A (i.e., the object of interest). On the other hand, the background bounding box can be treated as a prediction of an ROI that contains background objects in the first unlabeled image 312A (i.e., objects (one or multiple) that should not be considered for object detection). Similar to the set of candidate bounding boxes, the set of scores may include foreground scores for the foreground bounding boxes in the set of candidate bounding boxes and background scores for the background bounding boxes in the set of candidate bounding boxes. The set of scores may correspond to confidence scores, each of which may indicate the probability that an object exists within the corresponding bounding box. For example, the foreground score may correspond to the probability that the corresponding foreground bounding box contains an object, and the background score may correspond to the probability that the corresponding background bounding box contains an object in the first unlabeled image 312A.

[0049]

[0052] In one embodiment, the system 102 may be further configured to provide a first label generator 314 with a set of candidate bounding boxes generated for an object and a set of scores corresponding to the set of candidate bounding boxes. The first label generator 314 may include non-maximum suppression (NMS) operations and adaptive threshold filters.

[0050]

[0053] The NMS operation can be applied to remove one or more redundant bounding boxes from a set of candidate bounding boxes. Therefore, system 102 can apply the NMS operation to a set of candidate bounding boxes to extract a subset of candidate bounding boxes from that set.

[0051]

[0054] In order to apply an adaptive threshold filter after applying the NMS operation, a threshold must be determined for each input image to the training neural network 108 to filter out bounding boxes that are not part of the foreground. Thus, system 102 can determine a threshold score based on the set of scores included in the first result. With respect to the scores, system 102 can calculate the average foreground score and the average background score. The average foreground score can be calculated by dividing the sum of the foreground scores by the number of foreground bounding boxes. Similarly, the average background score can be calculated by dividing the sum of the background scores by the number of background bounding boxes. System 102 can then generate a value by dividing the average foreground score by the average background score. The threshold score can be determined by applying a floor function to the generated value. As an example, the threshold score can be mathematically expressed using equation (1), which is given as follows: TIFF0007834271000001.tif35156 In the above equation, τ a This represents the threshold score, TIFF0007834271000002.tif17153 represents the number of foreground bounding boxes, TIFF0007834271000003.tif16153 represents the number of background border boxes, TIFF0007834271000004.tif17153 represents the sum of the foreground scores, TIFF0007834271000005.tif16153 represents the sum of background scores, γ represents the degree of underestimation of the class (or labeled image), where γ = 0.95.

[0052]

[0055] After determining the threshold score, the system 102 can apply an adaptive threshold filter. According to one embodiment, the adaptive threshold filter can be applied after applying a non-maximal suppression operation. Applying the adaptive threshold filter may include comparing the score associated with each subset of candidate bounding boxes with the determined threshold score. If the score is greater than the adaptive threshold, the corresponding bounding box may be included in the first subset of candidate bounding boxes. Otherwise, the corresponding bounding box may be deleted or discarded and not included in the first subset of candidate bounding boxes. In some embodiments, the first subset of candidate bounding boxes may be called pseudo-bounding boxes.

[0053]

[0056] It should be noted that an adaptive threshold filter can be introduced to help the teacher-student framework 302 retain better pseudo-bounding boxes. Such pseudo-bounding boxes can also be used in conjunction with a classification loss function.

[0054]

[0057] System 102 can be further configured to select a foreground bounding box. The foreground bounding box can be selected from a first subset of the extracted candidate bounding boxes. In one embodiment, the selected bounding box may have the highest score among all the candidate bounding boxes in the first subset of candidate bounding boxes. Specifically, the foreground bounding box may contain the largest portion or entire object among all the other bounding boxes in the first subset of candidate bounding boxes. The foreground bounding box can be used as ground truth for the set of foreground bounding boxes generated by the student neural network 110. Such foreground bounding boxes can be used to calculate the training loss for which the student neural network 110 needs to be trained.

[0055]

[0058] To select a foreground bounding box for the second unlabeled image 312B, system 102 can apply a second label generator 316 to the first result associated with the second unlabeled image 312B. The first label generator 314 may include a jitter-bagging module and an adaptive threshold filter. When executed by system 102, the jitter-bagging module can generate a set of jitter boxes by repeatedly applying a jitter operation to the selected bounding box. By applying a jitter operation to the selected bounding box, an refined bounding box can be obtained. As an example, the refined bounding box can be mathematically represented using equation (2), which is given as follows: TIFF0007834271000006.tif13156 In the above equation, TIFF0007834271000007.tif17153 represents an elaborated bounding box, b i This represents the selected bounding box, f jitter This represents jitter operation.

[0056]

[0059] System 102 can perform a bagging operation on a set of jitter boxes. Specifically, the bagging operation can be performed as part of the execution of the jitter bagging module of the second label generator 316. By performing the bagging operation, the jitter box having the largest area among the areas of each jitter box in the set can be selected. As an example, the selected jitter box can be mathematically expressed using equation (3) given as follows: TIFF0007834271000008.tif13156 In the above equation, TIFF0007834271000009.tif18153 represents the selected jitter box, TIFF0007834271000010.tif16153 represents an elaborated bounding box, f baggingThis represents the bagging operation.

[0057]

[0060] System 102 can further select a foreground boundary box by applying an adaptive threshold filter to the selected jitter box. The selected jitter box can be used to calculate a box regression loss, which can be part of the training loss for the input batch 312. Details about the adaptive threshold filter are provided above. Note that the threshold score associated with the adaptive threshold filter of the second label generator 316 may differ from the threshold score associated with the adaptive threshold filter of the first label generator 314, since the second label generator 316 can be associated with a second unlabeled image 312B, while the first label generator 314 is associated with a first unlabeled image 312A.

[0058]

[0061] To train the student neural network 306, images from the input batch 312 can be fed to the student neural network one at a time, and the loss for each can be calculated. According to one embodiment, the system 102 can be configured to produce a second result. The produced second result may include bounding box prediction of an object and can be produced by applying the student neural network 306 to the first unlabeled image 312A. As previously stated, the student neural network 306 can be an untrained network that may need to be trained for an object detection task. Similar to the production of the second result, the system 102 can also produce a third result by applying the student neural network 306 to the labeled image 312C of the input batch 312.

[0059]

[0062] System 102 can calculate the total supervised loss 318 of a first result associated with a labeled image 312C and a third result associated with a labeled image 312C by using a supervised loss function and a supervised regression loss function. In one embodiment, the total supervised loss 318 includes a supervised classification loss and a supervised box regression loss. The supervised classification loss can be associated with a supervised loss function, and the supervised box regression loss can be associated with a supervised box regression loss. As an example, the total supervised loss 318 of the first and third results can be mathematically expressed using equation (4), which is given as follows: TIFF0007834271000011.tif23156 In the above equation, L sup This represents the total supervised loss, TIFF0007834271000012.tif17153 represents the supervised loss function, TIFF0007834271000013.tif16153 represents the supervised regression loss function, N l This represents the number of labeled images, TIFF0007834271000014.tif16153 represents the sum of background scores, TIFF0007834271000015.tif17153 represents the i-th labeled image.

[0060]

[0063] In one embodiment, system 102 can calculate a first unsupervised loss 320 for a first result (i.e., generated for a first unlabeled image 312A of the input batch 312). System 102 can also calculate a second unsupervised loss 322 for a first result (i.e., generated for a second unlabeled image 312B of the input batch 312). Each of the first and second unsupervised losses 320 and 322 can be calculated using an unsupervised loss function. Furthermore, each of the first and second unsupervised losses 320 and 322 can include an unsupervised classification loss and an unsupervised box regression loss. For the first unsupervised loss 320, the unsupervised box regression loss can be generated after applying a second label generator 316 to the first result (generated for a first unlabeled image 312A of the input batch 312). As an example, the first unsupervised loss 320 can be mathematically expressed using equation (5), given as follows: TIFF0007834271000016.tif23156 In the above equation, TIFF0007834271000017.tif18153 represents the first unsupervised loss of 320, TIFF0007834271000018.tif16153 represents the unsupervised classification loss function, TIFF0007834271000019.tif16153 represents the unsupervised box regression loss function, N u This represents the number of unlabeled images. TIFF0007834271000020.tif16153 represents the sum of background scores, TIFF0007834271000021.tif17153 represents the i-th weakly expanded unlabeled image.

[0061]

[0064] For the second unsupervised loss 322, the unsupervised box regression loss can be generated after applying the second label generator 316 to the first result (generated for the second unlabeled image 312B of the input batch 312). As an example, the second unsupervised loss 322 can be mathematically expressed using equation (6), which is given as follows: TIFF0007834271000022.tif23156 In the above equation, TIFF0007834271000023.tif18153 represents the second unsupervised loss of 322, TIFF0007834271000024.tif17153 represents the unsupervised classification loss function, TIFF0007834271000025.tif16153 represents the unsupervised box regression loss function, N u This represents the number of unlabeled images. TIFF0007834271000026.tif17153 represents the i-th strongly expanded unlabeled image.

[0062]

[0065] In one embodiment, the unsupervised classification loss function used in the first unsupervised loss 320 and the second unsupervised loss 322, respectively, can be equal to the sum of the foreground classification loss, background classification loss, background similarity loss, and foreground-background dissimilarity loss. As an example, the unsupervised classification loss function can be expressed mathematically using equation (7), which is given as follows: TIFF0007834271000027.tif13156 In the above equation, TIFF0007834271000028.tif17153 represents the first unsupervised classification loss, TIFF0007834271000029.tif17153 represents the foreground classification loss function, TIFF0007834271000030.tif16153 represents the background classification loss function, TIFF0007834271000031.tif16153 represents the background similarity loss function, TIFF0007834271000032.tif16153 represents the foreground-background dissimilarity loss function.

[0063]

[0066] The foreground classification loss can help the teacher-student framework 302 classify the foreground bounding boxes (i.e., b fg ) generated by applying the student neural network 306 to the input batch 312 from the foreground bounding boxes generated by applying the teacher neural network 304 to the input batch 312. Specifically, the foreground classification loss can be associated with the second unlabeled image 312B of the input batch 312. As an example, the foreground classification loss can be mathematically represented using Equation (8) given as follows. TIFF0007834271000033.tif25156 In the above equation, TIFF0007834271000034.tif18* represents the foreground classification loss, TIFF0007834271000035.tif17153 represents the number of foreground bounding boxes generated by the student neural network 306, l cls represents the box classification loss, TIFF0007834271000036.tif16* represents the i-th foreground bounding box, β cls represents the set of candidate bounding boxes of the object.

[0064]

[0067] The background classification loss can indicate the reliability of each bounding box candidate generated by the student neural network 306. As an example, the background classification loss can be mathematically represented using Equation (9) given as follows. TIFF0007834271000037.tif26156 In the above equation, TIFF0007834271000038.tif17153 represents the background classification loss, It should be noted that there are some asterisks (*) in the translation where the original text seems to have some incorrect or incomplete information in the form of "18*" and "16*" which are likely to be errors in the original. If this is a specific format that needs to be maintained exactly, the translation would be as shown above. If it's an error, it might need to be corrected in the original text for a more accurate translation.TIFF0007834271000039.tif16153 represents the number of background bounding boxes generated by student neural network 306, l cls This represents the standard cross-entropy loss (or box classification loss), TIFF0007834271000040.tif22153 represents the j-th background bounding box, β cls This represents a set of candidate bounding boxes for an object. δ j This represents the confidence weighting coefficient associated with the j-th background bounding box.

[0065]

[0068] In one embodiment, the circuit 104 can be configured to calculate a reliability weighting coefficient. The reliability weighting coefficient can be based on a reliability score that can be associated with the j-th background bounding box, which is a background bounding box. As an example, the reliability weighting coefficient can be expressed mathematically using equation (10), which is given as follows: TIFF0007834271000041.tif21156 In the above equation, δ j This represents the reliability weighting coefficient, TIFF0007834271000042.tif17153 represents the number of background bounding boxes generated by student neural network 306, r j This represents the confidence score of the j-th background bounding box. TIFF0007834271000043.tif22153 represents the j-th background bounding box.

[0066]

[0069] A background similarity loss can be used to match the set of background scores generated by the teacher neural network 304 with the set of background scores generated by the student neural network 306. Such a loss may need to be minimized so that the set of bounding boxes generated by the teacher neural network 304 is reliably similar to the set of bounding boxes generated by the student neural network 306. As an example, the background similarity loss can be expressed mathematically using equation (11), which is given as follows: TIFF0007834271000044.tif25156 In the above equation, TIFF0007834271000045.tif15153 represents background similarity loss, TIFF0007834271000046.tif16153 represents the number of background bounding boxes generated by student neural network 306, β represents the control parameter, TIFF0007834271000047.tif16153 represents the i-th score obtained from the background bounding box generated using the training neural network 304. TIFF0007834271000048.tif17153 represents the i-th score obtained from the background bounding box generated using student neural network 306.

[0067]

[0070] Foreground-background dissimilarity loss can be used to separate the foreground bounding box and background bounding box generated using the student neural network 306. In one embodiment, the foreground-background dissimilarity loss can follow the principle of a relativistic average discriminator loss function and is used to match two different probability distributions. The foreground-background dissimilarity loss can provide the dissimilarity between the background score and the foreground score associated with the background bounding box and foreground bounding box (generated by the student neural network 306). As an example, the foreground-background dissimilarity loss can be mathematically expressed using equation (12), which is given as follows: TIFF0007834271000049.tif26156 In the above equation, TIFF0007834271000050.tif16153 represents the foreground-background dissimilarity loss. TIFF0007834271000051.tif17153 represents the number of candidate bounding boxes for the set of background bounding boxes generated by student neural network 306. TIFF0007834271000052.tif16153 represents the number of candidate bounding boxes for the set of foreground bounding boxes generated by student neural network 306. β represents the control parameter, TIFF0007834271000053.tif16153 represents the i-th score obtained from the foreground bounding box generated using student neural network 306, TIFF0007834271000054.tif16153 represents the j-th score obtained from the background bounding box generated using student neural network 306.

[0068]

[0071] As mentioned above, the first and second unsupervised losses include unsupervised box regression losses. Unsupervised box regression losses can provide the error between the predicted boundary box and the pseudo-boundary box. As an example, an unsupervised box regression loss can be mathematically expressed using equation (13), which is given as follows: TIFF0007834271000055.tif25156 In the above equation, TIFF0007834271000056.tif16153 represents the unsupervised box regression loss, TIFF0007834271000057.tif16153 represents the number of candidate bounding boxes for the set of foreground bounding boxes generated by student neural network 306. β reg This represents the bounding box, TIFF0007834271000058.tif16153 represents the i-th bounding box that can be assigned as the foreground bounding box, or the selected jitter box represented by equation (3), l reg This represents the mean absolute error loss or box regression loss.

[0069]

[0072] After calculating the individual losses, the system 102 can calculate the training loss 324 for the input batch 312. In one embodiment, the training loss 324 can be calculated based on foreground bounding boxes and bounding box predictions. In another embodiment, the training loss 324 can be calculated based on the calculation of the total supervised loss 318 of the labeled images 312C of the input batch. In yet another embodiment, the training loss 324 can be calculated based on the calculation of a first unsupervised loss and a second unsupervised loss. Mathematically, the calculated training loss 324 can be expressed using equation (14), which is given as follows: TIFF0007834271000059.tif12156 In the above equation, L TotalThis represents the calculated training loss of 324. L sup This represents the total supervised loss of 318, expressed in equation (4). TIFF0007834271000060.tif17153 represents the first unsupervised loss 320, expressed by equation (5). TIFF0007834271000061.tif16153 represents the second unsupervised loss 322, expressed by equation (6). α represents the control value of the unsupervised loss contribution in the calculated training loss 324.

[0070]

[0073] System 102 can be configured to train the student neural network 306 on an object detection task based on the calculated training loss 324. Specifically, the calculated training loss 324 can be used in a backpropagation operation to update the weight parameters of the student neural network 306. To train (or retrain) the student neural network 306, System 102 can update the weight parameters of the student neural network 306 using the calculated training loss 324.

[0071]

[0074] Based on the updated weight parameters of the student neural network 306, system 102 can update the weight parameters of the teacher neural network 304. In one embodiment, updating the weight parameters of the teacher neural network 304 may include performing an exponential moving average (EMA) operation. The EMA operation can be a type of moving average function that applies more weight to the most recent data points than to the oldest data points. Mathematically, the exponential moving average (EMA) can be expressed using equation (15), which is given as follows: TIFF0007834271000062.tif12156 In the above equation, w(t) ts This is the current timestamp t sThis represents the weights of the training neural network 304 in the above. w(s) ts This is the current timestamp t s This represents the weights of the student neural network 306 in the following: α represents a control parameter (for example, α = 0.99).

[0072]

[0075] To improve update efficiency, system 102 can also perform an exponentially adaptive differential moving average (E-ADMA) operation. The E-ADMA operation can be referenced via a normalization term that can be added to normalize the updates of the weights of the training neural network 304. Mathematically, the exponentially adaptive differential moving average (E-ADMA) can be expressed using equation (16), which is given as follows: TIFF0007834271000063.tif16158 In the above equation, w(t) ts This is the current timestamp t s This represents the weights of the training neural network 304 in the above. w(s) ts This is the current timestamp t s This represents the weights of the student neural network 306 in the following: α represents a control parameter (for example, α = 0.99).

[0073]

[0076] First, it should be noted that the teacher neural network 304 can be updated via EMA operations and fine-tuned based on the execution of E-ADMA operations at every j-th iteration. This can be done to make the teacher neural network 304 more resilient to sudden weight turbulences in the student neural network 306 caused by incorrect label predictions of the teacher neural network 304. Even if the student neural network 306 is supplied with incorrect labels, its impact on the teacher neural network 304 is mitigated by the update mechanism described above, provided by equations (15) and (16). The process of updating weights can be repeated until the student neural network 306 is trained for the object detection task. Specifically, the system 102 can be configured to iteratively train the student neural network 306 on the object detection task based on the training loss 324 (or until the training loss 324 for a batch is minimized or falls below a threshold).

[0074]

[0077] Figure 4 is a flowchart illustrating an exemplary method for training a student network for end-to-end semi-supervised object detection according to one embodiment of the present disclosure. The description of Figure 4 will be made in relation to the elements of Figures 1, 2 and 3. Referring to Figure 4, flowchart 400 is shown. The operation of flowchart 400 can begin at 402 and proceed to 404.

[0075]

[0078] In 404, labeled images 308 and unlabeled images 310 can be extracted from the image dataset 118. In at least one embodiment, circuit 104 can be configured to extract labeled images 308 and unlabeled images 310 from the image dataset 118. Details regarding the extraction of labeled and unlabeled images are shown, for example, in Figures 1 and 3.

[0076]

[0079] In 406, an input batch 312 can be generated by applying a set of image transformations to the labeled image 308 and the unlabeled image 310. In at least one embodiment, circuit 104 can be configured to generate the input batch 312 by applying a set of image transformations to the labeled image 308 and the unlabeled image 310. Details regarding the generation of the input batch 120 are shown, for example, in Figure 3.

[0077]

[0080] In 408, a first result for each image in the input batch 312 can be generated by applying the training neural network 304 to the input batch 312. The training neural network 304 can be a network pre-trained for an object detection task. The first result for an object in the first unlabeled image 312A of the input batch 312 may include a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes. In at least one embodiment, the circuit 104 can be configured to generate a first result for each image in the input batch 312 by applying the training neural network 304 to the input batch 312. The training neural network 304 is a network pre-trained for an object detection task, and the first result for an object in the first unlabeled image 312A of the input batch 312 includes a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes. Details of generating the first result are shown, for example, in Figure 3.

[0078]

[0081] In 410, a threshold score can be determined based on a set of scores. In at least one embodiment, circuit 104 can be configured to determine a threshold score based on a set of scores. Details regarding the determination of the threshold score are shown, for example, in Figure 3.

[0079]

[0082] In 412, a foreground bounding box can be selected from a set of candidate bounding boxes based on a threshold score. In at least one embodiment, circuit 104 can be configured to select a foreground bounding box from a set of candidate bounding boxes based on a threshold score. Details regarding the selection of the foreground bounding box are shown, for example, in Figure 3.

[0080]

[0083] In 414, a second result including an object bounding box prediction can be generated by applying the student neural network 306 to the first unlabeled image 312A. The student neural network 306 can be an untrained network to be trained for an object detection task. In at least one embodiment, the circuit 104 can be configured to generate a second result including an object bounding box prediction by applying the student neural network 306 to the first unlabeled image 312A. The student neural network 306 can be an untrained network to be trained for an object detection task. Details about the student neural network 306 are shown, for example, in Figure 3.

[0081]

[0084] In 416, the training loss 324 for the input batch 312 can be calculated based on the foreground bounding box and bounding box prediction. In at least one embodiment, the circuit 104 can be configured to calculate the training loss 324 for the input batch 312 based on the foreground bounding box and bounding box prediction. Details of the calculation of the training loss 324 are shown, for example, in Figure 3.

[0082]

[0085] At 418, the student neural network 306 can be retrained on the object detection task based on the training loss 324. In at least one embodiment, the circuit 104 can be configured to retrain the student neural network 306 on the object detection task based on the training loss 324. Control can then proceed to termination.

[0083]

[0086] Based on experimental data obtained after conducting several experiments during the drafting of this patent application, the disclosed student-teacher framework significantly outperformed state-of-the-art semi-supervised object detection methods (i.e., improved in terms of average accuracy) when run on known datasets such as the Microsoft® COCO dataset.

[0084]

[0087] Various embodiments of the present disclosure can provide a non-temporary computer-readable medium storing computer-executable instructions that a circuit or machine can execute to operate a system (e.g., system 102) for training a student network for end-to-end semi-supervised object detection. The computer-executable instructions can cause a machine and / or computer to perform operations including extracting labeled and unlabeled images from an image dataset (e.g., image dataset 118). The operations may further include generating an input batch (e.g., input batch 312) by applying a set of image transformations to the labeled and unlabeled images. The operations may further include generating a first result for each image in the input batch by applying a teacher neural network (e.g., teacher neural network 108) to the input batch. The teacher neural network may be a network pre-trained for an object detection task, and the first result for an object in a first unlabeled image of the input batch (e.g., first unlabeled image 312A) includes a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes. The operation may further include determining a threshold score based on a set of scores. The operation may further include selecting a foreground bounding box from a set of candidate bounding boxes based on the threshold score. The operation may further include generating a second result, including an object bounding box prediction, by applying a student neural network (e.g., student neural network 110) to a first unlabeled image. The student neural network may be an untrained network to be trained for an object detection task. The operation may further include calculating a training loss (e.g., training loss 324) for an input batch based on the foreground bounding box and the bounding box prediction, and training the student neural network on an object detection task based on the training loss.

[0085]

[0088] Specific embodiments of this disclosure can be found in systems and methods for training student networks for end-to-end semi-supervised object detection. Various embodiments of this disclosure can provide a system 102 which may include a circuit 104 and a memory 106. The circuit 104 may be configured to extract labeled images 308 and unlabeled images 310 from an image dataset 118. The circuit 104 may be further configured to extract labeled images 308 and unlabeled images 310 by randomly sampling the image dataset 118 using a sample ratio. The circuit 104 may be further configured to generate an input batch 312 by applying a set of image transformations to the labeled images 308 and unlabeled images 310. The set of image transformations may include a first subset of image transformations associated with a first data enhancement type and a second subset of image transformations associated with a second data enhancement type which may be different from the first data enhancement type.

[0086]

[0089] According to one embodiment, a first subset of image transformations may include image inversion and image shift operations, and a second subset of image transformations may include one or more of image rotation, blurring, contrast changes, shearing, masking of one or more areas of an image, jitter addition, or random noise addition.

[0087]

[0090] According to one embodiment, the generated input batch 312 may include a first unlabeled image 312A, a second unlabeled image 312B, and a labeled image 312C. The first unlabeled image 312A and the labeled image 312C may be associated with a first data extension type. The second unlabeled image 312B may also be associated with a first data extension type.

[0088]

[0091] According to one embodiment, the circuit 104 can be configured to generate a first result for each image in the input batch 312 by applying a teacher neural network 304 to the input batch 312. The teacher neural network 304 can be a network pre-trained for an object detection task, and the first result for objects in the first unlabeled image 312A of the input batch 312 can include a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes. The set of scores corresponding to the set of candidate bounding boxes can include a foreground score for the foreground bounding box of the set of candidate bounding boxes and a background score for the background bounding box of the set of candidate bounding boxes.

[0089]

[0092] According to one embodiment, the circuit 104 can be further configured to determine a threshold score based on a set of scores. According to one embodiment, the circuit 104 can be configured to calculate an average foreground score by dividing the sum of foreground scores by the number of foreground bounding boxes. The circuit 104 can be further configured to calculate an average background score by dividing the sum of background scores by the number of background bounding boxes. In one embodiment, the threshold score can be determined by generating a value by dividing the average foreground score by the average background score and applying a floor function to that value.

[0090]

[0093] According to one embodiment, the circuit 104 can be further configured to extract a subset of candidate bounding boxes from the set of candidate bounding boxes by applying a non-maximal suppression operation to the set of candidate bounding boxes. The circuit 104 can further select a foreground bounding box from the subset of candidate bounding boxes.

[0091]

[0094] According to one embodiment, the circuit 104 can be configured to select a bounding box from a subset of candidate bounding boxes. The circuit 104 can further generate a set of jitter boxes by repeatedly applying a jitter operation to the selected bounding box. The circuit 104 can further perform a bagging operation on the set of jitter boxes to select the jitter box having the largest area among the areas of each jitter box in the set.

[0092]

[0095] According to one embodiment, the circuit 104 may be further configured to calculate a first unsupervised loss 320 for a first result that can be generated for a first unlabeled image 312A. The circuit 104 may be further configured to calculate a second unsupervised loss 322 for a first result that can be generated for a second unlabeled image 312B of the input batch 312. The first and second unsupervised losses are each calculated by using an unsupervised loss function, which includes an unsupervised classification loss and an unsupervised box regression loss.

[0093]

[0096] According to one embodiment, the circuit 104 can generate a second result, which includes the prediction of the bounding box of an object. The second result can be generated by applying the student neural network 306 to the first unlabeled image 312A. The student neural network 306 can be an untrained network to be trained for the object detection task.

[0094]

[0097] According to one embodiment, the circuit 104 can be further configured to generate a third result by applying the student neural network 306 to the labeled images 312C of the input batch 312. The circuit 104 can be further configured to calculate the total supervised loss of the first result and the third result associated with the labeled images 312C by using a supervised loss function and a supervised regression loss function. The total supervised loss includes a supervised classification loss and a supervised box regression loss. The circuit 104 can be further configured to calculate a training loss 324 for the input batch 312 based on foreground bounding boxes and bounding box predictions. In another embodiment, the circuit 104 can be further configured to calculate a training loss based on the calculation of the total supervised loss of the labeled images 312C of the input batch 312. In another embodiment, the training loss 324 is calculated based on the calculation of a first unsupervised loss 320 and a second unsupervised loss 322. In another embodiment, the training loss 324 is further calculated based on a selected jitter box used to calculate the box regression loss, which is part of the training loss.

[0095]

[0098] In one embodiment, the circuit 104 can be further configured to train the student neural network 306 on an object detection task based on the training loss. In one embodiment, the circuit 104 can be further configured to update the weight parameters of the student neural network 306 using the training loss in order to train the student neural network 306. In another embodiment, the circuit 104 can be further configured to update the weight parameters of the teacher neural network 304 based on the updated weight parameters of the student neural network 306.

[0096]

[0099] According to one embodiment, updating the weight parameters of the teacher neural network 304 includes performing an exponential moving average (EMA) operation and an exponential adaptive differential moving average (E-ADMA) operation.

[0097]

[0100] This disclosure can be implemented in hardware form or in a combination of hardware and software. This disclosure can be implemented centrally within at least one computer system or in a distributed manner, where different elements are distributed across multiple interconnected computer systems. A computer system or other device adapted to perform the methods described herein may be suitable. The hardware and software combination may be a general-purpose computer system including a computer program that, when loaded and executed, can control the computer system to perform the methods described herein. This disclosure can also be implemented in hardware form, including part of an integrated circuit that also performs other functions.

[0098]

[0101] This disclosure includes all features that enable the implementation of the methods described herein and can be incorporated into a computer program product that can perform these methods when loaded onto a computer system. In this context, a computer program means any expression in any language, code, or notation of an instruction set intended to be executed by a system having information processing capabilities, either directly or after either a) conversion to another language, code, or notation, or b) reproduction in a different form of content.

[0099]

[0102] While this disclosure has been described with reference to several embodiments, those skilled in the art will understand that various modifications can be made and equivalents can be substituted without departing from the scope of this disclosure. Furthermore, many modifications can be made to adapt the teachings of this disclosure to specific circumstances or content without departing from the scope of this disclosure. Thus, this disclosure is not limited to the specific embodiments disclosed, but is intended to include all embodiments that fall within the claims. [Explanation of Symbols]

[0100] 100 Network Environment 102 System 104 circuits 106 memory 108 Teacher Neural Networks 110 Student Neural Networks 112 Display Devices 114 Servers 116 Communication Networks 118 image dataset 118A Set of labeled images 118B Set of unlabeled images 120 input batches 200 Block Diagram 202 Input / Output (I / O) Devices 204 Network Interfaces 206 Inference Accelerators Figure 300 302 Teacher-Student Framework 304 Teacher Neural Networks 306 Student Neural Networks 308 labeled images 308A Animal 308B Boundary Box 310 Unlabeled images 312 Input Batch 312A First unlabeled image 312B Second unlabeled image 312C Labeled image 314 First Label Generator 316 Second Label Generator 318 Total supervised loss 320 First unsupervised loss 322 Second unsupervised loss 324 Training Losses 400 flowcharts 402 start Extract labeled and unlabeled images from a 404 image dataset. 406 Generate input batches by applying a set of image transformations to labeled and unlabeled images. 408 Apply a trained neural network to the input batch to generate a first result for each image in the input batch. The trained neural network is a network pre-trained for the object detection task, and the first result for an object in the first unlabeled image of the input batch includes a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes. The threshold score is determined based on a set of 410 scores. Based on the threshold score, select the foreground bounding box from the set of candidate bounding boxes. 414 Applying a student neural network to the first unlabeled image generates a second result, including object bounding box prediction. The student neural network is an untrained network that should be trained for the object detection task. 416 Calculate the training loss for the input batch based on the foreground bounding box and bounding box prediction. 418 Train student neural networks on object detection tasks based on training loss.

Claims

1. It is a method, The steps include extracting labeled and unlabeled images from an image dataset, The steps include generating an input batch by applying a set of image transformations to the labeled and unlabeled images, A step of generating a first result for each image in the input batch by applying a training neural network to the input batch, The aforementioned teacher neural network is a network pre-trained for the object detection task. The first result for an object in the first unlabeled image of the input batch includes a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes. Steps and A step of determining a threshold score based on the aforementioned set of scores, The steps include selecting a foreground bounding box from the set of candidate bounding boxes based on the threshold score, A step of generating a second result, including bounding box prediction of the object, by applying a student neural network to the first unlabeled image, The aforementioned student neural network is an untrained network to be trained for the object detection task. Steps and A step of calculating the training loss for the input batch based on the foreground bounding box and the bounding box prediction, The steps include: retraining the student neural network on the object detection task based on the training loss; A method characterized by including the following.

2. The method according to claim 1, further comprising the step of randomly sampling the image dataset using a sample ratio to obtain labeled and unlabeled images.

3. The method according to claim 1, characterized in that the set of image transformations includes a first subset of image transformations associated with a first data enhancement type and a second subset of image transformations associated with a second data enhancement type different from the first data enhancement type.

4. The first subset of the image transformations includes image inversion and image shift operations, The second subset of the image transformation includes one or more of the following: image rotation, blurring, contrast change, shear, masking of one or more areas of the image, jitter addition, or addition of random noise. The method according to claim 3, characterized in that

5. The method according to claim 3, characterized in that the input batch includes the first unlabeled image and the labeled image associated with the first data augmentation type, and the second unlabeled image associated with the second data augmentation type.

6. The steps include: generating a third result by applying the student neural network to the labeled images of the input batch; A step of calculating the total supervised loss of the first result associated with the labeled image and the third result associated with the labeled image by using a supervised loss function and a supervised regression loss function, The aforementioned total supervised loss includes supervised classification loss and supervised box regression loss, The training loss is further calculated based on the calculation of the total supervised loss of the labeled images in the input batch. Steps and The method according to claim 1, further comprising:

7. A step of calculating a first unsupervised loss for the first result generated for the first unlabeled image, A step of calculating a second unsupervised loss for the first result generated for the second unlabeled image of the input batch, The first unsupervised loss and the second unsupervised loss are each calculated using an unsupervised loss function, and include an unsupervised classification loss and an unsupervised box regression loss. The training loss is calculated based on the calculation of the first unsupervised loss and the second unsupervised loss. Steps and The method according to claim 1, further comprising:

8. The method according to claim 7, characterized in that each of the first unsupervised loss and the second unsupervised loss is equal to the sum of the foreground classification loss, background classification loss, background similarity loss, and foreground-background dissimilarity loss.

9. The method according to claim 1, further comprising the step of applying a non-maximum suppression operation to the set of candidate bounding boxes to extract a subset of candidate bounding boxes from the set of candidate bounding boxes.

10. The method according to claim 9, characterized in that the foreground bounding box is selected from a subset of the candidate bounding boxes.

11. The steps include selecting a bounding box from a subset of candidate bounding boxes, The steps include generating a set of jitter boxes by repeatedly applying a jitter operation to the selected bounding box, A step of performing a bagging operation on the set of jitter boxes to select the jitter box having the largest area among the areas of each jitter box in the set, The selected jitter box is used to calculate the box regression loss, which is part of the training loss. Steps and The method according to claim 9, further comprising:

12. The aforementioned set of scores is The foreground score of the foreground bounding box of the set of candidate bounding boxes, The background score of the background bounding box in the set of candidate bounding boxes, including, The method according to claim 1, characterized in that

13. The steps include: calculating the average foreground score by dividing the sum of the foreground scores by the number of foreground bounding boxes; A step of calculating the average background score by dividing the sum of the aforementioned background scores by the number of the aforementioned background bounding boxes, The aforementioned threshold score is, The average foreground score is divided by the average background score to generate a value, Apply the floor function to the above value. Determined by, Steps and The method according to claim 12, further comprising:

14. The method according to claim 1, further comprising the step of updating the weight parameters of the student neural network using the training loss in order to retrain the student neural network.

15. The method according to claim 14, further comprising the step of updating the weight parameters of the teacher neural network based on the updated weight parameters of the student neural network.

16. The method according to claim 15, characterized in that the update of the weight parameters of the teacher neural network includes performing an exponential moving average (EMA) operation and performing an exponential adaptive difference moving average (E-ADMA) operation.

17. It is a system, Extracting labeled and unlabeled images from an image dataset, The input batch is generated by applying a set of image transformations to the labeled and unlabeled images. The process involves applying a training neural network to the input batch to generate a first result for each image in the input batch, The aforementioned teacher neural network is a network pre-trained for the object detection task. The first result for an object in the first unlabeled image of the input batch includes a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes. That thing, Determining a threshold score based on the aforementioned set of scores, Based on the threshold score, a foreground bounding box is selected from the set of candidate bounding boxes, Applying the student neural network to the first unlabeled image generates a second result, which includes the prediction of the bounding box of the object. The aforementioned student neural network is an untrained network to be trained for the object detection task. That thing, The training loss for the input batch is calculated based on the foreground bounding box and the bounding box prediction. Based on the aforementioned training loss, the student neural network is retrained on the object detection task, A circuit configured to perform the following: A system characterized by including

18. The system according to claim 17, wherein the circuit is further configured to update the weight parameters of the student neural network using the training loss in order to retrain the student neural network.

19. The circuit is further configured to update the weight parameters of the teacher neural network based on the updated weight parameters of the student neural network. The update of the weight parameters of the aforementioned training neural network includes performing an exponential moving average (EMA) operation and an exponentially adaptive differential moving average (E-ADMA) operation. The system according to claim 18, characterized in that

20. A non-temporary computer-readable storage medium configured to store instructions that cause a computer in the system to perform an action in response to execution, wherein the action is: Extracting labeled and unlabeled images from an image dataset, The input batch is generated by applying a set of image transformations to the labeled and unlabeled images. The process involves applying a training neural network to the input batch to generate a first result for each image in the input batch, The aforementioned teacher neural network is a network pre-trained for the object detection task. The first result for an object in the first unlabeled image of the input batch includes a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes. That thing, Determining a threshold score based on the aforementioned set of scores, Based on the threshold score, a foreground bounding box is selected from the set of candidate bounding boxes, Applying the student neural network to the first unlabeled image generates a second result, which includes the prediction of the bounding box of the object. The aforementioned student neural network is an untrained network to be trained for the object detection task. That thing, The training loss for the input batch is calculated based on the foreground bounding box and the bounding box prediction. Based on the aforementioned training loss, the student neural network is retrained on the object detection task, including, A non-temporary computer-readable storage medium characterized by the following features.

Images