Teaching a Student Network for End-to-End Semi-Supervised Object Detection

Abstract

A system and method for end-to-end semi-supervised object detection is provided. The system generates an input batch by taking labeled and unlabeled images from an image dataset and applying an image transform to the images. The system further generates a first result for each image of the input batch by applying a teacher neural network to the input batch. For objects in unlabeled images of the batch, the first result includes candidate bounding boxes and scores for the boxes. The system determines a threshold score based on the scores to select a foreground bounding box from the candidates. The system generates a second result by applying a student neural network to the unlabeled images and calculates a training loss for the input batch based on the foreground bounding boxes and the second result. The system trains the student neural network based on the training loss.

Description

[Technical field]

[0001] [CROSS REFERENCE / INCORPORATION BY REFERENCE TO RELATED APPLICATIONS]

[0001] This application claims the benefit of priority to U.S. patent application Ser. No. 18 / 159,492, filed in the United States Patent and Trademark Office on January 25, 2023, which claims priority to U.S. Provisional Patent Application No. 63 / 268,863, filed on March 4, 2022, the entire contents of which are incorporated herein by reference.

[0002]

[0002] Various embodiments of the present disclosure relate to neural networks and object detection. More specifically, various embodiments of the present disclosure relate to systems and methods for training a student network for end-to-end semi-supervised object detection. [Background technology]

[0003]

[0003] Advances in the fields of computer vision and artificial intelligence have led to the development of various kinds of neural networks (or models) for various applications, such as object detection. Typically, the goal of object detection is to identify and locate objects associated with a particular class label from still images or video data. The location of the object in the image can be indicated via a bounding box superimposed on the image. Recently, neural network models have been used for object detection. Such models are trained on a training dataset that can include multiple images associated with each object class. For example, if a neural network is to be trained for the detection of an object (e.g., a dog), the training dataset can include several images of the object, a class label, and the coordinates of a bounding box that can be placed around the object. In many cases, the images in the dataset are manually annotated by a human. For example, a human can label each image with a class label and annotate the image with a bounding box that includes the coordinates of the bounding box. In some cases, the number of examples of labeled images of a particular class may be low. In such cases, semi-supervised learning (SSL) may be used. SSL exploits the potential of unlabeled data to facilitate model learning when large annotated datasets are not available. Although SSL techniques have been successfully applied to image classification and object detection tasks, the complexity of object detector architecture design hinders the transfer of existing semi-supervised techniques from image classification to object detection.

[0004]

[0004] The limitations and disadvantages of conventional approaches will become apparent to one skilled in the art by comparing the described system with certain aspects of the present disclosure illustrated in the remaining portions of this application and with reference to the drawings. Summary of the Invention [Problem to be solved by the invention]

[0005]

[0005] Provided are systems and methods for training a student network for end-to-end semi-supervised object detection, as substantially shown in and / or described in connection with at least one figure, and more fully set forth in the claims.

[0006]

[0006] These and other features and advantages of the present disclosure can be understood by considering the following detailed description of the disclosure in conjunction with the accompanying drawings, in which like elements are designated with like reference numerals throughout. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 illustrates a network environment for training a student network for end-to-end semi-supervised object detection, according to one embodiment of the present disclosure. [Diagram 2] FIG. 1 is an example block diagram of a system for training a student network for end-to-end semi-supervised object detection, according to one embodiment of the present disclosure. [Diagram 3] FIG. 1 illustrates an example architecture of a teacher-student framework for end-to-end semi-supervised object detection, in accordance with one embodiment of the present disclosure. [Figure 4] 1 is a flowchart illustrating an exemplary method for training a student network for end-to-end semi-supervised object detection, according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008]

[0011] The implementations described below can be found in the disclosed system and method for training a student network for end-to-end semi-supervised object detection. Object detection can be defined as the task of detecting instances of a particular class of object in an image or video. In some cases, object detection further includes another task of generating a bounding box around the detected object. Object detection has applications in various fields such as autonomous vehicles, unmanned aerial vehicles (UAVs), mobile video surveillance, and image retrieval systems. As an example, object detection can be used in advanced driver assistance systems (ADAS) to enable vehicles to detect driving lanes or to detect pedestrians to improve road safety.

[0009]

[0012] The present disclosure provides a system that can retrieve labeled and unlabeled images from an image dataset and generate an input batch by applying a set of image transformations to the labeled and unlabeled images. The system can further generate results for each image of the input batch by applying a teacher neural network to the input batch. The teacher neural network can be a network pre-trained for an object detection task, and the results for an object in a first unlabeled image of the input batch can include a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes. For the object, the system can determine a threshold score based on the set of scores and can select a foreground bounding box from the set of candidate bounding boxes based on the threshold score. The system can further generate results including a bounding box prediction for the object by applying a student neural network to the first unlabeled image. The student neural network can be an untrained network to be trained for the object detection task. The system can calculate a training loss for the input batch based on the foreground bounding box and the bounding box predictions, and can retrain the student neural network on the object detection task based on the training loss.

[0010]

[0013] Recently, the task of object detection is accomplished by using a neural network model (or neural network models) that is pre-trained for the task of detecting one or more objects. To train the neural network model (or neural network models), a dataset including a number of training samples needs to be generated. Each training sample may include at least one image of each object of the one or more objects to be detected. Furthermore, each training sample may include a class label associated with the object in the corresponding image and the coordinates of a bounding box that includes the object in the corresponding image.

[0011]

[0014] To generate a dataset, a large number of image examples (e.g., thousands) of each object must be collected. Typically, collection is done manually from various sources and is a tedious task. In some cases, the number of labeled image examples of a particular object class may be low. In such cases, semi-supervised learning (SSL) may be used. SSL exploits the potential of unlabeled data to facilitate model learning when large annotated datasets are not available. Although SSL techniques have been successfully applied to image classification and object detection tasks, the complexity of object detector architecture design has hindered the migration of existing semi-supervised techniques from image classification to object detection.

[0012]

[0015] The present disclosure provides a neural network framework that can be based on semi-supervised learning for object detection. Specifically, semi-supervised learning uses unannotated (or unlabeled) data to facilitate model learning of a neural network with limited annotated (or labeled) data. The present disclosure can use a teacher-student framework that performs pseudo-labeling on unlabeled images and uses these pseudo-labels along with some labeled images per iteration to train a detector (i.e., a student neural network). The teacher-student framework includes a teacher neural network and a student neural network. The teacher neural network can be a network that is pre-trained for the object detection task, and the student neural network can be an untrained network that may need to be trained for the object detection task. The student neural network is trained for the object detection task based on a training loss calculated from the individual predictions of the teacher neural network and the student neural network.

[0013]

[0016] The present disclosure can be used in scenarios where the number of labeled images is limited (e.g., 1%) compared to the amount of unlabeled images in the dataset. Thus, the present disclosure can significantly reduce the human effort that may be required to generate a dataset and label the images in the dataset.

[0014]

[0017] This disclosure proposes a new update mechanism consisting of Exponential Moving Average (EMA) and Exponential Adaptive Difference Moving Average (E-ADMA) to update the teacher neural network from the student neural network during training time. This disclosure also provides two new loss functions for classification, called background similarity loss function and foreground-background dissimilarity loss function, which can leverage the background / foreground predictions of the teacher neural network and the student neural network to improve classification performance. This disclosure can also disclose a jitter-bagging module, which can help refine the bounding box predictions of neural networks. This disclosure also proposes a new adaptive threshold mechanism to obtain optimal bounding boxes for classification and regression tasks.

[0015]

[0018] FIG. 1 is a diagram illustrating a network environment for training a student network for end-to-end semi-supervised object detection, according to one embodiment of the present disclosure. With reference to FIG. 1, a diagram of a network environment 100 is shown. The network environment 100 includes a system 102. The system 102 includes a circuit 104 and a memory 106. The memory 106 can include, for example, a teacher neural network 108 and a student neural network 110. With reference to FIG. 1, a display device 112, a server 114, and a communication network 116 are also shown. As an example, an image dataset 118 and an input batch 120 are also shown.

[0016]

[0019] The system 102 may include suitable logic, circuitry, and interfaces that may be configured to train the student neural network 110 for an object detection task. The object detection task may be a semi-supervised machine learning task where the number of training examples may be small (e.g., less than 4-5 images) for a target object class on which the student neural network 110 needs to be trained. Examples of the system 102 may include, but are not limited to, a computing device, a mainframe machine, a server, a computer workstation, a gaming device, and / or a consumer electronics (CE) device.

[0017]

[0020] The circuitry 104 may include suitable logic, circuits, and interfaces that may be configured to execute program instructions associated with different operations to be performed by the system 102. The circuitry 104 may be implemented based on a number of processor technologies known in the art. Examples of processor technologies may include, but are not limited to, a central processing unit (CPU), an x86-based processor, a reduced instruction set computer (RISC) processor, an application specific integrated circuit (ASIC) processor, a complex instruction set computer (CISC) processor, a graphics processing unit (GPU), a co-processor (such as an inference accelerator or an artificial intelligence (AI) accelerator), and / or combinations thereof.

[0018]

[0021] The memory 106 may include suitable logic, circuitry, and / or interfaces that may be configured to store program instructions to be executed by the circuitry 104. The memory 106 may also store the teacher neural network 108 and the student neural network 110. In at least one embodiment, the memory 106 may also store input batches 120 and intermediate or final results obtained from the teacher neural network 108 and the student neural network 110. Example implementations of the memory 106 may include, but are not limited to, a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a hard disk drive (HDD), a solid state drive (SSD), a CPU cache, and / or a secure digital (SD) card.

[0019]

[0022] Each of the teacher neural network 108 and the student neural network 110 may be a computational network or system of artificial neurons that may be arranged in multiple layers. The multiple layers of the corresponding neural network may include an input layer, one or more hidden layers, and an output layer. Each layer of the multiple layers may include one or more nodes (or artificial neurons). The output of every node in the input layer may be coupled to at least one node in the hidden layer(s). Similarly, the input of each hidden layer may be coupled to the output of at least one node in another layer of the corresponding neural network. The output of each hidden layer may be coupled to the input of at least one node in another layer of the corresponding neural network. The node(s) in the final layer may receive inputs from at least one hidden layer and output a result. The number of layers and the number of nodes in each layer may be determined from the hyperparameters of the corresponding neural network. Such hyperparameters may be set before or after training the corresponding neural network with a training dataset.

[0020]

[0023] Each node of a corresponding neural network may correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) having a set of parameters that can be adjusted during training of the network. The set of parameters may include, for example, weight parameters, regularization parameters, etc. Each node may use the mathematical function to calculate an output based on one or more inputs from nodes in other layer(s) (e.g., previous layer(s)) of the corresponding neural network. All or some of the nodes of a corresponding neural network may correspond to the same or different mathematical functions.

[0021]

[0024] In training the corresponding neural network, one or more parameters of each node of the corresponding neural network can be updated based on whether the output of the final layer for a given input (from the training data set) matches the correct result based on a loss function for the corresponding neural network. The above process can be repeated for the same or different inputs until a minimum of the loss function is reached and the training error is minimized. Several training methods are known in the art, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, metaheuristics, etc.

[0022]

[0025] The teacher neural network 108 may include electronic data that may be implemented, for example, as a software component of an application executable on the system 102. The teacher neural network 108 may rely on libraries, external scripts, or other logic / instructions executed by a processing device, such as the circuit 104. The teacher neural network 108 may include code and routines configured to enable a computing device, such as the circuit 104, to perform one or more operations for object detection. Additionally or alternatively, the teacher neural network 108 may be implemented using hardware, including a processor, a microprocessor (e.g., performing or controlling the execution of one or more operations), a field programmable gate array (FPGA), a coprocessor (e.g., an inference accelerator), or an application specific integrated circuit (ASIC). Alternatively, in some embodiments, the neural network may be implemented using a combination of both hardware and software.

[0023]

[0026] Similar to teacher neural network 108, student neural network 110 may include electronic data that may be implemented, for example, as a software component of an application executable on system 102. Student neural network 110 may rely on libraries, external scripts, or other logic / instructions executed by a processing device, such as circuit 104. Student neural network 110 may include code and routines that may be configured to enable a computing device, such as circuit 104, to perform one or more operations for object detection. Additionally or alternatively, student neural network 110 may be implemented using hardware, including a processor, a microprocessor (e.g., performing or controlling the performance of one or more operations), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). Alternatively, in some embodiments, the neural network may be implemented using a combination of hardware and software.

[0024]

[0027] In one embodiment, the teacher neural network 108 may be a neural network that has been pre-trained for the object detection task, while the student neural network 110 may be an untrained network that may need to be trained for the object detection task.

[0025]

[0028] Examples of the teacher neural network 108 and the student neural network 110 may include, but are not limited to, a deep neural network (DNN), a convolutional neural network (CNN), a region-based convolutional neural network (R-CNN), Fast R-CNN, Faster R-CNN, an artificial neural network (ANN), a (You Only Look Once) YOLO network, a CNN+ANN, a fully connected neural network, and / or a combination of such networks. In certain embodiments, the teacher neural network 108 and / or the student neural network 110 may be based on a hybrid architecture of multiple deep neural networks (DNNs).

[0026]

[0029] The display device 112 may include suitable logic, circuitry, and interfaces that may be configured to display labeled and unlabeled images from the image dataset 118. In one embodiment, the display device 112 may be configured to display an input batch 120 that may be generated by applying a set of image transformations to the labeled and unlabeled images. The display device 112 may be utilized to view the execution status of operations related to training the student neural network 110. The display device 112 may be realized through a number of known technologies, such as, but not limited to, at least one of a liquid crystal display (LCD) display, a light emitting diode (LED) display, a plasma display, or an organic LED (OLED) display technology, or other display devices. According to one embodiment, the display device 112 may refer to a display screen of a head mounted device (HMD), a smart glasses device, a see-through display, a projected display, an electrochromic display, or a transparent display.

[0027]

[0030] The server 114 may include suitable logic, circuitry, interfaces, and / or code that may be configured to store the image dataset 118. The server 114 may also be configured to store the input batches 120 and results associated with the neural network. According to one embodiment, the server 114 may be implemented as a cloud server and may perform operations through web applications, cloud applications, HTTP requests, repository operations, file transfers, and the like. Other implementations of the server 114 may include, but are not limited to, a media server, a database server, a file server, a web server, an application server, a mainframe server, or a cloud computing server.

[0028]

[0031] In at least one embodiment, the servers 114 can be implemented as multiple distributed cloud-based resources using a number of techniques known to those of skill in the art. Those skilled in the art will appreciate that the scope of the present disclosure is not limited to implementing the servers 114 and the system 102 as two separate entities. In certain embodiments, the functionality of the servers 114 can be incorporated in whole or at least in part into the system 102 without departing from the scope of the present disclosure.

[0029]

[0032] The communication network 116 may include a communication medium through which the system 102, the display device 112, and the server 114 may communicate with each other. The communication network 116 may include one of a wired connection or a wireless connection. Examples of the communication network 116 may include, but are not limited to, the Internet, a cloud network, a cellular or wireless mobile network (such as Long-Term Evolution and 5G New Radio), a Wireless Fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), or a metropolitan area network (MAN). Various devices in the network environment 100 may be configured to connect to the communication network 116 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communications, wireless access points (AP), device-to-device communications, cellular communications protocols, and Bluetooth (BT) communications protocols.

[0030]

[0033] The image dataset 118 may correspond to a collection of one or more object instances and may include a set of labeled images 118A and a set of unlabeled images 118B. Each image in the image dataset 118 may include at least one object. The objects may be animate or inanimate objects. An animate object may have properties or characteristics of a living being, while an inanimate object may lack such characteristics. Examples of animate objects may include humans, birds, animals, etc. Examples of inanimate objects may include rocks, chairs, vehicles, etc.

[0031]

[0034] Each image in the set of labeled images 118A may be labeled (or annotated) with the name of the object contained in the corresponding image. For example, if an image is of a dog, the image may be labeled as a dog. In one embodiment, the images may be further labeled with the coordinates of a bounding box that contains the object. Each unlabeled image 118B may not include a label for the object(s) contained in the image.

[0032]

[0035] In operation, the circuit 104 can retrieve labeled and unlabeled images from the image dataset 118. To retrieve such images, the circuit 104 can randomly sample the image dataset 118 using a sample ratio. For example, the sample ratio can be set to 0.2 for unlabeled images and 0.5 for labeled images.

[0033]

[0036] The circuit 104 may generate an input batch (e.g., input batch 120) by applying a set of image transforms to the labeled and unlabeled images. In one embodiment, the set of image transforms may include a first subset of image transforms that may be associated with a first data augmentation type (i.e., weak data augmentation) and a second subset of image transforms that may be associated with a second data augmentation type (i.e., strong data augmentation). The second data augmentation type may be different from the first data augmentation type. For example, the first subset of image transforms may include an image flip operation, an image shift operation, etc. The second subset of image transforms may include one or more of an image rotation operation, a blur operation, a contrast change, a shear operation, a masking operation for one or more regions of an image, a jitter addition operation, adding random noise, etc.

[0034]

[0037] After the input batch 120 is generated, images from the input batch 120 are provided to the teacher neural network 108 and the student neural network 110. For each image of the input batch 120, the circuit 104 can generate a first result (i.e., a supervised or unsupervised object detection result). The first result can be generated by applying the teacher neural network 108 to the images of the input batch 120. As described above, the teacher neural network 108 can be a network pre-trained for an object detection task. For an object in a first unlabeled image of the input batch 120, the first result can include a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes. Each score in the set of scores can correspond to a confidence score associated with the corresponding candidate bounding box. In particular, the confidence score can indicate the likelihood that an object is present in the corresponding bounding box.

[0035]

[0038] According to one embodiment, the set of scores may include a foreground score and a background score. The foreground score may be for a foreground bounding box of the set of candidate bounding boxes, and the background score may be for a background bounding box of the set of candidate bounding boxes. The circuit 104 may determine a threshold score based on the foreground score and the background score. Details regarding the determination of the threshold score are shown, for example, in FIG. 3.

[0036]

[0039] A threshold selection may be performed to adaptively filter out bounding boxes that are not part of the foreground. In one embodiment, the system 102 may be configured to apply a non-maximum suppression operation to the set of candidate bounding boxes to extract a subset of candidate bounding boxes from the set of candidate bounding boxes. The circuit 104 may select a foreground bounding box from the subset of candidate bounding boxes based on the determined threshold score. The foreground bounding box may be selected for use as ground truth for the student neural network 110. The circuit 104 may generate a second result by applying the student neural network 110 to the first unlabeled image. The second result may include a bounding box prediction of the object. As mentioned above, the student neural network 110 may be an untrained network that may need to be trained for an object detection task. The circuit 104 may calculate a training loss for the input batch 120 based on the selected foreground bounding box and the bounding box prediction. The calculated training loss may include a loss component for each image of the input batch 120. The circuit 104 can train a student neural network 110 on the object detection task based on the training loss. Details on training the student neural network 110 are shown, for example, in FIG.

[0037]

[0040] Figure 2 is an example block diagram of a system for training a student network for end-to-end semi-supervised object detection, according to one embodiment of the present disclosure. The description of Figure 2 is provided with reference to the elements of Figure 1. Referring to Figure 2, a block diagram 200 of the system 102 of Figure 1 is shown. The system includes circuitry 104, memory 106, teacher neural network 108, student neural network 110, display device 112, input / output (I / O) device 202, network interface 204, and inference accelerator 206.

[0038]

[0041] The I / O devices 202 may include suitable logic, circuitry, and / or interfaces that may be configured to receive one or more inputs and / or render information generated by the system 102. The I / O devices 202 may include a variety of input and output devices and may be configured to communicate with different operational components of the system 102. Examples of the I / O devices 202 may include, but are not limited to, a touch screen, a keyboard, a mouse, a joystick, a microphone, and a display device (such as the display device 112).

[0039]

[0042] The network interface 204 may include suitable logic, circuitry, interfaces, and / or code that may be configured to establish communications between the system 102, the display device 112, and the server 114 over the communications network 116. The network interface 204 may be configured to implement known technologies that support wired or wireless communications. The network interface 204 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, and / or a local buffer.

[0040]

[0043] The network interface 204 may be configured to communicate with networks, such as the Internet, an intranet, and / or wireless networks, such as a cellular telephone network, a wireless local area network (WLAN), a personal area network, and / or a metropolitan area network (MAN), via offline and online wireless communications. The wireless communication may use any of a number of communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), LTE, 5G New Radio, Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (such as IEEE 802.11, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or any other IEEE 802.11 protocols), Voice over Internet Protocol (VoIP), Wi-MAX, Internet of Things (IoT) technologies, Machine Type Communications (MTC) technologies, email protocols, instant messaging, and / or Short Message Service (SMS).

[0041]

[0044] The inference accelerator 206 may include suitable logic, circuitry, interfaces, and / or code that may be configured to operate as a co-processor for the circuit 104 to accelerate computations associated with the operation of the teacher neural network 108 and / or the student neural network 110. For example, the inference accelerator 206 may accelerate computations on the system 102 such that a first result and a second result are generated in a shorter time than would normally be required without the inference accelerator 206. The inference accelerator 206 may implement various acceleration techniques, such as parallelizing some or all of the operations of the teacher neural network 108 and the student neural network 110. The inference accelerator 206 may be implemented as software, hardware, or a combination thereof. Examples of implementations of the inference accelerator 206 include, but are not limited to, a GPU, a tensor processing unit (TPU), a neuromorphic chip, a vision processing unit (VPU), a field programmable gate array (FPGA), a reduced instruction set computer (RISC) processor, an application specific integrated circuit (ASIC) processor, a complex instruction set computer (CISC) processor, a microcontroller, and / or combinations thereof.

[0042]

[0045] FIG. 3 illustrates an example architecture of a teacher-student framework for end-to-end semi-supervised object detection, according to one embodiment of the present disclosure. The description of FIG. 3 is provided in relation to the elements of FIG. 1 and FIG. 2. Referring to FIG. 3, a diagram 300 of a teacher-student framework 302 is shown. The teacher-student framework 302 may include a teacher neural network 304 and a student neural network 306. The teacher neural network 304 may be a pre-trained network for the object detection task, and the student neural network 306 may be an untrained network that may need to be trained for the object detection task. Referring to FIG. 3, a labeled image 308 and an unlabeled image 310 are further illustrated.

[0043]

[0046] At any instant in time, the circuit 104 can retrieve a labeled image 308 and an unlabeled image 310 from the image dataset 118. The image dataset 118 can include a set of labeled images 118A and a set of unlabeled images 118B. In one embodiment, the circuit 104 can randomly sample the image dataset 118 using a sample ratio to retrieve the labeled image 308 and the unlabeled image 310. For example, the sample ratio of the set of labeled images 118A can be 0.5 and the sample ratio of the set of unlabeled images 118B can be 0.2. As shown, for example, the labeled image 308 can be an image of an animal 308A in a forest. The labeled image 308 includes a bounding box 308B around the animal 308A. The animal 308A can correspond to an object. In one embodiment, the student neural network 110 can need to be trained for the detection of the animal 308A.

[0044]

[0047] The system 102 can generate an input batch 312 based on retrieval of labeled images 308 and unlabeled images 310 from the image dataset 118. The input batch 312 can be generated by applying a set of image transformations to the labeled images 308 and the unlabeled images 310. In one embodiment, the set of image transformations can include a first subset of image transformations associated with a first data augmentation type. The set of image transformations can also include a second subset of image transformations associated with a second data augmentation type. The second data augmentation type can be different from the first data augmentation type. The first data augmentation type can refer to a weak data augmentation technique, while the second data augmentation type can refer to a strong data augmentation technique.

[0045]

[0048] By way of example and not limitation, the first subset of image transformations may include an image flip operation and an image shift operation. The image flip operation may correspond to an operation of rotating an image along a horizontal or vertical axis. The image shift operation may correspond to an operation of shifting pixels of an image to a new position within the image. The second subset of image transformations may include one or more of an image rotation operation, a blur operation, a contrast change, a shear operation, a masking operation for one or more regions of an image, an operation of adding jitter, or an operation of adding random noise. The image rotation operation may correspond to an operation of rotating an image by a particular angle in a clockwise or counterclockwise direction. The blur operation may correspond to adding a Gaussian blur to an image to reduce the sharpness of an object(s) in the image. The contrast change may correspond to an operation of modifying the contrast of the image. Similarly, the shear operation may correspond to an operation of dividing an image into parts and separating the parts by a particular distance (in terms of pixel coordinates). The masking operation for one or more regions of an image may correspond to an operation of hiding one or more regions of an image. The jitter addition operation may correspond to adding an additional horizontal line in the corresponding image. In some embodiments, the second subset of image transforms may include the first subset of image transforms.

[0046]

[0049] The input batch 312 may include a first unlabeled image 312A, a second unlabeled image 312B, and a labeled image 312C. The first unlabeled image 312A may be associated with a second data augmentation type (i.e., strong data augmentation). Meanwhile, the second unlabeled image 312B and the labeled image 312C may be associated with a first data augmentation type (i.e., weak data augmentation). Specifically, the first unlabeled image 312A may be generated based on applying at least one operation of a first subset of image transforms to the unlabeled image 310. The second unlabeled image 312B may be generated based on applying at least one operation of a second subset of image transforms to the unlabeled image 310. Similarly, the labeled image 312C may be generated based on applying at least one operation of a first subset of image transforms to the labeled image 308. As shown, for example, a first unlabeled image 312A and a labeled image 312C may be generated by applying an image inversion operation to the unlabeled image 310 and the labeled image 308, respectively. A second unlabeled image 312B may be generated by applying an image inversion operation and a masking operation to certain portions of the unlabeled image 310.

[0047]

[0050] Upon generation of the input batch 312, the system 102 may be configured to apply a teacher neural network 304 to each image of the input batch 312. As described above, the teacher neural network 304 may be a network pre-trained for an object detection task. The system 102 may generate a first result for each input based on applying the teacher neural network 304 to each image of the input batch 312. The first result for an object (i.e., a player) in a first unlabeled image 312A of the input batch 312 may include a set of candidate bounding boxes for the object (and / or other foreground or background objects) and a set of scores corresponding to the set of candidate bounding boxes. Similarly, the first result for an object (i.e., a player) in a second unlabeled image 312B of the input batch 312 may include a first set of candidate bounding boxes for the object (and / or other foreground or background objects) and a first set of scores corresponding to the first set of candidate bounding boxes. The first result for the object (i.e., animal 308A) in the third labeled image 312C of the input batch 312 may also include a second set of candidate bounding boxes for the object (and / or other foreground or background objects) and a second set of scores corresponding to the second set of candidate bounding boxes.

[0048]

[0051] In one embodiment, the set of candidate bounding boxes may include a foreground bounding box and a background bounding box. The foreground bounding box may be treated as a prediction of a region of interest (ROI) that includes at least a portion of an object or an entire object (i.e., an object of interest) in the first unlabeled image 312A. Meanwhile, the background bounding box may be treated as a prediction of a ROI that includes background objects (i.e., object(s) that should not be considered for object detection) in the first unlabeled image 312A. Similar to the set of candidate bounding boxes, the set of scores may include a foreground score of a foreground bounding box of the set of candidate bounding boxes and a background score of a background bounding box of the set of candidate bounding boxes. The set of scores may correspond to confidence scores, each of which may indicate a probability that an object is present in the corresponding bounding box. For example, the foreground score may correspond to a probability that the corresponding foreground bounding box includes an object, and the background score may correspond to a probability that the corresponding background bounding box includes an object in the first unlabeled image 312A.

[0049]

[0052] In one embodiment, the system 102 may be further configured to provide the set of generated candidate bounding boxes for the object and the set of scores corresponding to the set of candidate bounding boxes to a first label generator 314. The first label generator 314 may include a non-maximum suppression (NMS) operation and an adaptive threshold filter.

[0050]

[0053] An NMS operation may be applied to remove one or more redundant bounding boxes from the set of candidate bounding boxes. Thus, the system 102 may apply the NMS operation to the set of candidate bounding boxes to extract a subset of candidate bounding boxes from the set of candidate bounding boxes.

[0051]

[0054] In order to apply the adaptive threshold filter after application of the NMS operation, a threshold must be determined for each input image to the teacher neural network 108 to filter out bounding boxes that are not part of the foreground. Thus, the system 102 can determine the threshold score based on the set of scores included in the first result. In terms of the scores, the system 102 can calculate an average foreground score and an average background score. The average foreground score can be calculated by dividing the sum of the foreground scores by the number of foreground bounding boxes. Similarly, the average background score can be calculated by dividing the sum of the background scores by the number of background bounding boxes. The system 102 can then divide the average foreground score by the average background score to generate a value. A floor function can be applied to the generated value to determine the threshold score. As an example, the threshold score can be mathematically represented using equation (1) given as follows: TIFF2025512701000001.tif35156 In the above formula, τ a represents the threshold score, TIFF2025512701000002.tif17153 represents the number of foreground bounding boxes, TIFF2025512701000003.tif16153 represents the number of background bounding boxes, TIFF2025512701000004.tif17153 represents the sum of the foreground scores, TIFF2025512701000005.tif16153 represents the total background score, γ represents the degree of under-represented classes (or labeled images), and γ=0.95.

[0052]

[0055] After determining the threshold score, the system 102 may apply an adaptive threshold filter. According to one embodiment, the adaptive threshold filter may be applied after applying the non-maximum suppression operation. The application of the adaptive threshold filter may include comparing a score associated with each of the subset of candidate bounding boxes to the determined threshold score. If the score is greater than the adaptive threshold, the corresponding bounding box may be included in the first subset of candidate bounding boxes. If not, the corresponding bounding box may be removed or discarded and not included in the first subset of candidate bounding boxes. In some embodiments, the first subset of candidate bounding boxes may be referred to as pseudo bounding boxes.

[0053]

[0056] Note that an adaptive threshold filter can be introduced to help the teacher-student framework 302 retain better pseudo bounding boxes, which can also be further used with the classification loss function.

[0054]

[0057] The system 102 may further be configured to select a foreground bounding box. The foreground bounding box may be selected from a first subset of the extracted candidate bounding boxes. In one embodiment, the selected bounding box may have the largest score among all the candidate bounding boxes of the first subset of the candidate bounding boxes. Specifically, the foreground bounding box may include the largest portion or the entire object among all the other bounding boxes of the first subset of the candidate bounding boxes. The foreground bounding box may be used as a ground truth for the set of foreground bounding boxes generated by the student neural network 110. Such foreground bounding boxes may be used to calculate a training loss on which the student neural network 110 needs to be trained.

[0055]

[0058] To select a foreground bounding box for the second unlabeled image 312B, the system 102 can apply a second label generator 316 to the first result associated with the second unlabeled image 312B. The first label generator 314 can include a jitter bagging module and an adaptive threshold filter. The jitter bagging module, when executed by the system 102, can apply a jitter operation to the selected bounding boxes multiple times to generate a set of jitter boxes. The jitter operation can be applied to the selected bounding boxes to obtain refined bounding boxes. As an example, the refined bounding boxes can be mathematically represented using equation (2) given as follows: TIFF2025512701000006.tif13156 In the above formula, TIFF2025512701000007.tif17153 represents the refined bounding box, b i represents the selected bounding box, f jitter represents the jitter operation.

[0056]

[0059] The system 102 may perform a bagging operation on the set of jitter boxes. Specifically, the bagging operation may be performed as part of the execution of the jitter bagging module of the second label generator 316. The bagging operation may be performed to select a jitter box having a maximum area among the respective areas of the set of jitter boxes. As an example, the selected jitter box may be mathematically represented using equation (3) given as follows: TIFF2025512701000008.tif13156 In the above formula, TIFF2025512701000009.tif18153 represents the selected jitter box, TIFF2025512701000010.tif16153 represents the refined bounding box, f baggingrepresents the bagging operation.

[0057]

[0060] The system 102 may further apply an adaptive threshold filter to the selected jitter box to select a foreground bounding box. The selected jitter box may be used to calculate a box regression loss, which may be part of the training loss for the input batch 312. Details regarding the adaptive threshold filter are provided above. Note that since the second label generator 316 may be associated with the second unlabeled image 312B, but the first label generator 314 is associated with the first unlabeled image 312A, the threshold score associated with the adaptive threshold filter of the second label generator 316 may be different from the threshold score associated with the adaptive threshold filter of the first label generator 314.

[0058]

[0061] To train the student neural network 306, images from the input batch 312 can be fed to the student neural network one at a time and a respective loss can be calculated. According to an embodiment, the system 102 can be configured to generate a second result. The generated second result can include bounding box predictions of objects and can be generated by applying the student neural network 306 to the first unlabeled image 312A. As mentioned above, the student neural network 306 can be an untrained network that may need to be trained for an object detection task. Similar to generating the second result, the system 102 can also generate a third result by applying the student neural network 306 to the labeled image 312C of the input batch 312.

[0059]

[0062] The system 102 can calculate a total supervised loss 318 of the first result associated with the labeled image 312C and the third result associated with the labeled image 312C by using the supervised loss function and the supervised regression loss function. In one embodiment, the total supervised loss 318 includes a supervised classification loss and a supervised box regression loss. The supervised classification loss can be associated with the supervised loss function, and the supervised box regression loss can be associated with the supervised box regression loss. As an example, the total supervised loss 318 of the first result and the third result can be mathematically represented using equation (4) given as follows: TIFF2025512701000011.tif23156 In the above formula, L sup represents the total supervised loss, TIFF2025512701000012.tif17153 represents the supervised loss function, TIFF2025512701000013.tif16153 represents the supervised regression loss function, N l represents the number of labeled images, TIFF2025512701000014.tif16153 represents the total background score, TIFF2025512701000015.tif17153 represents the i-th labeled image.

[0060]

[0063] In one embodiment, the system 102 may calculate a first unsupervised loss 320 for the first result (i.e., generated for the first unlabeled image 312A of the input batch 312). The system 102 may also calculate a second unsupervised loss 322 for the first result (i.e., generated for the second unlabeled image 312B of the input batch 312). Each of the first unsupervised loss 320 and the second unsupervised loss 322 may be calculated by using an unsupervised loss function. Each of the first unsupervised loss 320 and the second unsupervised loss 322 may include an unsupervised classification loss and an unsupervised box regression loss. For the first unsupervised loss 320, the unsupervised box regression loss may be generated after applying the second label generator 316 to the first result (generated for the first unlabeled image 312A of the input batch 312). As an example, the first unsupervised loss 320 may be mathematically represented using equation (5) given as follows: TIFF2025512701000016.tif23156 In the above formula, TIFF2025512701000017.tif18153 represents the first unsupervised loss 320, TIFF2025512701000018.tif16153 represents the unsupervised classification loss function, TIFF2025512701000019.tif16153 represents the unsupervised box regression loss function, N u represents the number of unlabeled images, TIFF2025512701000020.tif16153 represents the sum of the background scores, TIFF2025512701000021.tif17153 represents the i-th weakly augmented unlabeled image.

[0061]

[0064] For the second unsupervised loss 322, an unsupervised box regression loss can be generated after applying the second label generator 316 to the first result (generated for the second unlabeled image 312B of the input batch 312). As an example, the second unsupervised loss 322 can be mathematically represented using equation (6) given as follows: TIFF2025512701000022.tif23156 In the above formula, TIFF2025512701000023.tif18153 represents the second unsupervised loss322, TIFF2025512701000024.tif17153 represents the unsupervised classification loss function, TIFF2025512701000025.tif16153 represents the unsupervised box regression loss function, N u represents the number of unlabeled images, TIFF2025512701000026.tif17153 represents the i-th strongly augmented unlabeled image.

[0062]

[0065] In one embodiment, the unsupervised classification loss function used in each of the first unsupervised loss 320 and the second unsupervised loss 322 may be equal to the sum of a foreground classification loss, a background classification loss, a background similarity loss, and a foreground-background dissimilarity loss. As an example, the unsupervised classification loss function may be mathematically represented using equation (7) given as follows: TIFF2025512701000027.tif13156 In the above formula, TIFF2025512701000028.tif17153 represents the first unsupervised classification loss, TIFF2025512701000029.tif17153 represents the foreground classification loss function, TIFF2025512701000030.tif16153 represents the background classification loss function, TIFF2025512701000031.tif16153 represents the background similarity loss function, TIFF2025512701000032.tif16153 represents the foreground-background dissimilarity loss function.

[0063]

[0066] The foreground classification loss is calculated by the teacher-student framework 302 calculating the foreground bounding boxes (i.e., b fg ) from the foreground bounding boxes generated by applying the teacher neural network 304 to the input batch 312. Specifically, the foreground classification loss can be associated with the second unlabeled image 312B of the input batch 312. As an example, the foreground classification loss can be mathematically represented using equation (8) given as follows: TIFF2025512701000033.tif25156 In the above formula, TIFF2025512701000034.tif18153 represents the foreground classification loss, TIFF2025512701000035.tif17153 represents the number of foreground bounding boxes generated by the student neural network 306, l cls denotes the box classification loss, TIFF2025512701000036.tif16153 represents the i-th foreground bounding box, β cls represents the set of candidate bounding boxes for the object.

[0064]

[0067] The background classification loss may indicate the reliability of each bounding box candidate generated by the student neural network 306. As an example, the background classification loss may be mathematically expressed using equation (9) given as follows: TIFF2025512701000037.tif26156 In the above formula, TIFF2025512701000038.tif17153 represents the background classification loss, TIFF2025512701000039.tif16153 represents the number of background bounding boxes generated by the student neural network 306, and l cls denotes the standard cross-entropy loss (or box classification loss), TIFF2025512701000040.tif22153 represents the jth background bounding box, β cls represents the set of candidate bounding boxes for an object, δ j represents the confidence weighting factor associated with the jth background bounding box.

[0065]

[0068] In one embodiment, the circuit 104 can be configured to calculate a reliability weighting factor. The reliability weighting factor can be based on a reliability score that can be associated with the jth background bounding box, which is the background bounding box. As an example, the reliability weighting factor can be mathematically represented using Equation (10) given as follows: TIFF2025512701000041.tif21156 In the above formula, δ j represents the reliability weighting coefficient, TIFF2025512701000042.tif17153 represents the number of background bounding boxes generated by the student neural network 306, r j represents the confidence score of the jth background bounding box, TIFF2025512701000043.tif22153 represents the jth background bounding box.

[0066]

[0069] A background similarity loss may be used to match the set of background scores generated by the teacher neural network 304 with the set of background scores generated by the student neural network 306. Such a loss may need to be minimized to ensure that the set of bounding boxes generated by the teacher neural network 304 is similar to the set of bounding boxes generated by the student neural network 306. As an example, the background similarity loss may be mathematically expressed using equation (11) given as follows: TIFF2025512701000044.tif25156 In the above formula, TIFF2025512701000045.tif15153 represents background similarity loss, TIFF2025512701000046.tif16153 represents the number of background bounding boxes generated by the student neural network 306, β represents a control parameter, TIFF2025512701000047.tif16153 represents the i-th score obtained from the background bounding box generated using the teacher neural network 304, TIFF2025512701000048.tif17153 represents the i-th score obtained from the background bounding box generated using the student neural network 306.

[0067]

[0070] The foreground-background dissimilarity loss can be used to separate the foreground and background bounding boxes generated using the student neural network 306. In one embodiment, the foreground-background dissimilarity loss can follow the principle of a relativistic average discriminator loss function and is used to match two different probability distributions. The foreground-background dissimilarity loss can provide a dissimilarity between the background bounding box (generated by the student neural network 306) and the background and foreground scores associated with the foreground bounding box. As an example, the foreground-background dissimilarity loss can be mathematically represented using Equation (12) given as follows: TIFF2025512701000049.tif26156 In the above formula, TIFF2025512701000050.tif16153 represents the foreground-background dissimilarity loss, TIFF2025512701000051.tif17153 represents the number of candidate bounding boxes for the set of background bounding boxes generated by the student neural network 306; TIFF2025512701000052.tif16153 represents the number of candidate bounding boxes for the set of foreground bounding boxes generated by the student neural network 306; β represents a control parameter, TIFF2025512701000053.tif16153 represents the i-th score obtained from the foreground bounding box generated using the student neural network 306; TIFF2025512701000054.tif16153 represents the jth score obtained from the background bounding box generated using the student neural network 306.

[0068]

[0071] As mentioned above, the first unsupervised loss and the second unsupervised loss include an unsupervised box regression loss. The unsupervised box regression loss can provide an error between a predicted bounding box and a pseudo bounding box. As an example, the unsupervised box regression loss can be mathematically expressed using Equation (13) given as follows: TIFF2025512701000055.tif25156 In the above formula, TIFF2025512701000056.tif16153 represents the unsupervised box regression loss, TIFF2025512701000057.tif16153 represents the number of candidate bounding boxes for the set of foreground bounding boxes generated by the student neural network 306; β reg represents the bounding box, TIFF2025512701000058.tif16153 represents the i-th bounding box that can be assigned as the foreground bounding box, or the selected jitter box represented by equation (3), l reg represents the mean absolute error loss or box regression loss.

[0069]

[0072] After computing the individual losses, the system 102 may compute a training loss 324 for the input batch 312. In one embodiment, the training loss 324 may be computed based on the foreground bounding boxes and the bounding box predictions. In another embodiment, the training loss 324 may be computed based on computing the total supervised loss 318 for the labeled images 312C of the input batch. In another embodiment, the training loss 324 may be computed based on computing the first unsupervised loss and the second unsupervised loss. Mathematically, the computed training loss 324 may be expressed using equation (14) given as follows: TIFF2025512701000059.tif12156 In the above formula, L Totalrepresents the computed training loss,324, L sup represents the total supervised loss,318,expressed in Equation (4), TIFF2025512701000060.tif17153 represents the first unsupervised loss 320 expressed by equation (5), TIFF2025512701000061.tif16153 represents the second unsupervised loss 322 expressed by equation (6), α represents a value that controls the contribution of the unsupervised loss in the computed training loss 324.

[0070]

[0073] The system 102 may be configured to train the student neural network 306 on the object detection task based on the calculated training loss 324. Specifically, the calculated training loss 324 may be used in a backpropagation operation to update weight parameters of the student neural network 306. To train (or retrain) the student neural network 306, the system 102 may use the calculated training loss 324 to update the weight parameters of the student neural network 306.

[0071]

[0074] Based on the updated weight parameters of the student neural network 306, the system 102 may update the weight parameters of the teacher neural network 304. In one embodiment, updating the weight parameters of the teacher neural network 304 may include performing an exponential moving average (EMA) operation. The EMA operation may be a type of moving average function that may apply more weight to the most recent data points than older data points. Mathematically, the exponential moving average (EMA) may be represented using equation (15) given as follows: TIFF2025512701000062.tif12156 In the above formula, w(t) ts is the current timestamp t srepresents the weights of the teacher neural network 304 in w(s) ts is the current timestamp t s represents the weights of the student neural network 306 in α represents a control parameter (eg, α=0.99).

[0072]

[0075] To increase update efficiency, the system 102 may also perform an exponential adaptive differential moving average (E-ADMA) operation. The E-ADMA operation may be referred to via a regularization term that may be added to regularize the weight updates of the teacher neural network 304. Mathematically, the exponential adaptive differential moving average (E-ADMA) may be represented by using equation (16) given as follows: TIFF2025512701000063.tif16158 In the above formula, w(t) ts is the current timestamp t s represents the weights of the teacher neural network 304 in w(s) ts is the current timestamp t s represents the weights of the student neural network 306 in α represents a control parameter (eg, α=0.99).

[0073]

[0076] First, note that the teacher neural network 304 can be updated via the EMA operation and fine-tuned based on performing the E-ADMA operation every j-th iteration. This can be done to make the teacher neural network 304 more resistant to sudden weight turbulence in the student neural network 306 caused by an incorrect label prediction of the teacher neural network 304. Even if the student neural network 306 is fed with an incorrect label, its effect on the teacher neural network 304 is mitigated by the above-mentioned update mechanism provided by equations (15) and (16). The process of updating the weights can be performed iteratively until the student neural network 306 is trained for the object detection task. Specifically, the system 102 can be configured to iteratively train the student neural network 306 on the object detection task based on the training loss 324 (or until the training loss 324 for the batch is minimized or falls below a threshold).

[0074]

[0077]

[0023] Figure 4 is a flow chart illustrating an exemplary method for training a student network for end-to-end semi-supervised object detection, according to one embodiment of the present disclosure. Figure 4 is described with reference to elements of Figures 1, 2, and 3. With reference to Figure 4, a flow chart 400 is shown. Operation of flow chart 400 may begin at 402 and proceed to 404.

[0075]

[0078] At 404, labeled images 308 and unlabeled images 310 may be retrieved from the image dataset 118. In at least one embodiment, the circuit 104 may be configured to retrieve labeled images 308 and unlabeled images 310 from the image dataset 118. Details regarding retrieving labeled and unlabeled images are shown, for example, in Figures 1 and 3.

[0076]

[0079] At 406, an input batch 312 may be generated by applying a set of image transformations to the labeled images 308 and the unlabeled images 310. In at least one embodiment, the circuit 104 may be configured to generate the input batch 312 by applying a set of image transformations to the labeled images 308 and the unlabeled images 310. Details regarding the generation of the input batch 120 are shown, for example, in FIG.

[0077]

[0080] At 408, a first result for each image of the input batch 312 may be generated by applying the teacher neural network 304 to the input batch 312. The teacher neural network 304 may be a network pre-trained for an object detection task. The first result for an object in the first unlabeled image 312A of the input batch 312 may include a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes. In at least one embodiment, the circuit 104 may be configured to generate a first result for each image of the input batch 312 by applying the teacher neural network 304 to the input batch 312. The teacher neural network 304 is a network pre-trained for an object detection task, and the first result for an object in the first unlabeled image 312A of the input batch 312 includes a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes. Details of the generation of the first result are shown, for example, in FIG. 3.

[0078]

[0081] At 410, a threshold score can be determined based on the set of scores. In at least one embodiment, the circuit 104 can be configured to determine the threshold score based on the set of scores. Details regarding determining the threshold score are shown, for example, in FIG. 3.

[0079]

[0082] At 412, a foreground bounding box may be selected from the set of candidate bounding boxes based on a threshold score. In at least one embodiment, the circuit 104 may be configured to select a foreground bounding box from the set of candidate bounding boxes based on a threshold score. Details regarding the selection of a foreground bounding box are shown, for example, in FIG. 3.

[0080]

[0083] At 414, a second result including a bounding box prediction of an object may be generated by applying the student neural network 306 to the first unlabeled image 312A. The student neural network 306 may be an untrained network to be trained for an object detection task. In at least one embodiment, the circuit 104 may be configured to generate a second result including a bounding box prediction of an object by applying the student neural network 306 to the first unlabeled image 312A. The student neural network 306 may be an untrained network to be trained for an object detection task. Details regarding the student neural network 306 are shown, for example, in FIG. 3 .

[0081]

[0084] At 416, a training loss 324 can be calculated for the input batch 312 based on the foreground bounding boxes and the bounding box predictions. In at least one embodiment, the circuit 104 can be configured to calculate the training loss 324 for the input batch 312 based on the foreground bounding boxes and the bounding box predictions. Details regarding the calculation of the training loss 324 are shown, for example, in FIG. 3.

[0082]

[0085] At 418, the student neural network 306 may be retrained on the object detection task based on the training loss 324. In at least one embodiment, the circuit 104 may be configured to retrain the student neural network 306 on the object detection task based on the training loss 324. Control may proceed to an end.

[0083]

[0086] Based on experimental data obtained after conducting several experiments while drafting this patent application, the disclosed student-teacher framework significantly outperformed (i.e., improved in terms of average mean accuracy) state-of-the-art semi-supervised object detection methods when performed on known datasets such as the Microsoft® COCO dataset.

[0084]

[0087] Various embodiments of the present disclosure may provide a non-transitory computer-readable medium having stored thereon computer-executable instructions that a circuit or machine may execute to operate a system (e.g., system 102) for training a student network for end-to-end semi-supervised object detection. The computer-executable instructions may cause a machine and / or computer to perform operations including retrieving labeled and unlabeled images from an image dataset (e.g., image dataset 118). The operations may further include generating an input batch (e.g., input batch 312) by applying a set of image transformations to the labeled and unlabeled images. The operations may further include generating a first result for each image of the input batch by applying a teacher neural network (e.g., teacher neural network 108) to the input batch. The teacher neural network may be a network pre-trained for an object detection task, and the first result for an object in a first unlabeled image (e.g., first unlabeled image 312A) of the input batch includes a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes. The operations may further include determining a threshold score based on the set of scores. The operations may further include selecting a foreground bounding box from the set of candidate bounding boxes based on the threshold score. The operations may further include generating a second result including a bounding box prediction of an object by applying a student neural network (e.g., student neural network 110) to the first unlabeled image. The student neural network may be an untrained network to be trained for an object detection task. The operations may further include calculating a training loss (e.g., training loss 324) for the input batch based on the foreground bounding box and the bounding box prediction, and training the student neural network on the object detection task based on the training loss.

[0085]

[0088] Certain embodiments of the present disclosure may be found in a system and method for training a student network for end-to-end semi-supervised object detection. Various embodiments of the present disclosure may provide a system 102 that may include a circuit 104 and a memory 106. The circuit 104 may be configured to retrieve labeled images 308 and unlabeled images 310 from an image dataset 118. The circuit 104 may be further configured to randomly sample the image dataset 118 using a sample ratio to retrieve the labeled images 308 and the unlabeled images 310. The circuit 104 may be further configured to generate an input batch 312 by applying a set of image transforms to the labeled images 308 and the unlabeled images 310. The set of image transforms may include a first subset of image transforms associated with a first data augmentation type and a second subset of image transforms associated with a second data augmentation type, which may be different from the first data augmentation type.

[0086]

[0089] According to one embodiment, a first subset of image transformations may include an image flip operation and an image shift operation, and a second subset of image transformations may include one or more of an image rotation operation, a blurring operation, a contrast change, a shear operation, a masking operation for one or more regions of an image, an add jitter operation, or the addition of random noise.

[0087]

[0090] According to one embodiment, the generated input batch 312 may include a first unlabeled image 312A, a second unlabeled image 312B, and a labeled image 312C. The first unlabeled image 312A and the labeled image 312C may be associated with a first data augmentation type. The second unlabeled image 312B may be associated with a first data augmentation type.

[0088]

[0091] According to one embodiment, the circuit 104 may be configured to generate a first result for each image of the input batch 312 by applying a teacher neural network 304 to the input batch 312. The teacher neural network 304 may be a network pre-trained for an object detection task, and the first result for an object in a first unlabeled image 312A of the input batch 312 may include a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes. The set of scores corresponding to the set of candidate bounding boxes may include foreground scores for foreground bounding boxes of the set of candidate bounding boxes and background scores for background bounding boxes of the set of candidate bounding boxes.

[0089]

[0092] According to one embodiment, the circuit 104 may be further configured to determine a threshold score based on the set of scores. According to one embodiment, the circuit 104 may be configured to calculate an average foreground score by dividing a sum of the foreground scores by the number of foreground bounding boxes. The circuit 104 may be further configured to calculate an average background score by dividing a sum of the background scores by the number of background bounding boxes. In one embodiment, the threshold score may be determined by dividing the average foreground score by the average background score to generate a value and applying a floor function to the value.

[0090]

[0093] According to one embodiment, the circuitry 104 may be further configured to apply a non-maximum suppression operation to the set of candidate bounding boxes to extract a subset of candidate bounding boxes from the set of candidate bounding boxes. The circuitry 104 may further select a foreground bounding box from the subset of candidate bounding boxes.

[0091]

[0094] According to one embodiment, the circuit 104 may be configured to select a bounding box from the subset of candidate bounding boxes. The circuit 104 may further apply a jitter operation to the selected bounding boxes multiple times to generate a set of jitter boxes. The circuit 104 may further perform a bagging operation on the set of jitter boxes to select a jitter box with a maximum area among the respective areas of the set of jitter boxes.

[0092]

[0095] According to one embodiment, the circuit 104 may be further configured to calculate a first unsupervised loss 320 for a first result that may be generated for the first unlabeled image 312A. The circuit 104 may be further configured to calculate a second unsupervised loss 322 for a first result that may be generated for a second unlabeled image 312B of the input batch 312. Each of the first unsupervised loss and the second unsupervised loss is calculated by using an unsupervised loss function, and includes an unsupervised classification loss and an unsupervised box regression loss.

[0093]

[0096] According to one embodiment, the circuit 104 can generate a second result including a bounding box prediction of the object. The second result can be generated by applying a student neural network 306 to the first unlabeled image 312A. The student neural network 306 can be an untrained network to be trained for an object detection task.

[0094]

[0097] According to one embodiment, the circuit 104 may be further configured to generate a third result by applying the student neural network 306 to the labeled images 312C of the input batch 312. The circuit 104 may be further configured to calculate a total supervised loss of the first result associated with the labeled image 312C and the third result associated with the labeled image 312C by using a supervised loss function and a supervised regression loss function. The total supervised loss includes a supervised classification loss and a supervised box regression loss. The circuit 104 may be further configured to calculate a training loss 324 for the input batch 312 based on the foreground bounding box and the bounding box prediction. In another embodiment, the circuit 104 may be further configured to calculate a training loss based on the calculation of the total supervised loss of the labeled images 312C of the input batch 312. In another embodiment, the training loss 324 is further calculated based on the calculation of the first unsupervised loss 320 and the second unsupervised loss 322. In another embodiment, the training loss 324 is calculated further based on a selected jitter box used to calculate a box regression loss that is a portion of the training loss.

[0095]

[0098] According to one embodiment, the circuitry 104 may be further configured to train the student neural network 306 on the object detection task based on the training loss. In one embodiment, the circuitry 104 may be further configured to update weight parameters of the student neural network 306 using the training loss to train the student neural network 306. In another embodiment, the circuitry 104 may be further configured to update weight parameters of the teacher neural network 304 based on the updated weight parameters of the student neural network 306.

[0096]

[0099] According to one embodiment, updating the weight parameters of the teacher neural network 304 includes performing an Exponential Moving Average (EMA) operation and performing an Exponential Adaptive Differential Moving Average (E-ADMA) operation.

[0097]

[0100] The present disclosure can be realized in the form of hardware or in the form of a combination of hardware and software. The present disclosure can be realized in a centralized manner in at least one computer system or in a distributed manner where different elements can be distributed across several interconnected computer systems. Any computer system or other device adapted to perform the methods described herein can be suitable. The combination of hardware and software can be a general-purpose computer system that includes a computer program that, when loaded and executed, can control a computer system to perform the methods described herein. The present disclosure can be realized in the form of hardware, including parts of an integrated circuit that also perform other functions.

[0098]

[0101] The present disclosure may also be embodied in a computer program product, which includes all features enabling the implementation of the methods described herein and which is capable of executing these methods when loaded into a computer system. A computer program in this context means any expression, in any language, code or notation, of a set of instructions intended to cause a system capable of processing information to perform a particular function, either directly or after a) conversion into another language, code or notation, b) reproduction in a different content form, or both.

[0099]

[0102] Although the present disclosure has been described with reference to several embodiments, those skilled in the art will recognize that various modifications can be made and equivalents substituted without departing from the scope of the present disclosure. Many modifications can also be made to adapt a particular situation or content to the teachings of the present disclosure without departing from the scope of the present disclosure. Therefore, it is not intended that the present disclosure be limited to the particular embodiments disclosed, but rather, it is intended to include all embodiments falling within the scope of the appended claims. [Explanation of symbols]

[0100] 100 Network Environment 102 System 104 Circuit 106 Memory 108 Teacher Neural Network 110 Student Neural Network 112 Display Devices 114 Server 116 Communication Networks 118 Image Datasets 118A Set of Labeled Images 118B Set of unlabeled images 120 Input Batches 200 Block Diagram 202 Input / Output (I / O) Devices 204 Network Interface 206 Inference Accelerator 300 Figures 302 Teacher-Student Framework 304 Teacher Neural Network 306 Student Neural Network 308 Labeled Images 308A Animal 308B Bounding Box 310 Unlabeled Images 312 Input Batches 312A First Unlabeled Image 312B Second unlabeled image 312C Labeled Image 314 First Label Generator 316 Second Label Generator 318 total supervised losses 320 First Unsupervised Loss 322 Second Unsupervised Loss 324 Training Loss 400 Flowchart 402 start 404 Extract labeled and unlabeled images from an image dataset 406 Generate an input batch by applying a set of image transformations to labeled and unlabeled images. 408. Generate a first result for each image of the input batch by applying a teacher neural network to the input batch. The teacher neural network is a network pre-trained for an object detection task, and the first result for an object in a first unlabeled image of the input batch includes a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes. Determine the threshold score based on the set of 410 scores 412 Select a foreground bounding box from the set of candidate bounding boxes based on a threshold score. 414 Applying the student neural network to the first unlabeled image produces a second result including bounding box predictions for the object. The student neural network is an untrained network to be trained for the object detection task. 416 Calculate the training loss for the input batch based on the foreground bounding boxes and bounding box predictions. 418 Train a student neural network on the object detection task based on the training loss

Claims

1. 1. A method comprising: Extracting labeled and unlabeled images from an image dataset; generating an input batch by applying a set of image transformations to the labeled and unlabeled images; generating a first result for each image of the input batch by applying a teacher neural network to the input batch, the teacher neural network is a network pre-trained for an object detection task; the first result for an object in a first unlabeled image of the input batch includes a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes; Steps and determining a threshold score based on the set of scores; selecting a foreground bounding box from the set of candidate bounding boxes based on the threshold score; generating a second result including a bounding box prediction of the object by applying a student neural network to the first unlabeled image; the student neural network is an untrained network to be trained for the object detection task; Steps and computing a training loss for the input batch based on the foreground bounding boxes and the bounding box predictions; retraining the student neural network on the object detection task based on the training loss; The method according to claim 1, further comprising:

2. The method of claim 1 , further comprising randomly sampling the image dataset using a sample ratio to extract the labeled and unlabeled images.

3. 2. The method of claim 1, wherein the set of image transforms includes a first subset of image transforms associated with a first data augmentation type and a second subset of image transforms associated with a second data augmentation type different from the first data augmentation type.

4. the first subset of image transformations includes an image flip operation and an image shift operation; the second subset of image transformations includes one or more of an image rotation operation, a blurring operation, a contrast change, a shear operation, a masking operation for one or more regions of an image, a jitter addition operation, or the addition of random noise; 4. The method according to claim 3 .

5. 4. The method of claim 3, wherein the input batch includes a second unlabeled image and a labeled image associated with the first data augmentation type, the first unlabeled image being associated with the second data augmentation type.

6. generating a third result by applying the student neural network to the input batch of labeled images; Calculating a total supervised loss of the first result associated with the labeled image and the third result associated with the labeled image by using a supervised loss function and a supervised regression loss function, The total supervised loss includes a supervised classification loss and a supervised box regression loss, The training loss is further calculated based on a calculation of the total supervised loss of the labeled images of the input batch. Steps and The method of claim 1 further comprising:

7. Calculating a first unsupervised loss for the first result generated for the first unlabeled image; Calculating a second unsupervised loss for the first result generated for a second unlabeled image of the input batch, Each of the first unsupervised loss and the second unsupervised loss is calculated by using an unsupervised loss function, and includes an unsupervised classification loss and an unsupervised box regression loss; The training loss is calculated further based on the calculation of the first unsupervised loss and the second unsupervised loss. Steps and The method of claim 1 further comprising:

8. 8. The method of claim 7, wherein each of the first unsupervised loss and the second unsupervised loss is equal to a sum of a foreground classification loss, a background classification loss, a background similarity loss, and a foreground-background dissimilarity loss.

9. 2. The method of claim 1, further comprising applying a non-maximum suppression operation to the set of candidate bounding boxes to extract a subset of candidate bounding boxes from the set of candidate bounding boxes.

10. The method of claim 9 , wherein the foreground bounding box is selected from a subset of the candidate bounding boxes.

11. selecting a bounding box from the subset of candidate bounding boxes; generating a set of jitter boxes by applying a jitter operation to the selected bounding boxes multiple times; performing a bagging operation on the set of jitter boxes to select a jitter box having a maximum area among the areas of the set of jitter boxes; The selected jitter box is used to calculate a box regression loss, which is a portion of the training loss. Steps and 10. The method of claim 9 further comprising:

12. The set of scores may include: foreground scores for foreground bounding boxes of the set of candidate bounding boxes; background scores of background bounding boxes of the set of candidate bounding boxes; Including, 2. The method according to claim 1 .

13. calculating an average foreground score by dividing the sum of the foreground scores by the number of foreground bounding boxes; calculating an average background score by dividing the sum of the background scores by the number of background bounding boxes; The threshold score is: dividing the average foreground score by the average background score to generate a value; applying a floor function to said values; It is determined by Steps and The method of claim 12 further comprising:

14. 2. The method of claim 1, further comprising updating weight parameters of the student neural network using the training loss to retrain the student neural network.

15. 15. The method of claim 14, further comprising updating weight parameters of the teacher neural network based on the updated weight parameters of the student neural network.

16. 16. The method of claim 15, wherein updating the weight parameters of the teacher neural network includes performing an exponential moving average (EMA) operation and performing an exponential adaptive difference moving average (E-ADMA) operation.

17. 1. A system comprising: Extracting labeled and unlabeled images from an image dataset; generating an input batch by applying a set of image transforms to the labeled and unlabeled images; generating a first result for each image of the input batch by applying a teacher neural network to the input batch; the teacher neural network is a network pre-trained for an object detection task; the first result for an object in a first unlabeled image of the input batch includes a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes; And, determining a threshold score based on the set of scores; selecting a foreground bounding box from the set of candidate bounding boxes based on the threshold score; generating a second result including a bounding box prediction of the object by applying a student neural network to the first unlabeled image; the student neural network is an untrained network to be trained for the object detection task; And, computing a training loss for the input batch based on the foreground bounding boxes and the bounding box predictions; retraining the student neural network on the object detection task based on the training loss; a circuit configured to: A system comprising:

18. 20. The system of claim 17, wherein the circuitry is further configured to use the training loss to update weight parameters of the student neural network to retrain the student neural network.

19. the circuitry is further configured to update weight parameters of the teacher neural network based on the updated weight parameters of the student neural network; The updating of the weight parameters of the teacher neural network includes performing an exponential moving average (EMA) operation and performing an exponential adaptive differential moving average (E-ADMA) operation.

20. The system of claim 18.

20. A non-transitory computer-readable storage medium configured to store instructions that, upon execution, cause a computer in a system to perform an operation, the operation including: Extracting labeled and unlabeled images from an image dataset; generating an input batch by applying a set of image transforms to the labeled and unlabeled images; generating a first result for each image of the input batch by applying a teacher neural network to the input batch; the teacher neural network is a network pre-trained for an object detection task; the first result for an object in a first unlabeled image of the input batch includes a set of candidate bounding boxes for the object and a set of scores corresponding to the set of candidate bounding boxes; And, determining a threshold score based on the set of scores; selecting a foreground bounding box from the set of candidate bounding boxes based on the threshold score; generating a second result including a bounding box prediction of the object by applying a student neural network to the first unlabeled image; the student neural network is an untrained network to be trained for the object detection task; And, computing a training loss for the input batch based on the foreground bounding boxes and the bounding box predictions; retraining the student neural network on the object detection task based on the training loss; Including, A non-transitory computer-readable storage medium comprising:

Images