Optimizing models for open vocabulary detection

Abstract

A system and method for optimizing a model for open vocabulary detection. Region proposals can be obtained using a pre-trained visual-language model and a pre-trained region proposal network. Object feature predictions can be obtained using a teacher neural network trained with region proposals. Object feature predictions exceeding a threshold can be filtered to obtain pseudo-labels. A student neural network with a split-fusion detection head can be trained using region proposals, basic ground truth class labels, and pseudo-labels. Optimized object detection can be obtained by optimizing pseudo-labels by reducing noise from them using the trained split-fusion detection head of the trained student neural network. Based on the optimized object detection, actions can be taken on the scene layout.

Description

Technical Field

[0001] Related Application Information This application claims priority to U.S. Provisional Application No. 63 / 465,607, filed May 11, 2023; U.S. Provisional Application No. 63 / 466,827, filed May 16, 2023; U.S. Provisional Application No. 63 / 599,137, filed Nov. 15, 2023; and U.S. Patent Application No. 18 / 659,738, filed May 9, 2024, the entire contents of which are incorporated herein by reference.

[0002] The present invention relates to open vocabulary object detection, and more particularly to systems and methods for optimizing models for open vocabulary detection.

Background Art

[0003] Description of Related Art Conventional closed-set object detectors can only detect objects with a limited number of categories. Increasing the size of the detection vocabulary requires a huge amount of human labor for collecting annotated data. Furthermore, training an object detector with a large detection size may generate noisy and unreliable pseudo labels. Therefore, a balance of model accuracy, efficiency, and flexibility is required.

Summary of the Invention

[0004] According to one aspect of the present invention, a method for optimizing a model for open vocabulary detection is provided. This method includes obtaining region proposals using a pre-trained visual language model and a pre-trained region proposal network; obtaining object feature predictions using a teacher neural network trained with the region proposals; obtaining pseudo-labels by filtering out object feature predictions that exceed a proposal threshold; training a student neural network having a split-fusion detection head using the region proposals, ground truth class labels, ground truth bounding boxes, and the pseudo-labels; obtaining optimized object detection by optimizing the pseudo-labels by reducing noise from the pseudo-labels using the trained split-fusion detection head of the trained student neural network; and performing actions on a scene layout using the optimized object detection.

[0005] According to another aspect of the present invention, a system for optimizing a model for open vocabulary detection is provided. The system comprises a memory and one or more processor devices that communicate with the memory, configured to acquire region proposals using a pre-trained visual language model and a pre-trained region proposal network; acquire object feature predictions using a teacher neural network trained with the region proposals; acquire pseudo-labels by filtering out object feature predictions that exceed a proposal threshold; train a student neural network having a split-fusion detection head using the region proposals, ground truth class labels, ground truth bounding boxes, and the pseudo-labels; acquire optimized object detection by optimizing the pseudo-labels by reducing noise from the pseudo-labels using the trained split-fusion detection head of the trained student neural network; and perform actions on the scene layout using the optimized object detection.

[0006] According to another aspect of the present invention, a non-temporary computer program product is provided, comprising a computer-readable storage medium containing program code for optimizing a model for open vocabulary detection, wherein, when executed on a computer, the program code causes the computer to: obtain region proposals by using a pre-trained visual language model and a pre-trained region proposal network; obtain object feature predictions by using a teacher neural network trained with the region proposals; obtain pseudo-labels by filtering out object feature predictions exceeding a proposal threshold; train a student neural network having a split-fusion detection head using the region proposals, ground truth class labels, ground truth bounding boxes, and the pseudo-labels; optimize the pseudo-labels by reducing noise from the pseudo-labels using the trained split-fusion detection head of the trained student neural network; obtain optimized object detection by fusing prediction scores from closed and open branches by calculating the geometric mean of the prediction scores at inference; and perform actions on the scene layout using the optimized object detection.

[0007] These features and advantages, as well as other features and advantages, will become apparent from the following detailed description of exemplary embodiments of the present invention, which should be read in conjunction with the accompanying drawings. [Brief explanation of the drawing]

[0008] This disclosure provides further details in the following description of preferred embodiments with reference to the following drawings.

[0009] [Figure 1] This flowchart illustrates a high-level method for optimizing a model for open vocabulary detection according to one embodiment of the present invention.

[0010] [Figure 2]This flowchart illustrates a method for obtaining region proposals using a pre-trained visual language model and a pre-trained region proposal network, according to one embodiment of the present invention.

[0011] [Figure 3] This flowchart illustrates a method for obtaining object feature predictions by using a supervised neural network trained with a domain proposal, according to one embodiment of the present invention.

[0012] [Figure 4] This flowchart illustrates a method for obtaining pseudo-labels by filtering object feature predictions that exceed a threshold, according to one embodiment of the present invention.

[0013] [Figure 5] This flowchart illustrates a method for training a student neural network with a split-fusion detection head using a region proposal, ground truth class label, ground truth bounding box, and pseudo-label according to one embodiment of the present invention.

[0014] [Figure 6] This flowchart illustrates a method for optimizing pseudolabels by reducing noise from pseudolabels using a trained split-fusion detection head of a trained student neural network, according to one embodiment of the present invention.

[0015] [Figure 7] This block diagram shows an overview of a system for optimizing a model for open vocabulary detection according to one embodiment of the present invention.

[0016] [Figure 8] This is a block diagram showing a system for optimizing a model for open vocabulary detection according to one embodiment of the present invention.

[0017] [Figure 9]A flowchart showing a system for optimizing a model for open vocabulary detection according to an embodiment of the present invention.

[0018] [Figure 10] A flowchart showing an overview of a deep learning neural network according to an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0019] According to an embodiment of the present invention, a system and method for optimizing a model for open vocabulary detection are provided.

[0020] This embodiment can optimize a model for open vocabulary detection. In one embodiment, region proposals can be obtained using a pre-trained vision-language model (VLM) and a pre-trained external region proposal network (RPN). In one embodiment, object feature prediction can be obtained by employing a teacher neural network trained using the region proposals. In one embodiment, object feature predictions exceeding a threshold can be filtered to obtain pseudo labels. In one embodiment, a student neural network equipped with a split-merge detection head can be trained using the region proposals, ground truth class labels, ground truth bounding boxes, and pseudo labels. In one embodiment, the pseudo labels can be optimized and optimized object detection can be obtained by reducing noise from the pseudo labels using the trained split-merge (SAF) detection head of the trained student neural network. In one embodiment, actions can be performed on the scene layout using the optimized object detection. In one embodiment, the teacher neural network can be periodically updated using the parameters of the student neural network.

[0021] In one embodiment, the action may be a change in driving behavior in relation to a traffic scene layout. In another embodiment, the action may be the creation of an interior design plan in relation to an interior design scene layout. In yet another embodiment, the action may be the creation of an exterior design plan in relation to an exterior design scene layout. In yet another embodiment, the action may be vehicle control in relation to a traffic scene layout.

[0022] The SAF detection head can avoid noisy locations of pseudolabels and bounding boxes by reducing the accumulation of noise during self-training. In one embodiment, the SAF detection head can be used to optimize pseudolabels by reducing the noise in the pseudolabels. In one embodiment, the SAF detection head can be divided into two branches, a closed branch and an open branch, which are fused during inference. In one embodiment, the closed branch may include a classification module and a box improvement module. In one embodiment, the closed branch may be trained using only ground truth class labels for the base categories, thereby reducing the impact of noisy pseudolabels on the performance of the base categories. In one embodiment, the open branch may be a classification module trained using ground truth class labels and pseudolabels. In one embodiment, the open branch may acquire complementary knowledge from the closed branch and significantly improve performance by merging with the closed branch.

[0023] The recent emergence of powerful visual-language models (VLMs) has given open-vocabulary object detection (OVD) a new direction for approaching this challenge. OVD detectors are trained using annotations for basic categories and are expected to generalize to novel target categories by leveraging the power of pre-trained VLMs. Recent research on OVD has leveraged VLMs to obtain pseudo-labels beyond basic categories, but self-training has not been considered. Given an image, an object detection model can identify objects for any category name that can be provided as free-form text, and does not need to be part of the object detection training data (e.g., including bounding box annotations). Large-scale visual-language models (VLMs) can be trained with vast amounts of image-caption pairs that can be easily collected on a large scale. While VLMs have excellent zero-shot classification capabilities, they cannot identify objects within an image.

[0024] In self-training for closed-set tasks, a confidence threshold is set to remove noisy pseudolabels. This is because the quality of pseudolabels is positively correlated with their confidence. However, the VLM used in OVD is pre-trained for image-level text alignment, not instance-level object detection with location estimation. Therefore, confidence scores from pre-trained VLMs are not typically a good indicator of the quality of the box locations (e.g., pseudoboxes) provided by pseudolabels. This problem is further exacerbated when self-training is applied directly, as noise accumulates and performance in new categories deteriorates. Furthermore, these methods treat noisy pseudolabels the same way as ground truth class labels in the base categories during training, resulting in even worse performance in the base categories.

[0025] Furthermore, self-training for closed-set object detection typically follows an online teacher-student model. In each training iteration, the teacher generates pseudo-labels, and the student is trained with a mixture of ground truth class labels and pseudo-labels. The student then updates the teacher using an exponential moving average (EMA). However, EMA updates degrade the OVD model and destabilize the training.

[0026] This embodiment can optimize VLM for open vocabulary detection. By separating the detection head and periodically updating the training neural network, this embodiment can mitigate the degradation seen during updates with EMA and generate pseudo-labels more efficiently than conventional methods. Therefore, this embodiment can balance the accuracy, efficiency, and flexibility of an open vocabulary object detection model by using an SAF detection head and periodically updating the training neural network.

[0027] Next, we will refer in detail to the drawings where the same number represents identical or similar elements. Referring first to Figure 1, a high-level overview of a method 100 for optimizing a model for open vocabulary detection according to one embodiment of the present invention is illustrated.

[0028] In one embodiment, a region proposal 510 (shown in Figure 8) can be obtained using a visual language model 504 (shown in Figure 8) and an external region proposal network 509 (shown in Figure 8). In one embodiment, a teacher-student architecture can be employed, where the teacher neural network 511 (shown in Figure 8) can generate pseudo-labels 518 (shown in Figure 8) for any list of categories. In one embodiment, the student neural network 522 (shown in Figure 8) utilizes the pseudo-labels 518 along with an input dataset 501 (shown in Figure 7) for training. In one embodiment, the teacher neural network 511 can use a split-fusion detection head 512 (shown in Figure 8). In one embodiment, the student neural network 522 can optimize the pseudo-labels 518 using a split-fusion detection head 523 (shown in Figure 8) to obtain optimized pseudo-labels 540 (shown in Figure 8).

[0029] In one embodiment, the detection head can be divided into separate branches, each undergoing supervised learning in a different manner. In one embodiment, closed branches (e.g., teacher closed branch 513 and student closed branch 524 shown in Figure 8) can be supervised learning using closed set detection data (e.g., ground truth class labels 520 and ground truth bounding boxes 519 shown in Figure 8). In one embodiment, open branches (e.g., teacher open branch 514 and student open branch 525 shown in Figure 8) can be used for classification and trained using ground truth class labels 520 (shown in Figure 8) and pseudo-labels 518. Optimized pseudo-labels 540 correspond to optimized object detection 541. In one embodiment, a scene layout 608 (shown in Figure 9) can be created based on optimized object detection 541.

[0030] In block 110, a domain proposal 510 is obtained by using a pre-trained visual language model 504 and a pre-trained domain proposal network 509.

[0031] In block 120, the region proposal 510 is used to train the training neural network 511 and obtain object feature predictions 516.

[0032] In block 130, the object feature prediction 516 can be filtered against the proposed threshold 517 (shown in Figure 8) to obtain a pseudo-label 518.

[0033] In block 140, a student neural network 522 equipped with a split-fusion detection head 523 can be trained using the region proposal 510, ground truth class label 520, and pseudo-label 518.

[0034] In block 150, the pseudo-labels 518 are optimized by reducing noise from the pseudo-labels 518 using the split-fusion head 523, thereby obtaining optimized object detection 541 (shown in Figure 8).

[0035] In block 160, the teacher neural network 511 can be updated with the parameters of the student neural network 522 by using the periodic updater 532 (shown in Figure 8).

[0036] In block 170, it is checked whether the number of iterations is below a threshold. If the number of iterations is below the threshold, the number of iterations is increased in block 171, and the process returns to block 120, where the parameters of the training neural network 511 are updated. Otherwise, the process proceeds to block 180. In one embodiment, the threshold can be a predetermined value based on the learning rate schedule of the training neural network 511. For example, the threshold can be 3.

[0037] In block 180, an action 610 can be performed on a scene layout 608 created based on optimized object detection 541. In one embodiment, the scene layout 608 is interpreted by a decision-making entity, which can then perform an action 610 (shown in Figure 9). In one embodiment, the action 610 can be a modification of driving behavior, such as braking or turning. In another embodiment, the action 610 can be the creation of an external design plan for an external scene layout 608. In yet another embodiment, the action 610 can be the creation of an internal design plan for an internal scene layout 608.

[0038] Referring to Figure 2, an exemplary method for obtaining region proposals by using a pre-trained visual language model and a pre-trained region proposal network 110 according to one embodiment of the present invention is shown.

[0039] In one embodiment, a pre-trained visual-language model 504 can be used to obtain ground truth bounding boxes 519 for basic categories. In another embodiment, a pre-trained external domain proposal network 509 can be used to obtain domain proposals 510 by using the ground truth bounding boxes 519 for basic categories and the input dataset 501.

[0040] In block 111, a visual-language model 504 can be pre-trained with an input dataset 501 and image-text alignment to obtain a ground truth bounding box 519 for basic categories. In one embodiment, the visual-language model 504 can be a contrasting language-image pre-trained (CLIP) model. Other visual-language models can also be used. In one embodiment, the Common Objects in Context (COCO) dataset can be used as the input dataset 501. In another embodiment, the Large Vocabulary Instance Segmentation (LVIS) dataset can be used as the input dataset 501. The ground truth bounding box 519 for basic categories can be a bounding box containing basic categories. Basic categories are categories already learned by the VLM based on the training dataset. For example, basic categories can include objects (e.g., bench, cat, dog) and people.

[0041] In block 112, the external domain proposed network (RPN) 509 can be pre-trained using the ground truth bounding box 519 of the basic categories and the input dataset 501. In one embodiment, the RPN 509 can be a faster domain-based convolutional neural network (RCNN). Other RPN frameworks are also possible.

[0042] In block 113, a domain proposal 510 can be obtained by using a pre-trained external RPN 509. The domain proposal 510 is a candidate bounding box that is likely to contain the object of interest based on its basic category. For example, the domain proposal 510 may be a bounding box that contains a basic category such as bench.

[0043] In block 114, the region of interest (ROI) aligner 521 is used to align the region proposals 510 and obtain the proposed features 542.

[0044] In block 115, a region embedding can be obtained by inputting the proposed feature 542 into the final layer of the visual-language model. In one embodiment, the proposed feature 542 can be obtained by applying an alignment algorithm to the fourth feature map of the visual-language model 504 for the i-th region proposal from an external RPN. These features are input into the final layer of the VLM to obtain the region embedding Ri. A set of concepts given with a prompt (e.g., known objects such as chairs and sofas) can be converted into a text embedding that can function as a classifier. The probability (Pi,c) that the i-th region embedding is classified into the c-th category is:

[0045]

number

[0046] Here<Ri,Tc> This includes region embedding Ri, text embedding Tc for category c, and temperature.

number

[0047] Referring to Figure 3, an exemplary method for obtaining object feature predictions by using a supervised neural network trained with region proposal 120, according to one embodiment of the present invention, is shown.

[0048] In one embodiment, the teacher neural network 511 can be trained using region proposals 510 obtained from an external RPN. In one embodiment, the teacher neural network 511 can use a split-fuse-fuse (SAF) detection head that can be periodically updated using parameters from the trained student neural network 522.

[0049] Block 121 can receive and process the region proposal 510.

[0050] In block 122, the training neural network 511 can be trained using the region proposal 510. In one embodiment, the training neural network 511 can be pre-trained using pairs of images with corresponding captions. In one embodiment, the training neural network 511 may include a VLM such as CLIP.

[0051] In block 123, the object feature prediction 516 can be predicted using the trained supervising neural network 511. In one embodiment, the object feature prediction 516 may include the class label of the region proposal and the bounding box coordinates of the predicted object.

[0052] Referring to Figure 4, an exemplary method for obtaining pseudo-labels 130 by filtering object feature predictions that exceed a proposed threshold is shown according to one embodiment of the present invention.

[0053] In one embodiment, a pseudo-label 518 can be obtained by filtering the object feature prediction 516 with a proposed threshold 517.

[0054] In block 131, object feature predictions 516 can be received and processed.

[0055] In block 132, the proposed threshold 517 can be determined. In one embodiment, the intersection union (IoU) score can be used to determine the proposed threshold 517. In one embodiment, the proposed threshold 517 may be 0.5 (for example, for the COCO dataset). In another embodiment, the proposed threshold 517 may be in the range of 0.5 to 0.95 (for example, for the LVIS dataset).

[0056] In block 133, a pseudo-label 518 can be obtained by comparing the confidence score of the predictive inference with the proposed threshold 517. In one embodiment, if the confidence score is greater than 0.5, the classification of the region proposal can be a pseudo-label.

[0057] Referring to Figure 5, an exemplary method is shown for training a student neural network with a split-fusion detection head by utilizing a region proposal, a ground truth class label, and a pseudo-label 140 according to one embodiment of the present invention.

[0058] In one embodiment, the detection head 523 can be split into two branches, an open branch 525 and a closed branch 524, to handle noisy pseudo-labels 518 during training. During inference, predictions from both branches are merged to improve performance. In one embodiment, the closed branch 524 can be trained with a ground truth bounding box 519 and a ground truth class label 520 by applying cross-entropy loss for classification. In one embodiment, the closed branch 524 can be trained with a ground truth bounding box 519 and a ground truth class label 520 by using box regression loss for location estimation. In one embodiment, the open branch 525 can be trained with pseudo-labels 518 and a ground truth class label 520 by using cross-entropy loss for classification.

[0059] Block 141 can receive and process the ground truth class label 520, the pseudo-label 518, and the ground truth bounding box 519.

[0060] In block 142, the closed branch 524 can be trained using the ground truth bounding box 519 and the ground truth class label 520 by using cross-entropy loss for classification.

[0061] In block 143, a closed branch 524 can be trained using a ground truth bounding box 519 and ground truth class labels 520 by using a box regression loss for localization.

[0062] In block 144, the open branch 525 can be trained using the pseudo-label 518 and the ground truth class label 520 by using cross-entropy loss for classification.

[0063] Referring to Figure 6, an exemplary method of optimizing pseudolabels by reducing noise from pseudolabels using a trained split-fusion detection head of a trained student neural network 150 is shown according to one embodiment of the present invention.

[0064] In one embodiment, complementary prediction scores can be obtained from both branches. In one embodiment, a closed class loss 529 (see Figure 8) can be obtained by using a trained closed branch detection head. In one embodiment, an open class loss 531 (see Figure 8) can be obtained by using a trained open branch detection head. In one embodiment, a closed box loss 528 (see Figure 8) can be obtained by using a trained closed branch detection head.

[0065] In block 151, the closed class loss 529 can be obtained by using the trained closed branch 524.

[0066] In block 152, the open class loss 531 can be obtained by using the trained open branch 525.

[0067] In block 153, the closed-box loss 528 can be obtained by using the trained closed-branch 524.

[0068] In block 154, prediction scores can be obtained from both closed branch 524 and open branch 525. In one embodiment, the prediction score from closed branch 524 can be obtained by calculating a weighted average of the box loss 528 and the closed class loss 529. In one embodiment, the prediction score from open branch 530 can be the open class loss 531.

[0069] In block 155, the fusion module 526 (see Figure 8) is used to calculate the geometric mean of the prediction scores during inference to obtain the final score, thereby fusing the complementary prediction scores. In one embodiment, the complementary prediction score can be the prediction scores from both the open branch 525 and the closed branch 524 related to a common object detection category. The category is the basic category C. b Or a new category C N This can be done. In one embodiment, the new category is a category newly discovered by the system and is not included in the basic categories of the input dataset 501.

[0070] The final score can be calculated as follows:

[0071]

number

[0072] Here, i is an element of category C, and C b This is a basic category, C N This is a new category,

number

number

number

[0073] Referring to Figure 7, an exemplary system 500 of a method for optimizing a model for open vocabulary detection according to one embodiment of the present invention is shown.

[0074] The arithmetic unit 500 includes, exemplary, a processor device 594, an input / output (I / O) subsystem 590, memory 591, a data storage device 592, and a communication subsystem 593, and / or other components and devices commonly found in a server or similar arithmetic unit. In other embodiments, the arithmetic unit 500 may include other or additional components, such as those commonly found in a server computer (e.g., various input / output devices). Furthermore, in some embodiments, one or more of the exemplary components may be incorporated into or form part of other components. For example, in some embodiments, memory 591 or a part thereof may be incorporated into the processor device 594.

[0075] The processor device 594 can be implemented as any type of processor capable of performing the functions described herein. The processor device 594 can be implemented as a single processor, multiple processors, a central processing unit (CPU), a graphics processing unit (GPU), a single-core processor or a multi-core processor, a digital signal processor, a microcontroller, or any other processor or processing / control circuit.

[0076] Memory 591 can be implemented as any type of volatile or non-volatile memory, or data storage device, capable of performing the functions described herein. When in operation, memory 591 can store various data and software (such as operating systems, applications, programs, libraries, and drivers) used during the operation of the arithmetic unit 500. Memory 591 is communicatively connected to the processor device 594 via the I / O subsystem 590, which can be implemented as circuits and / or components that facilitate input / output operations with the processor device 594, memory 591, and other components of the arithmetic unit 500. For example, the I / O subsystem 590 can be implemented as, or otherwise include, a memory control hub, an input / output control hub, a platform control hub, an integrated control circuit, a firmware device, communication links (such as point-to-point links, bus links, wires, cables, optical guides, and traces on printed circuit boards), and / or other components and subsystems that facilitate input / output operations. In some embodiments, the I / O subsystem 590 can form part of a system-on-a-chip (SOC) and be integrated onto a single integrated circuit chip together with other components of the processor device 594, memory 591, and arithmetic unit 500.

[0077] The data storage device 592 can be implemented as any type of device(s) configured for short-term or long-term data storage, such as a memory device or circuit, a memory card, a hard disk drive, a solid-state drive, or other data storage device. The data storage device 592 can store program code for optimizing the model for open vocabulary detection 100. Some or all of these program code blocks may be included in a given arithmetic system.

[0078] The communication subsystem 593 of the computing unit 500 can be implemented as any network interface controller, or other communication circuits, devices, or sets thereof, that enable communication between the computing unit 500 and other remote devices over a network. The communication subsystem 593 can be configured to employ any one or more communication technologies (e.g., wired or wireless communication) and associated protocols (e.g., Ethernet, InfiniBand®, Bluetooth®, Wi-Fi®, WiMAX, etc.) to enable such communication.

[0079] As shown in the figure, the arithmetic unit 500 may also include one or more peripheral devices 595. Peripheral devices 595 may include any number of additional input / output devices, interface devices, and / or other peripheral devices. For example, in some embodiments, peripheral devices 595 may include a display, a touchscreen, graphics circuitry, a keyboard, a mouse, a speaker system, a microphone, a network interface, and / or other input / output devices, interface devices, a GPS, a camera, and / or other peripheral devices.

[0080] Of course, the arithmetic unit 500 may include other elements (not shown) that are readily conceivable to those skilled in the art, and certain elements may be omitted. For example, the arithmetic unit 500 may include various other sensors, input devices, and / or output devices depending on its specific implementation, as will be readily understood to those skilled in the art. For example, various types of wireless and / or wired input and / or output devices may be used. Furthermore, additional processors, controllers, memories, etc., in various configurations may also be used. These and other variations of the processing system 500 are readily conceivable to those skilled in the art based on the teachings of the present invention as described herein.

[0081] Referring to Figure 8, an exemplary system for optimizing a model for open vocabulary detection according to one embodiment of the present invention is shown.

[0082] In one embodiment, a pre-trained visual-language model 504 can be used to obtain a ground truth bounding box 519. The visual-language model may include an image encoder 505 and a text encoder 506. In one embodiment, a pre-trained external region suggestion network 509 can be used to obtain region suggestions 510.

[0083] In one embodiment, a training neural network 511 can be used to obtain object feature predictions 516. The object feature predictions 516 can be compared with a proposed threshold 517 to obtain pseudo-labels 518. The training neural network 511 may include a training partition-fusion (SAF) detection head 512. The training SAF detection head 512 may further include a training closed branch 513, a training open branch 514, and a training fusion module 515. In one embodiment, a region of interest aligner 521 can be used to align the region proposals 510.

[0084] In one embodiment, a student neural network 522 can be trained using a region suggestion 510, a pseudo-label 518, a ground truth bounding box 519, and a ground truth class label 520 to obtain an optimized pseudo-label 540 and an optimized object detection 541. In one embodiment, the student neural network 522 may include a student SAF detection head 523. In one embodiment, the student SAF detection head 523 may include a student closed branch 524, a student open branch 525, and a student fusion module 526.

[0085] In one embodiment, a periodic updater 532 can be used to update the parameters of the teacher neural network 511 to match the parameters of the student neural network 522.

[0086] Referring to Figure 9, an exemplary system 600 for optimizing a model for open vocabulary detection is shown according to one embodiment of the present invention.

[0087] In one embodiment, entity 601 can capture the current image 603 by using camera 602. Entity 601 transmits the current image 603 via network 604 to computer system 605. Computer system 605 implements computer implementation method 100 for optimizing a model for open vocabulary detection and can obtain optimized object detection 606 based on optimized pseudo-labels 540. Optimized object detection 541 can be further processed to obtain a scene layout 608. The scene layout 608 can be interpreted by decision-making entity 609 and an action 610 can be performed.

[0088] In one embodiment for traffic analysis, the optimized object detection 606 can be objects in the traffic scene such as cars, buildings, pedestrians, animals (e.g., dogs, cats), traffic signs, and lampposts. The decision-making entity 609 can interpret the scene layout 608 and perform actions 610 such as changing driving behavior (e.g., turning, accelerating, braking) relative to the scene layout 608. In another embodiment, the system 600 can update the vehicle's route trajectory relative to the scene layout 608 using the optimized object detection 606 by using a trajectory model that calculates an appropriate route trajectory based on the current driving conditions (e.g., traffic volume, vehicle speed, adjacent traffic) and the scene layout 608. In yet another embodiment, the system 600 can control the vehicle relative to the scene layout (e.g., braking, turning) using the optimized object detection 606.

[0089] In an embodiment for exterior design, the optimized object detection 606 can be exterior design objects such as buildings, tables, benches, animals (e.g., dogs, cats), streetlights, walls, and rocks. The decision-making entity 609 can interpret the scene layout 608 and perform actions 610, such as creating an exterior design plan for the scene layout 608. In another embodiment, the system 600 can create an exterior design plan for the scene layout 608. Action 610 may involve the decision-making entity 609 approving the created exterior design plan and sending it to the appropriate entity concerned.

[0090] In an embodiment for interior design, the optimized object detection 606 can be interior design objects such as tables, chairs, appliances, animals (e.g., dogs, cats), lighting fixtures, and walls. The decision-making entity 609 can interpret the scene layout 608 and perform actions 610, such as creating an interior design plan based on the scene layout 608. In another embodiment, the system 600 can create an interior design plan for the scene layout 608. Action 610 may be that the decision-making entity 609 approves the interior design plan and sends it to the appropriate entity concerned. Other practical applications are also conceivable.

[0091] In one embodiment, neural networks such as a visual language model 504, an external RPN model, a teacher neural network 511, and a student neural network 522 can be used.

[0092] Referring to Figure 10, an illustrative overview of a deep learning neural network according to one embodiment of the present invention is shown.

[0093] A neural network is a general-purpose system that improves its function and accuracy by being exposed to additional empirical data. Neural networks are trained by exposure to empirical data. During training, the neural network remembers and adjusts multiple weights applied to the input empirical data. By applying these adjusted weights to the data, it can identify whether the data belongs to a specific predefined class from a group of classes, or output the probability that the input data belongs to each class.

[0094] The empirical data obtained from a series of examples (also known as training data) can be formatted as a string of values ​​and fed into a neural network. Each example can be associated with a known result or output. Each example can be represented as a pair (x,y), where x is the input data and y is the known output. The input data can contain a variety of different data types and can contain multiple different values. The network can have one input node for each value that makes up the example's input data, and individual weights can be applied to each input value. The input data can be formatted as a vector, array, or string, for example, depending on the architecture of the neural network being built and trained.

[0095] A neural network "learns" by comparing the neural network output generated from input data with known values ​​from examples and adjusting stored weights to minimize the difference between the output and the known values. This adjustment of stored weights can be performed using backpropagation, where the effect of the weights on the output is determined by calculating a mathematical gradient and adjusting the weights to bring the output closer to the minimum difference. This optimization is called gradient descent and is a non-restrictive example of how training can be performed. The accuracy of the neural network can be tested and validated using a subset of examples with known values ​​that were not used in training.

[0096] During operation, the trained neural network can be used on new data that has not been previously used for training or validation by generalization. The weights of the tuned neural network can be applied to the new data, where the weights estimate a function constructed from the training examples. The parameters of the estimated function captured by the weights are based on statistical inference.

[0097] Figure 10 shows a deep neural network. A deep neural network 1000, such as a multilayer perceptron, may comprise an input layer 911 with source nodes 912, one or more computation layers 926 with one or more computation nodes 932, and an output layer 940 with a single output node 942 for each category into which the input example may be classified. The input layer 911 may have a number of source nodes 912 equal to the number of data values ​​912 in the input data 911. The computation nodes 932 in the computation layer 926 are located between the source nodes 912 and the output nodes 942 and are not directly observed, and are therefore also called hidden layers. Each node 932, 942 in the computation layer generates a linear combination of weighted values ​​from the output values ​​of the nodes in the previous layer and applies a differentiable nonlinear activation function over the range of the linear combination. The weights applied to the values ​​from each previous node are, for example, w1, w2, ... w n-1 ,w n It can be represented as follows: The output layer provides the network's overall response to the input data. Deep neural networks can be fully connected, where each node in the computational layer is connected to all other nodes in the previous layer, or the connections between layers can be in other configurations. If there are missing links between nodes, the network is said to be partially connected.

[0098] In one embodiment, the computing layer 926 of a closed branch (e.g., teacher closed branch 513 and student closed branch 524) can calculate the box loss by utilizing the ground truth class label 520 and the ground truth bounding box 519. The output layer 940 of the closed branch (e.g., teacher closed branch 513 and student closed branch 524) can provide the overall network response in the form of the box loss. In another embodiment, the computing layer 926 of a closed branch (e.g., teacher closed branch 513 and student closed branch 524) can calculate the class loss by utilizing the ground truth class label 520 and the ground truth bounding box 519. The output layer 940 of the closed branch (e.g., teacher closed branch 513 and student closed branch 524) can provide the overall network response in the form of the class loss. In yet another embodiment, the computing layer 926 of an open branch (e.g., teacher open branch 514 and student open branch 525) can calculate the class loss by utilizing the ground truth class label 520 and the pseudo-label. The output layer 940 of the open branches (e.g., teacher open branch 514 and student open branch 525) can provide the overall network response in the form of class loss.

[0099] Training a deep neural network may involve two phases: a forward phase in which the weights of each node are fixed and the input is propagated through the network, and a backward phase in which error values ​​are backpropagated through the network and the weight values ​​are updated.

[0100] One or more computational (hidden) layers 926 computation nodes 932 perform a nonlinear transformation on the input data 912 that generates the feature space. Classes or categories can be separated more easily in the feature space than in the original data space.

[0101] The embodiments described herein may be entirely hardware, entirely software, or may include both hardware and software elements. In preferred embodiments, the present invention is implemented in software including, but not limited to, firmware, resident software, and microcode.

[0102] Embodiments may include computer program products accessible from computer-usable or computer-readable media that provide program code for use by or in connection with a computer or any instruction execution system. Computer-usable or computer-readable media may include any device that stores, communicates, propagates, or transfers programs for use by or in connection with an instruction execution system, apparatus, or device. The medium may be magnetic, optical, electronic, electromagnetic, infrared, or semiconductor systems (or apparatus or devices), or propagation media. The medium may include computer-readable storage media such as semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random-access memory (RAM), read-only memory (ROM), rigid magnetic disks, and optical disks.

[0103] Each computer program can be substantially stored in a machine-readable storage medium or device (e.g., program memory or magnetic disk) that is readable by a general-purpose or special-purpose programmable computer, in order to configure and control the operation of the computer when the storage medium or device is read by the computer in order to perform the procedures described herein. The system of the present invention can also be considered to be implemented on a computer-readable storage medium configured with a computer program, in which case the configured storage medium causes the computer to operate in a specific predetermined manner to perform the functions described herein.

[0104] A data processing system suitable for storing and / or executing program code may include at least one processor directly or indirectly coupled to a memory element via a system bus. The memory element may include local memory, bulk storage, and cache memory that provides at least some temporary storage for the program code to reduce the number of times the code is retrieved from bulk storage during execution. Input / output or I / O devices (including, but not limited to, keyboards, displays, pointing devices, etc.) may be coupled to the system directly or via an intermediary I / O controller.

[0105] Network adapters can also be integrated into a system to enable a data processing system to connect to other data processing systems or remote printers or storage devices via an intervening private or public network. Modems, cable modems, and Ethernet cards are just a few of the types of network adapters currently available.

[0106] As used herein, the terms “hardware processor subsystem” or “hardware processor” can refer to a processor, memory, software, or combination thereof that works together to perform one or more specific tasks. In useful embodiments, a hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements may be contained in a central processing unit, a graphics processing unit, and / or a separate processor- or arithmetic element-based controller (e.g., logic gates, etc.). A hardware processor subsystem may include one or more onboard memories (e.g., caches, dedicated memory arrays, read-only memory, etc.). In some embodiments, a hardware processor subsystem may include one or more memories (e.g., ROM, RAM, Basic Input / Output System (BIOS), etc.) that may be onboard or offboard, or that may be dedicated for use by the hardware processor subsystem.

[0107] In some embodiments, a hardware processor subsystem may include and execute one or more software elements. These software elements may include an operating system and / or one or more applications and / or specific code to achieve a specified result.

[0108] In other embodiments, the hardware processor subsystem may include dedicated circuits that perform one or more electronic processing functions to achieve a specified result. Such circuits may include one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs).

[0109] These and other variations of the hardware processor subsystem are also intended in accordance with embodiments of the present invention.

[0110] In this specification, any reference to “one embodiment” or “one embodiment” of the present invention, and to other modifications, means that certain features, structures, properties, etc., described in relation to the embodiments are included in at least one embodiment of the present invention. Therefore, expressions such as “in one embodiment” or “in one embodiment” appearing elsewhere in this specification, and any other modifications, do not necessarily all refer to the same embodiment. However, it should be understood that, considering the teachings of the present invention provided herein, features of one or more embodiments can be combined.

[0111] For example, in the case of "A / B," the use of any of the following " / ," "and / or," or "at least one," such as "A and / or B" or "at least one of A and B," will be understood as intended to include the selection of only the first listed option (A), only the second listed option (B), or both options (A and B). As further examples, in the case of "A, B, and / or C" and "at least one of A, B, and C," such expressions are intended to include the selection of only the first listed option (A), only the second listed option (B), only the third listed option (C), only the first and second listed options (A and B), only the first and third listed options (A and C), only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended as many times as there are listed items.

[0112] The foregoing is to be understood in all respects to be illustrative and not restrictive, and the scope of the invention disclosed herein is to be determined not from the detailed description but from the claims as interpreted in accordance with the full width permitted by patent law. The embodiments shown and described herein are merely illustrative of the invention, and those skilled in the art should understand that various modifications can be implemented without departing from the scope and spirit of the invention. Those skilled in the art can implement various other combinations of features without departing from the scope and spirit of the invention. Thus, while aspects of the invention have been described with the detail and specificity required by patent law, what is claimed and intended to be protected by the patent is as stated in the appended claims.

Claims

1. A computer implementation method for optimizing a model for open vocabulary detection, To obtain domain proposals using a pre-trained visual language model and a pre-trained domain proposal network (110), Object feature predictions are obtained by using a supervising neural network trained using the aforementioned domain proposal (120), The object feature predictions that exceed the proposed threshold are filtered (130) to obtain pseudo-labels, By utilizing the proposed domain, ground truth class labels, ground truth bounding boxes, and pseudo-labels, a student neural network having a split-fusion detection head is trained (140). The pseudolabels are optimized by reducing noise from the pseudolabels using the trained split-fusion detection head of the trained student neural network (150) to obtain optimized object detection, A computer implementation method comprising (180) performing an action on the scene layout using the optimized object detection described above.

2. The computer implementation method according to claim 1, further comprising periodically updating the teacher neural network with the parameters of the student neural network to self-train the teacher neural network and the student neural network.

3. The computer implementation method according to claim 1, further comprising optimizing the pseudolabel by dividing the detection head into closed branches and open branches to obtain an optimized pseudolabel.

4. The computer implementation method according to claim 3, wherein the closed branch is trained using a ground truth bounding box and a ground truth class label.

5. The computer implementation method according to claim 4, wherein the trained closed branch obtains a class loss for classification using cross-entropy loss.

6. The computer implementation method according to claim 4, wherein the trained closed branch obtains a box loss for localization using a box regression loss.

7. The computer implementation method according to claim 3, wherein the open branch is trained using the pseudo-label and the ground truth class label.

8. The computer implementation method according to claim 7, wherein the open branch obtains a class loss for classification using cross-entropy loss.

9. The computer implementation method according to claim 1, further comprising optimizing the pseudolabel by fusing the prediction scores of the closed branch and open branch of the detection head by calculating the geometric mean of the prediction scores during inference to obtain an optimized pseudolabel.

10. A system for optimizing models for open vocabulary detection, Memory (592) and, A domain proposal (510) is obtained by using a pre-trained visual language model (504) and a pre-trained domain proposal network (509). Object feature prediction (516) is obtained by using a supervising neural network (511) trained using the domain proposal (510). Object feature predictions (516) that exceed the proposed threshold (517) are filtered to obtain pseudo-labels (518). A student neural network (522) having a split-fusion detection head (523) is trained using the aforementioned domain proposal (510), ground truth class label (520), ground truth bounding box (519), and pseudo-label (518). The pseudo-labels (518) are optimized by reducing noise from the pseudo-labels (518) using the trained split-fusion detection head (523) of the trained student neural network (522) to obtain optimized object detection (541). The system is configured to perform an action (610) on the scene layout (608) using the optimized object detection (541), A system comprising one or more processor devices (594) that communicate with the memory (592).

11. The system according to claim 10, further comprising periodically updating the teacher neural network with the parameters of the student neural network to self-train the teacher neural network and the student neural network.

12. The system according to claim 10, further comprising optimizing the pseudolabel by dividing the detection head into closed branches and open branches to obtain an optimized pseudolabel.

13. The system according to claim 12, wherein the closed branch is trained using a ground truth bounding box and a ground truth class label.

14. The system according to claim 13, wherein the trained closed branch obtains a class loss for classification using cross-entropy loss.

15. The system according to claim 13, wherein the trained closed branch obtains a box loss for localization using a box regression loss.

16. The system according to claim 12, wherein the open branch is trained using the pseudo-label and the ground truth class label.

17. The system according to claim 16, wherein the open branch obtains a class loss for classification using cross-entropy loss.

18. The system according to claim 10, further comprising optimizing the pseudolabel by fusing the prediction scores of the closed and open branches of the detection head by calculating the geometric mean of the prediction scores during inference to obtain an optimized pseudolabel.

19. A non-temporary computer program product comprising a computer-readable storage medium containing program code for optimizing a model for open vocabulary detection, wherein the program code, when executed on a computer, To obtain domain proposals using a pre-trained visual language model and a pre-trained domain proposal network (110), Object feature predictions are obtained by using a supervising neural network trained using the aforementioned domain proposal (120), The object feature predictions that exceed the proposed threshold are filtered (130) to obtain pseudo-labels, By utilizing the proposed domain, ground truth class labels, ground truth bounding boxes, and pseudo-labels, a student neural network having a split-fusion detection head is trained (140). The pseudolabels are optimized by reducing noise from the pseudolabels using the trained split-fusion detection head of the trained student neural network (150) to obtain optimized object detection, The pseudolabels are optimized by reducing noise from them using the trained split-fusion detection head of the trained student neural network (150), and the predicted scores from the closed branch and the open branch are fused at inference by calculating the geometric mean of the predicted scores to obtain optimized object detection. A non-temporary computer program product that causes the computer to perform an action (180) on the scene layout using the optimized object detection described above.

20. The non-temporary computer program product according to claim 19, further comprising periodically updating the teacher neural network with the parameters of the student neural network to self-train the teacher neural network and the student neural network.

Images