System and method for active testing based on visual transducers

Abstract

The present disclosure relates to systems and methods for active testing based on visual transformers for label efficient evaluation of visual tasks. One method includes dividing an input image into a plurality of patches using a visual transformer, where each patch corresponds to a different region of the input image. The input image is defined using model outputs of a vision model. The method further includes defining a plurality of position embeddings using the visual transformer, the plurality of position embeddings including a position embedding for each of the plurality of patches and for the input image as a whole, labeling identified regions of the original image based on an estimated loss map to define a labeled image; and outputting a test performance qualifier indicative of an expected performance of the vision model when the vision model is part of a vision system. The test performance qualifier is computed using a weighted analysis based on an image loss level and a region loss level for each patch the image is provided to be labeled.

Description

Technical Field

[0001] Various aspects of this disclosure generally relate to systems and methods for active testing of labels for efficient evaluation of visual models. Background Technology

[0002] Computer vision systems acquire and analyze digital images to understand the visual content captured within them. These systems can perform various processes to output specific measurements, extract specific features from the visual content, and / or provide decision operations (e.g., pass-fail decision when inspecting for defects in an object; object identification; and / or labeling of object types in the visual content). In a non-limiting example, the computer vision system is configured to detect, classify, and / or identify objects in the visual content and can be used in a variety of applications, such as, but not limited to, surveillance systems, autonomous vehicles, and manufacturing processes. In another example, the computer vision system is configured to perform specific operations, such as, but not limited to, image segmentation, dividing the pixels of a digital image into discrete groups.

[0003] Machine learning techniques have played a crucial role in developing computer vision systems, for example, by training models using large, annotated datasets. However, generating accurate and detailed annotated datasets can be slow and expensive. To improve efficiency and reduce costs, active testing can be used to develop computer vision systems for label-efficient model evaluation, where the goal is to estimate the performance of the vision model on an entire unlabeled test dataset using a limited annotation budget.

[0004] Active testing focuses on accurate estimation of loss values, providing a more accurate understanding of the loss distribution across all instances. In some implementations, one goal of active testing is to select a subset of a large unlabeled test dataset using an acquisition function, then label the selected dataset and use it to accurately estimate the performance of the visual model across the entire dataset. For example, the Active Alternate Estimator (ASE) uses a weighted cognitive uncertainty score estimated by an ensemble model to effectively pinpoint information instances. A key feature of ASE is iteratively updating the ensemble model with newly acquired labels, which helps reduce overconfidence and enhances predictions for unknown test data. Summary of the Invention

[0005] In one form, this disclosure relates to a method for actively testing a visual model that will be used as part of a vision system to recognize visual content of an original image. The method includes dividing an input image into multiple patches using a visual transformer, where each patch corresponds to a different region of the input image, wherein the input image is defined using the model output of the visual model. The method also includes defining multiple location embeddings using the visual transformer, the multiple location embeddings including location embeddings for each of the multiple patches and for the input image as a whole, and providing an estimated loss map for the original image based on an image loss level and a region loss level for each patch, wherein the image loss level and the region loss level for each patch are estimated using the multiple location embeddings and a transformer encoder-multilayer perceptron of the visual transformer. The method further includes labeling one or more identified regions of the original image based on the estimated loss map to define a labeled image, and outputting a test performance qualifying parameter indicating the expected performance of the visual model when the visual model is part of a vision system. The test performance qualifying parameter is calculated using weighted analysis based on the image loss level and the region loss level for each patch, each patch being provided with a labeled image.

[0006] In one form, this disclosure relates to a system for actively testing a visual model that will be used as part of a vision system to recognize visual content of an original image. The system includes one or more hardware computing devices configured to use a visual transformer to divide an input image into multiple patches, each patch corresponding to a different region of the input image, wherein the input image is defined using model output. The one or more hardware computing devices are also configured to use the visual transformer to define multiple location embeddings, including location embeddings for each patch and for the input image as a whole, and to provide an estimated loss map of the original image based on an image loss level and a region loss level for each patch, wherein the image loss level and the region loss level for each patch are estimated using the multiple location embeddings and a transformer encoder-multilayer perceptron of the visual transformer. The one or more hardware computing devices are also configured to provide one or more recognition regions of the original image labeled based on the estimated loss map to define the labeled image, and, when the visual model is part of a vision system, output a test performance qualifier indicating the expected performance of the visual model. The test performance qualifier is calculated based on a weighted analysis of the image loss level and the region loss level for each patch, which is provided with a labeled image. Attached Figure Description

[0007] Figure 1 This is an illustrative example of a vehicle using an object classification model tested with an Active Vision Model Evaluation (AVME) system; Figure 2 This is a block diagram of the AVME system; Figure 3 yes Figure 2 A block diagram of the Visual Model Under Test (VMUT) and Visual Model Evaluation (VMA) modules of the AVME system; Figure 4 This is a block diagram of the AVME system, where VMUT ​​is the segmentation model; Figure 5 This is a block diagram of the AVME system, where VMUT ​​is the object detection model; Figure 6A This is a block diagram of the visual metadata model of the VMA module, where VMUT ​​is the segmentation model; Figure 6B This is a block diagram of the visual metadata model used in the VMA module, where VMUT ​​is the object detection model; and Figure 7 This is a flowchart of an example active vision model evaluation routine. Detailed Implementation

[0008] This document describes embodiments of the present disclosure. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various alternative forms. The drawings are not necessarily drawn to scale; some features may be enlarged or minimized to show details of specific components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to use the embodiments in various ways. As will be understood by those skilled in the art, various features illustrated and described with reference to any of the drawings may be combined with features illustrated in one or more other drawings to produce embodiments not explicitly illustrated or described. The combinations of illustrated features provide representative embodiments of typical applications. However, for a particular application or implementation, various combinations and modifications of features consistent with the teachings of this disclosure may be required.

[0009] While Active Alternate Estimator (ASE) and other active testing methods (such as Active Test Alternate (AST)) evaluate image classification models, their effectiveness may be limited in dense recognition tasks (such as segmentation and object detection) due to the inherent challenges in the instance and / or label acquisition phase.

[0010] For example, regarding instance acquisition, deep ensemble models are often impractical due to their high computational cost and the challenge of achieving sufficient diversity within ensemble models used for dense recognition tasks. Furthermore, updating these models with only a few newly labeled instances per iteration may provide insufficient information for retraining. Additionally, from a label acquisition perspective, the iterative process increases communication overhead between researchers and annotators. Moreover, previous methods involving labeling the entire image are inefficient for tasks where only specific regions (e.g., boundaries and blurred regions) carry the majority of the test error.

[0011] In one form, this disclosure provides a system and / or method for an Active Visual Model Evaluation (AVME) system with efficient label evaluation to identify highly informative images or regions by estimating the loss of an unlabeled test dataset in a single pass. The AVME may be configured to include a visual metadata model with a visual transformer to address long-term dependencies linking small, critical regions such as object boundaries, which can lead to errors in dense recognition tasks. In one form, the AVME system of this disclosure is configured to divide an input image into multiple patches, each patch corresponding to a different region of the input image, wherein the input image is defined using a Visual Model Under Test (VMUT). The AVME system may be further configured to define multiple location embeddings, including location embeddings for each patch and for the input image as a whole, and to use a multiple location embedding and transformer encoder to estimate the image loss level and the region loss level for each patch. The AVME system identifies high-loss regions in the input image to be labeled and, based on the image loss level and the region loss level for each patch, uses weighted analysis to output a performance qualifyer for the Visual Model Under Test. Among other features, AVME systems with metadata models can identify highly informative images or regions by estimating the loss of the unlabeled test dataset in a single pass.

[0012] refer to Figure 1In the example application, the vision system 100 of vehicle 104 includes an object classification model 106, wherein the vision system 100 is tested using at least in part the AVME system of this disclosure. Vehicle 104 includes one or more cameras 108 arranged around vehicle 104 to capture one or more images 110 of the area surrounding vehicle 104. Image 110 may include pixel data captured by camera 108. For example, camera 108 may be a 2D sensor configured to capture image pixels at various resolutions (e.g., standard definition (SD), high definition (HD), full high definition, ultra-high definition (UHD), 4K, etc.), dynamic range (8 bits, 10 bits, or 12 bits per pixel per color, etc.), and the frequency and number of color channels (e.g., infrared, red-green-blue (RGB), black and white, etc.). Camera 108 may also include a 3D sensor, such as a LiDAR sensor. The LiDAR sensor may be configured to generate point clouds of various distance points. When these points are detected, the LiDAR scanner emits brief pulses of light that are reflected back to the LiDAR sensor by various objects. The travel time of these returned pulses is used to calculate the distance between the lidar sensor and the object. Regardless of the format, image 110 is processed by vision system 100 to detect and applicably identify objects in image 110 (e.g., dog 112) using object classification model 106.

[0013] In one embodiment, by detecting and identifying objects, vehicle 104 can perform certain operations to monitor the position of the object relative to vehicle 104 and / or take certain actions, such as stopping vehicle 104 or using one or more human-machine interfaces in vehicle 104 to warn passengers about the object. For example, if the object classifier identifies dog 112 in image 110, vehicle 104 can monitor the position of dog 112 relative to vehicle 104 and identify that dog 112 may move toward vehicle 104.

[0014] While specific implementations are provided, the AVME system disclosed herein is configured to test other types of vision systems and should not be limited to the examples provided herein. In a non-limiting example, the vision system may include segmentation features for monitoring autonomous vehicles and / or as part of biometric detection for individual identification in a security system. Another specific implementation includes using the vision system as a control system to determine or identify actuation signals based on decisions made. In a non-limiting example, the vision system is configured to determine whether a manufactured part is defective and, based on whether the part is defective or nominal, output an actuation signal to cause the manufactured part to travel in a specified direction on a conveyor system. In yet another example, as part of a security system, the vision system is configured to determine whether a product has been removed from an establishment (e.g., a painting removed from a museum) and output an actuation signal to activate one or more security protocols (e.g., using a loudspeaker to sound an alarm, notify security personnel, and / or cordon off an area). The actuation signal may be provided or transmitted to a controlled system. The actuation signal may be used to activate and / or control the controlled system.

[0015] refer to Figure 2 The AVME system 200 includes a Visual Model Under Test (VMUT) 202 and a Visual Model Evaluation (VMA) module 204. The VMUT ​​is a visual model used as part of the vision system to recognize the visual content of the original image. In a non-limiting example, VMUT ​​202 is a segmentation model and / or an object detection model. The visual model (f) maps the input (x ∈ X) to the corresponding label. Without any assumptions about VMUT ​​202, VMA module 204 is configured to use the predicted output of VMUT ​​202 to estimate the expected loss of the predicted output as a performance qualifier (PQ).

[0016] In one form, reference Figure 3 The test dataset 300 includes multiple test images 302 provided as raw images, which are unlabeled. The test dataset 300 is provided to VMUT ​​202 for processing to generate model output 304. For example, refer to... Figure 4 VMUT 400 is provided as the segmentation model, and the model output 304 includes the segmented image 402 and the entropy image 404. In another example, refer to... Figure 5 VMUT 500 is provided as an object detection model. The model output 304 of the object detection model includes an object query 503, which provides features extracted from image 302 and is used to generate an object identifier image 504, which has bounding boxes 506 highlighting detected objects and / or provides data for the classification of detected objects.

[0017] Continue to refer to Figure 3 In one form, the VMA module 204 is configured to include a visual metadata model 310, a high-loss image filter (HLIF) 312, a label annotation process 313, and a subsample performance estimator 314.

[0018] The visual metadata model 310 is configured to reduce the high variance associated with the limited labels in active testing and to predict the loss for dataset 300. The visual metadata model 310 processes the input image defined by model output 304, and in some applications, the test image 302 (e.g., the original image) to define the estimated loss of the input image (Figure 316).

[0019] In the non-restrictive example, Figure 4 In the case where VMUT ​​400 is a segmentation model, the input image of the visual metadata model 310 includes a test image 302 and a model output 304 having a segmentation image 402 and an entropy image 404. In one form, the test image 302, the segmentation image 402 (e.g., a category prediction distribution image), and the entropy image 404 are linked or stitched together to form the input image.

[0020] In another example, Figure 5 In the case of VMUT ​​500 being an object detection model, the input image includes object query 503. Typically, for segmentation models, a prediction is provided for each pixel, so the entire image is usually provided as part of the input image. For object detection models, predictions are provided for specific regions, so the entire image may not be needed as part of the input.

[0021] refer to Figure 3 In one form, the visual metadata model 310 is configured to include a visual transformer (ViT) 318. The ViT 318 handles long-term dependencies in the image by relating these regions to a broader image context. In a non-limiting example, refer to... Figure 6A The ViT 318 includes an image inpainting process 602, a linear projection process 604, and a transducer encoder. The multilayer perceptron (TE-MLP) 606 has a transducer encoder 606A and a multilayer perceptron 606B following the encoder 606A.

[0022] In the non-restrictive example, VMUT ​​400 is a segmentation model (e.g., Figure 4 In the case of an image inpainting process 602, the image inpainting process 602 is configured to divide / wrap the input image into multiple patches 610, where each patch corresponds to a different region of the input image.

[0023] The linear projection process 604 is configured to define a plurality of position embeddings 612, including a position embedding for the input image as a whole (e.g., image position embedding 612A) and a position embedding 610 for each patch 610 (e.g., patch position embedding 612B). In a non-limiting example, the linear projection process 604 performs a linear projection to map the patch 610 to a D-dimensional token, and then adds the position embedding to the patch token, which is provided as input to the transformer encoder 606A. The image position embedding 612A is provided as a category embedding at position 0, and the patch position embeddings 612B for the plurality of patches 610 are at positions 1 to E, where E is equal to the number of patches (e.g., 16 patches).

[0024] The TE-MLP606 is configured to use multiple location embeddings 612 to estimate the image loss level 614A and the region loss level 614B for each patch. Labeling the entire test image 302 may not be necessary and can be resource-intensive, especially for segmentation and object detection models. To facilitate region level selection and annotation, the MLP 606B estimates the region loss level 614B for each patch of the input image. The image loss level 614A and the region loss level 614B are used to determine which regions of the test image 302 will be labeled.

[0025] ViT 318 processes the input image of the object detection model (e.g., VMUT500) in a similar manner to that of a segmentation model (e.g., VMUT400) to generate an estimated loss map 316. In a non-limiting example, Figure 6B The figure shows the output of the ViT 318 object detection model. Here, the ViT 318 does not include the image inpainting process 602, but instead receives object queries 503 from the VMUT ​​500 and provides them to the linear projection process 604.

[0026] In one form, the visual metadata model 310 is tested using a ground truth loss map that provides the actual loss values ​​to be predicted by the visual metadata model 310. For example, the metadata model training process 650 uses training labels (Y) that are provided to VMUT ​​202 to output labels 645. TRAIN The associated training graph input (X) TRAIN The labeled dataset (D) TRAIN (X TRAIN Y TRAIN At least a portion of the ground truth map 652 is used to generate the ground truth map. Since small and challenging regions disproportionately affect the test error in the visual model, a focus loss technique can be used to mitigate this disproportion. The ground truth map is used to adjust the loss for ViT 318.

[0027] In the non-restrictive example, Figure 6A In the process, label 654 and the output of VMUT ​​400 (e.g., entropy image 404) are used to determine the ground truth 652 of the segmentation model. During training, the visual metadata model 310 receives the output of VMUT ​​400 from the metadata model training 650.

[0028] In another example, Figure 6B The figure illustrates metadata model training 670, where VMUT ​​500 provides object query 503, which provides object identifier image 504. Ground truth 672 is determined using at least VMUT ​​500's object query 503 and the predefined labeled image 674.

[0029] In the illustrative example, during the training of the visual metadata model 310, data from D... TRAIN A set of labeled datasets (e.g., , where b=1, ..., B), are processed by VMUT ​​202 to obtain the model output. The total loss of the visual metadata model 310 is provided by the following Equation 1, where: " "r" represents the trained visual metadata model; "r" represents the region feature; and "f" is a function of VMUT. It is the loss function of the visual metadata model; " is the loss of VMUT. The region features (r) used for the segmentation model are provided as r = [x, f(x), entropy (f(x))] and for the object detection model, the region features are provided as r = query features from f(x).

[0030] Equation 1..... .

[0031] For the training dataset (D) TRAIN The loss is estimated for all instances in the training dataset. For the image loss level, N = S, where "S" is the number of images in the training dataset and an instance is an image. For the region loss level, "N" is the number of regions considered in all images, and each instance is a region in an image. During training, for n = 1 to S, "N" will be used to estimate the loss for all instances in the training dataset. Add to "q", where "q" is the distribution. In the case where "M" is the annotation budget, select an instance (i) with a probability defined by distribution q. m ; where m = 1 to M), and instances are added to the training dataset to obtain observations of the training dataset. Then, labels are applied to all instances in the observed training dataset.

[0032] In a non-restricted example, during the training of the visual metadata model 310, the transformer encoder 606A and MLP606B are trained simultaneously to accurately provide the image loss level and region loss level, respectively. The transformer encoder can generate a category for the entire image. E categories are generated for all regions. During inference, these categories can then be converted back to numerical values. The loss function of TE-MLP 606 can be represented by Equation 2 below, where: "FL" is the focusing loss function; E is "c0", and... It represents the ground truth and predicted class distribution of the entire image; and It represents the ground truth and predicted class distribution for each region.

[0033] Equation 2..... .

[0034] For each test image 302, the visual metadata model 310 outputs an estimated loss map 316, which includes an image loss level 614A and a region loss level 614B for each patch. In a non-limiting example, for Figure 4 The segmentation model in Figure 516 provides an example of estimated loss, and for Figure 5 The object detection model in the example provides an estimated loss graph 516. In the estimated loss graph 516, different dashed lines represent different detected objects with losses.

[0035] A High Loss Image Filter (HLIF) 312 is configured to filter out the estimated loss map 316 with low loss, which may be a large portion of the test image 302. Small regions, such as boundaries and blurred areas, carry most of the test error, and HLIF 312 is configured to identify high loss regions, or in other words, highly blurred regions (e.g., informational regions), for further processing. In a non-limiting example, HLIF 312 is configured to remove or discard the test image 302 associated with the estimated loss map 316, which has a loss level less than or equal to a loss threshold. The remaining image 302 and the associated estimated loss map 316 are provided as a selected high loss map 319 in the labeling process 313.

[0036] During the labeling process 313, portions of the test image 302 corresponding to regions with high loss levels 614B (e.g., loss greater than or equal to a loss threshold) are labeled (e.g., by an individual). In a non-limiting example, refer to Figure 4 For the segmentation model, the labeled image 410 has already identified the regions 412 to be labeled. As illustrated, these regions correspond to the boundaries at the region loss level 614B. Similarly, refer to... Figure 5Using an object detection model, the labeled image 510 is provided with a recognition region 512 for the labeling by the individual.

[0037] Subsample performance estimator 314 is configured to offset potential biases from visual metadata model 310 and is configured to output a (test) performance qualifier (PQ) 322 for VMUT ​​202 using weighted analysis based on image loss level and region loss level for each patch. In one form, the performance qualifier 322, which may also be referred to as risk or prediction loss, is a loss function value about new, unseen data to indicate or measure how well VMUT ​​202 is expected to perform in practice when making predictions on data not used during training. In one example, VMUT ​​202 can be applied for use if the performance qualifier 322 indicates performance above a predefined performance threshold. In one example, this may include storing VMUT ​​202 in the memory of the vision system of vehicle 104, for example, for analyzing pixel data in image 110 captured by sensor 108 of vehicle 104. For example, the vision system may act as a control system to determine or identify actuation signals based on decisions made.

[0038] In a non-limiting example, the subsample performance estimator 314 uses data from the labeled image 320 to determine the PQ 322, including the image loss level and the region loss level for each patch. To mitigate bias that may be introduced by selecting images with high-loss maps 319 that are labeled in the labeled image 320, the subsample performance estimator 314 is configured to employ a hierarchical unbiased risk estimator (LURE) technique to reduce selection bias by correcting weights, thereby determining the performance qualifier 322. Furthermore, the capability of the LURE technique can be extended to variance reduction, assuming it is based on importance sampling, a technique defined as variance reduction.

[0039] In a non-restrictive example, using the LURE technique, the subsampling performance estimator 314 is represented by the following Equation 3, where It is PQ 322 and It is the prediction loss for a given image or region.

[0040] Equation 3..... .

[0041] Given an unlabeled test dataset, the VMA module 204 with the visual metadata model 310 can accurately predict the loss for all instances (e.g., images or regions). The visual metadata model utilizes the output of VMUT ​​202 to provide identification of high-information (high-loss) instances to reduce variance from methods such as random sampling. However, directly selecting high-information instances is risky. Labeling and calculations can introduce high bias because these instances may be considered "bad cases," potentially jeopardizing the entire test dataset. (e.g., PQ 322) is overestimated. To mitigate potential bias introduced by selecting high-information-content instances, VMA module 204 uses a subsample performance estimator 320 to calculate risk using a weighted average based on the loss distribution predicted by visual metadata model 310. (For example, PQ 322).

[0042] refer to Figure 7 The example active visual model evaluation routine 700, performed by the AVME system 200 of this disclosure, is used to actively test a visual model (e.g., VMUT ​​202) to be used as part of a visual system to identify the visual content of the original image.

[0043] In operation 702, the AVME system 200 is configured to divide the input image into multiple patches, each patch corresponding to a different region of the input image. The input image is defined using the model output 304 of VMUT ​​202. In one form, the AVME system 200 employs a visual metadata model 310 with ViT 318 to segment the input image.

[0044] In operation 704, the AVME system 200 uses, for example, ViT 318 to define multiple location embeddings, which include location embeddings for each of multiple patches and for the input image as a whole.

[0045] In operation 706, the AVME system 200 provides an estimated loss map of the original image based on the image loss level and the region loss level for each patch. In a non-limiting example, multiple location embeddings and a TE-MLP 606 with ViT 318 are used to estimate the image loss level and the region loss level for each patch.

[0046] In operation 708, the AVME system 200 uses weighted analysis based on the image loss level and the region loss level for each patch to output a performance qualifier 322 for the VMUT ​​202. In one example, if the performance qualifier 322 indicates that the performance is above a predefined threshold, the VMUT ​​202 can be applied for use. In one example, this may include storing the VMUT ​​202 in the memory of the vision system of the vehicle 104, for example, for analyzing pixel data in the image 110 captured by the sensors 108 of the vehicle 104. For example, the vision system may function as a control system to determine or identify actuation signals based on decisions made.

[0047] In operation 710, the AVME system 200 labels one or more identified regions of the original image based on the estimated loss map. In a non-limiting example, portions of the original image associated with loss levels in high-loss regions are labeled, while portions with low loss are left unlabeled.

[0048] While exemplary embodiments have been described above, this does not mean that these embodiments describe all possible forms of the invention. Rather, the language used in this specification is descriptive and not restrictive, and it should be understood that various changes can be made without departing from the spirit and scope of the invention. Furthermore, features of various implementation embodiments can be combined to form other embodiments of the invention.

[0049] In a non-limiting example, the AVME system 200 may include: hardware computing devices; application-specific integrated circuits (ASICs); digital, analog, or mixed-signal / digital discrete circuits; digital, analog, or mixed-signal / digital integrated circuits; combinational logic circuits; field-programmable gate arrays (FPGAs); processor circuitry (shared, dedicated, or grouped) for executing code; memory circuitry (shared, dedicated, or grouped) for storing code executed by the processor circuitry; other suitable hardware components providing the aforementioned functionality; or combinations of some or all of the above, such as in a system-on-a-chip.

[0050] The term memory or memory circuitry can be a subset of the term computer-readable medium. As used herein, the term "computer-readable medium" does not include transient electrical or electromagnetic signals propagating through a medium (e.g., on a carrier wave); therefore, the term computer-readable medium can be considered tangible and non-transitory. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory circuitry (e.g., flash memory circuitry, erasable programmable read-only memory circuitry, or mask read-only circuitry), volatile memory circuitry (e.g., static random access memory circuitry or dynamic random access memory circuitry), magnetic storage media (e.g., analog or digital magnetic tape or hard disk drives), and optical storage media (e.g., USB, CD, DVD, or Blu-ray disc).

[0051] The AVME system 200 described in this application may be implemented, in part or in whole, by a dedicated computer created by configuring a general-purpose computer to perform one or more specific functions embodied in a computer program. The components used in the AVME system 200 may be provided in a single device or distributed across multiple devices communicating using wireless communication (e.g., cellular networks, WiFi networks, Bluetooth, etc.) and / or wired communication.

[0052] As used herein, the phrase at least one of A, B, and C should be interpreted as meaning the logic of using the non-exclusive logical "or" (A or B or C), and should not be interpreted as meaning "at least one of A, at least one of B, and at least one of C".

[0053] The description in this disclosure is merely exemplary in nature, and therefore, variations that do not depart from the spirit and scope of this disclosure are intended to be within its scope. Such variations should not be considered as departing from the spirit and scope of this disclosure.

Claims

1. A method for actively testing a visual model, which is used as part of a vision system to identify visual content of an original image, the method comprising: The input image is divided into multiple patches using a visual transformer, where each patch corresponds to a different region of the input image. The input image is defined using the model output of the visual model. Multiple location embeddings are defined using a visual transformer, which includes location embeddings for each of the multiple patches and for the input image as a whole. An estimated loss map of the original image is provided based on the image loss level and the region loss level of each patch. A transformer encoder-multilayer perceptron with multiple location embeddings and visual transformers is used to estimate the image loss level and the region loss level of each patch. Based on the estimated loss map, one or more identified regions of the original image are labeled to define the labeled image; When the visual model is part of a vision system, the output is a test performance qualifier indicating the expected performance of the visual model. This test performance qualifier is calculated using weighted analysis based on the image loss level and the region loss level for each patch, which is provided with a labeled image. and Visual models are used in vision systems to detect and / or identify visual content in pixel data captured by the vehicle's sensors.

2. The method according to claim 1, wherein, The positional embedding of the input image is provided as a category embedding at position 0, and the positional embeddings of the plurality of patch blocks are at positions 1 to E, where E is equal to the number of patch blocks.

3. The method according to claim 1, wherein, The model output includes an entropy image and a segmentation image.

4. The method according to claim 3 further includes stitching together the input image into an original image, an entropy image, and a segmented image.

5. The method according to claim 1, wherein, The model output includes a set of object queries and object identifier images, the object identifier images having one or more bounding boxes to identify the objects provided in the set of object queries.

6. The method of claim 1, further comprising labeling at least a portion of the original image in response to an estimated loss map associated with the original image being greater than or equal to a loss threshold.

7. The method according to claim 6, wherein, At least a portion of the labeled original image is associated with one or more patches having a regional loss level greater than or equal to a loss threshold.

8. The method of claim 6, further comprising discarding the labeling of the original image in response to an estimated loss map associated with the original image being less than or equal to a loss threshold.

9. The method according to claim 1, wherein, The original image was selected from multiple test images.

10. The method of claim 1, further comprising training a first visual transformer using a training dataset of images with multiple labels using ground truth data provided by the visual model; and outputting the first visual transformer as a trained visual transformer model for testing the visual model using unlabeled raw images.

11. A system for actively testing a visual model, the visual model being used as part of a visual system to identify visual content of an original image, the system comprising: One or more hardware computing devices are configured to: The input image is segmented into multiple patches using a visual transformer, where each patch corresponds to a different region of the input image, and the input image is defined using the model output. Multiple location embeddings are defined using a visual transformer, which includes location embeddings for each of the multiple patches and for the input image as a whole. An estimated loss map of the original image is provided based on the image loss level and the region loss level of each patch. A transformer encoder-multilayer perceptron with multiple location embeddings and visual transformers is used to estimate the image loss level and the region loss level of each patch. Provide one or more recognition regions of the original image based on the estimated loss map to define the labeled image; When the visual model is part of a vision system, the output is a test performance qualifier indicating the expected performance of the visual model. This test performance qualifier is calculated using weighted analysis based on the image loss level and the region loss level for each patch, which is provided with a labeled image. and Visual models are used in vision systems to detect and / or identify visual content in pixel data captured by vehicle sensors.

12. The system according to claim 11, wherein, The positional embedding of the input image is provided as a category embedding at position 0, and the positional embeddings of multiple patches are provided at positions 1 to E, where E is equal to the number of patches.

13. The system according to claim 11, wherein, The model output includes an entropy image and a segmentation image.

14. The system of claim 13, wherein the one or more hardware computing devices are further configured to form the input image as a stitched image of the original image, the entropy image, and the segmented image.

15. The system according to claim 11, wherein, The model output includes a set of object queries and object identifier images, the object identifier images having one or more bounding boxes to identify the objects provided in the set of object queries.

16. The system according to claim 11, wherein, The one or more hardware computing devices are also configured to label at least a portion of the original image in response to an estimated loss map associated with the original image being greater than or equal to a loss threshold.

17. The system according to claim 16, wherein, At least a portion of the labeled original image is associated with one or more patches having a regional loss level greater than or equal to a loss threshold.

18. The system according to claim 16, wherein, The one or more hardware computing devices are also configured to discard the labeling of the original image in response to an estimated loss map associated with the original image being less than or equal to a loss threshold.

19. The system according to claim 11, wherein, The original image was selected from multiple test images.

20. The system according to claim 11, wherein, The one or more hardware computing devices are also configured to train a first visual transformer using a training dataset of images with multiple labels, based on ground truth data provided by the visual model. The first visual transformer is output as a trained visual transformer model, which is used to test the visual model using unlabeled raw images.

Images