Apparatus and computer-implemented method for evaluating digital images
Through the pre-trained feature and quantitative normalization processing of semantic segmentation network model, quantization errors are calculated to evaluate semantic segmentation of digital images, and the accuracy and efficiency of semantic segmentation and abnormal detection in autonomous vehicles and automatic optical inspection equipment is solved, and the accuracy and efficiency of semantic segmentation and abnormal detection in autonomous vehicles and automatic optical inspection equipment is realized. Efficient abnormality detection.
Patent Information
- Application Number
- CN202410697629.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-06-01
- Filing Date
- 2024-05-31
- Publication Date
- 2024-12-03
AI Technical Summary
Existing digital image evaluation methods are difficult to effectively perform semantic segmentation and abnormality detection in autonomous vehicles and automatic optical inspection equipment, especially in the evaluation of feature quantization errors.
Using a pre-trained semantic segmentation network model, quantization errors are calculated through normalization of features and quantization to evaluate semantic segmentation of digital images, quantization errors are used to detect abnormalities, and quantization sets and denominations are determined without modifying the original model. Integrate parameters to improve the accuracy and efficiency of evaluation.
It realizes efficient semantic segmentation and abnormal detection of digital images, improves the evaluation accuracy and efficiency in autonomous vehicles and automatic optical inspection equipment, and can detect abnormalities without modifying the original model, which is suitable for safety-related applications.
Smart Images

Figure CN119068292A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an apparatus and a computer-implemented method for evaluating digital images. Background Art
[0002] Digital images are evaluated, for example, in the field of at least partially autonomous vehicles or automated optical inspection devices. Summary of the invention
[0003] The device and computer implemented method for evaluating a digital image according to the independent claims provide a simple score for classifying a digital image.
[0004] The method comprises providing a digital image, providing a first portion of a predetermined model, wherein the predetermined model is configured to determine a semantic segmentation of the digital image using a second portion of the predetermined model, wherein the first portion is configured to determine a feature depending on the digital image, wherein the second portion is configured to determine the semantic segmentation depending on the feature, wherein the method comprises determining a feature depending on the digital image using the first portion, providing a quantization set for quantizing the feature, determining a quantization of the feature depending on the quantization set and depending on the feature, determining a quantization error depending on the feature and the quantization, and evaluating the digital image depending on the quantization error. The feature and the quantization may be vectors. The quantization error may be a distance between the vectors used as a simple score for evaluating the digital image.
[0005] The method may include providing a second portion of the model, wherein the second portion is configured to determine the semantic segmentation of the digital image in dependence on the quantization, and utilizing the second portion to determine the semantic segmentation of the digital image in dependence on the quantization.Providing a score in addition to the semantic segmentation allows for evaluating the semantic segmentation.
[0006] In one embodiment, providing a quantization set includes providing a reference for semantic segmentation of a digital image, determining the semantic segmentation of the digital image, and determining a quantization in the quantization set depending on a difference between the reference and the semantic segmentation and depending on a quantization error. The first portion and the second portion are predetermined portions of a predetermined model. The quantization is determined without modifying the first portion or the second portion.
[0007] In one embodiment, providing the pre-trained model includes training a first part to determine features, and training a second part to determine semantic segmentation depending on the features. The quantization set is inserted into the pre-trained model between the pre-trained first part and the pre-trained second part.
[0008] In one embodiment, the method includes determining a feature having a predetermined normalization, and determining a quantization of the feature having the predetermined normalization. The normalization normalizes the feature and the quantization in the same manner to facilitate determination of quantization error. The vectors representing the feature and the quantization may be normalized to unit length.
[0009] In one embodiment, the method includes upscaling features from a first scale to a second scale, determining quantization in the second scale based on the features in the second scale, downscaling the quantization from the second scale to the first scale, and determining a quantization error in the first scale. A change in resolution can improve accuracy.
[0010] Preferably, the feature is a vector and the quantization of the feature is a vector, wherein determining the quantization error comprises determining a cosine distance between the feature and the quantization of the feature.
[0011] Providing the digital image may include capturing the digital image using a camera, in particular, of an at least partially autonomous vehicle or automated optical inspection equipment.
[0012] Evaluating the digital image may include detecting an anomaly if the quantization error exceeds a threshold, or not detecting an anomaly otherwise.
[0013] The device for evaluating a digital image comprises at least one processor and at least one storage device for storing a digital image and instructions, the instructions, when executed by the at least one processor, causing the at least one processor to perform the method, wherein the at least one processor is configured to execute the instructions. The device provides advantages corresponding to those provided by the method.
[0014] A program may be provided, wherein the program includes instructions which, when executed by at least one processor, cause the at least one processor to perform the method. The program provides advantages corresponding to those provided by the method. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Further embodiments are derived from the following description and the accompanying drawings. In the drawings:
[0016] Figure 1 schematically depicts an apparatus for evaluating a digital image,
[0017] Figure 2 The steps of a method for evaluating a digital image are depicted. DETAILED DESCRIPTION
[0018] Figure 1 A device 100 for evaluating a digital image x is schematically depicted.
[0019] The digital images are, for example, video images, radar images, lidar images, ultrasound images, motion images, thermal images.
[0020] The apparatus 100 comprises at least one processor 102 and at least one storage device 104 for storing a digital image x and instructions.
[0021] The instructions, when executed by the at least one processor 102 , cause the at least one processor 102 to perform a method for evaluating a digital image x.
[0022] At least one processor 102 is configured to execute instructions.
[0023] The device 100 may include a camera 106 or an interface for receiving digital images from the camera 106 .
[0024] A program may include instructions.
[0025] The method for evaluating a digital image x comprises a step 202 .
[0026] In step 202, a digital image x is provided.
[0027] Providing the digital image x may include capturing the digital image x.
[0028] According to one example, the digital image x is captured using the camera 106 .
[0029] The camera 106 is, for example, part of or mounted to an at least partially autonomous vehicle or automated optical inspection apparatus.
[0030] The method includes step 204 .
[0031] In step 204, a predetermined model is provided
[0032] Pre-order model It includes a first part F and a second part D.
[0033] In this example, the predefined model It is a pre-trained semantic segmentation network, i.e., an artificial neural network.
[0034] Pre-order model For example, based on the data set X = {x i ,y i} i=1,...,N The data set X={x i ,y i} i=1,...,N Including the corresponding annotations y i ∈(0,...,N c -1) H×W N input images Where N c Is the network able to perform semantic segmentation The number of classes assigned in .
[0035] Part II.D covers the network The first part F includes the network other layers.
[0036] The first part is configured to determine a feature z depending on the digital image x e =F(x).
[0037] Pre-order model is configured to determine a semantic segmentation of the digital image x using the second part D
[0038] The second part D is configured to depend on the feature z e Determine semantic segmentation
[0039] Providing a pre-trained model may include training a first part F to determine the feature z e , and train the second part D to depend on the feature z e Determine semantic segmentation
[0040] The method includes step 206 .
[0041] Step 206 includes providing a method for quantizing the feature z e A predetermined set of quantization
[0042] The method includes step 208 .
[0043] In step 208, the first part F is used to determine the feature z depending on the digital image x. e .
[0044] Feature z e For example, it is determined using a predetermined normalization N.
[0045] The method includes step 210 .
[0046] In step 210, depending on the quantization set and depends on the feature z e To determine the feature z e Quantization of z q .
[0047] For feature z e Quantization of z q For example, using a predetermined normalization N to determine:
[0048] z q =Q(z e )
[0049] Quantized collection with normalization
[0050]
[0051] The method includes step 212 .
[0052] In step 212, the quantization error
[0053]
[0054] Depends on feature z e and quantization z q And sure.
[0055] According to an example, feature z e is a vector, and the characteristic z e Quantization of z q is a vector. In this example, feature z e With quantization z q have the same dimensions.
[0056] According to one example, determining the quantization error includes determining the characteristic z e and feature z e Quantization of z q The cosine distance between .
[0057] The method includes step 214 .
[0058] In step 214, the second portion D is used depending on the quantization z q To determine the semantic segmentation of the digital image x
[0059] In one example, semantic segmentation It is determined using the normalized quantization set Q:
[0060]
[0061] In one example, semantic segmentation By quantization set Sure:
[0062]
[0063] G represents the quantization set Or the normalized quantized set Q is added to the predetermined model The corrected model is obtained.
[0064] The second part D is configured to depend on the feature z e Quantization of z q To determine the semantic segmentation of the digital image x In this example, quantizing z q and feature z e The dimensions are the same.
[0065] The method includes step 216 .
[0066] In step 216, depending on the quantization error E quant Evaluate the digital image x.
[0067] Evaluating a digital image may include if the quantization error E quant If the threshold is exceeded, an anomaly is detected, otherwise no anomaly is detected.
[0068] In one example, the quantization error is used as a signal to determine whether a region in a digital image is anomalous or nominal.
[0069] Step 216 may include triggering an action when an anomaly is detected.Step 216 may include triggering an action when the absence of an anomaly is detected.
[0070] The action may be emergency braking of the vehicle or alerting the user to an abnormality. The action may be actively selecting the digital image for transmission to a back-end computer for further processing, or not selecting it for that purpose.
[0071] Further processing may include performing machine learning using the digital images, or testing, verifying, or validating a machine learning system using the digital images.
[0072] The action may be to capture another digital image without forwarding the digital image if an anomaly is detected, or otherwise directly transmit the evaluated digital image to, in particular, a backend.
[0073] The action may be singling out the digital image for evaluation for labeling if an anomaly is detected.
[0074] In this example, step 214 is performed to determine the semantic segmentation and an assessment of the presence or absence of an abnormality in the digital image x.
[0075] Step 214 may be omitted so that the quantization error E quant to assess the presence or absence of anomalies.
[0076] The modified model G represents the anomaly segmentation system.
[0077] One use case for such an assessment is distinguishing outliers from nominal data in semantic segmentation of digital images. This is particularly useful in safety-related applications such as autonomous driving.
[0078] For example, semantic segmentation is provided to detect the presence of objects representing road surfaces, pedestrians, and vehicles. Outliers among them may be unknown objects.
[0079] One use case for such an assessment is the classification of outliers from nominal data in automated optical inspection.
[0080] In one example, the quantized set Or a normalized quantized set Q is provided as a module. This module is inserted into an existing neural network that has been trained for semantic segmentation as the penultimate layer of the neural network.
[0081] Quantized Collection Or the normalized quantized set Q can be added to another layer of the neural network.
[0082] Preferably, add a quantized set Or a vector quantization layer with a normalized quantization set Q.
[0083] The trained vector quantization layer is used to detect anomalies, for example by marking the input data as anomaly if the quantization error of the input data (ie, the vector quantizer determined by the vector quantization layer) is above a given threshold.
[0084] In one example, the only parameter retrained is the quantized set Or the parameters of the normalized quantized set Q. These parameters are usually much fewer than the parameters of the entire neural network. As a result, it becomes possible to incorporate a module for outlier detection into an already trained neural network.
[0085] In one example, the revised model G is a fine-tuning of the last layer, ie, the classifier layer of the revised model G is retrained to further improve the performance.
[0086] The method may include transforming the feature z e From the first scale to the second scale, the feature z in the second scale e Determine the quantization z in the second scale q , quantize z q Downscaling from the second scale to the first scale, and determining the quantization error E in the first scale quant .
[0087] The modified model can be trained in training using the training data.
[0088] Training can include determining a defined quantization set and / or a parameter for normalizing N. Preferably, the training comprises determining the defined quantization set without modifying the first part and / or the second part of the predetermined model and / or normalize N parameters.
[0089] The training data can include semantic segmentation Reference (k) and the digital image xk Yes. (k) For example, is a one hot encoding of label y.
[0090] Providing quantized sets during training include:
[0091] Provide digital image x k and the digital image x k for semantic segmentation Reference (k) , using the modified model G to determine the digital image x k Semantic segmentation
[0092] Depends on the reference y (k) and semantic segmentation The difference between them depends on the loss function to determine the quantization set The quantization q in the loss function includes the cross entropy loss
[0093]
[0094] and quantization error loss
[0095]
[0096] Where sq[·] indicates stopping the gradient operation, and β is a parameter that controls the update rate of the encoding parameters compared to the parameter set update. An example is β=0.25.
[0097] Digital images contain pixel boundaries between classes assigned in semantic segmentation.
[0098] According to one example, the quantized set is a codebook including codebook vectors, i.e., a quantized set The vector in .
[0099] Pixel boundaries between classes are for example pixels with large quantization errors E quant This is due to the absence of nearby codebook vectors.
[0100] This may cause large false positive rates in anomaly predictions. To mitigate this, an additional loss term (boundary loss L boundary ) to train the modified model G, the boundary loss L boundary The quantization error E on the class boundary during training quant The weighting is performed and the allocation of codebook vectors on the class boundaries is encouraged.
[0101] Note that due to the continuous nature of the network output, logit-based uncertainty methods cannot avoid larger uncertainties and, hence, anomaly scores when traversing pixel boundaries between classes.
[0102] Let mi denote a suitably resized binary mask of the class boundaries of reference y, i.e., m = 1 on the class boundary region and zero elsewhere. Boundary loss L boundary Then it is defined by:
[0103] L boundary =m☉E quant
[0104] Where ⊙ denotes the point-wise product. Use, for example, the following loss function to train the modified model G:
[0105] L=aL CE +L quant +λL boundary
[0106] where α, λ, and β are hyperparameters. For example, from E scaled up to the second resolution quant Calculate anomaly score.
[0107] For example, the decision of whether to classify a pixel as an outlier is based on the pixel’s quantization error E quant The threshold value t is, for example, 0≤t≤1.
[0108] According to one example, the modified model G is a neural network including normalized quantization Q. The neural network is trained using a loss function L. A threshold t is selected to determine the sensitivity. During inference, the digital image is passed through the neural network. At the layer representing normalized quantization, the quantization error E is calculated. quant . Corresponding to the quantization error E quant Pixels with >t are marked as abnormal, and the rest are marked as normal. quant can be upscaled to the original image resolution. Similar processing includes quantizing the set Instead of normalizing the neural network of quantized Q.
Claims
1. A computer-implemented method for evaluating a digital image, characterized in that A digital image is provided (202), a first part of a predetermined model is provided (204), wherein the predetermined model is configured to determine a semantic segmentation of the digital image using a second part of the predetermined model, wherein the first part is configured to determine a feature depending on the digital image, wherein the second part is configured to determine the semantic segmentation depending on the feature, wherein the method comprises: determining (208) the feature depending on the digital image using the first part, providing (206) a quantization set for quantizing the feature, determining (210) a quantization of the feature depending on the quantization set and depending on the feature, determining (212) a quantization error depending on the feature and the quantization, and evaluating (216) the digital image depending on the quantization error.
2. The method according to claim 1, characterized in that: A second portion of the model is provided (204), wherein the second portion is configured to determine semantic segmentation of the digital image in dependence on the quantization, and the semantic segmentation of the digital image is determined (214) using the second portion in dependence on the quantization.
3. The method according to claim 2, characterized in that Providing (206) a quantization set includes providing a reference for semantic segmentation of the digital image, determining the semantic segmentation of the digital image, and determining a quantization in the quantization set depending on a difference between the reference and the semantic segmentation and depending on a quantization error.
4. The method according to any one of the preceding claims, wherein: Providing (204) a pre-trained model includes training a first part to determine features, and training a second part to determine semantic segmentation depending on the features.
5. The method according to any one of the preceding claims, characterized in that The method includes determining (208) a feature using a predetermined normalization, and determining a quantization of the feature using the predetermined normalization.
6. The method according to any one of the preceding claims, characterized in that The method includes upscaling features from a first scale to a second scale, determining quantization in the second scale based on the features in the second scale, downscaling the quantization from the second scale to the first scale, and determining a quantization error in the first scale.
7. The method according to any one of the preceding claims, characterized in that The feature is a vector and the quantization of the feature is a vector, wherein determining (212) the quantization error comprises determining a cosine distance between the feature and the quantization of the feature.
8. The method according to any one of the preceding claims, characterized in that Providing (202) the digital image includes capturing the digital image using a camera, in particular, of an at least partially autonomous vehicle or automated optical inspection equipment.
9. The method according to any one of the preceding claims, characterized in that Evaluating (216) the digital image includes detecting an anomaly if the quantization error exceeds a threshold, or otherwise not detecting an anomaly.
10. A device (100) for evaluating a digital image, characterized in that The device (100) comprises at least one processor (102) and at least one storage device (104) for storing digital images and instructions, which, when executed by the at least one processor (102), cause the at least one processor (102) to perform a method according to one of claims 1 to 9, wherein the at least one processor (102) is configured to execute the instructions.
11. A program, characterized in that The program comprises instructions which, when executed by at least one processor, cause the at least one processor to perform the method according to one of claims 1 to 9.
