Debias network and system and vehicle including same
By introducing a debiased network into the feature extractor, the deviation problem caused by data differences in traditional feature extractors is solved in training and practical applications, and the refined feature extraction on low-processing equipment is realized, reducing the retraining cost And provides flexibility and real-time operation capabilities.
Patent Information
- Application Number
- CN202410729792.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-06-06
- Filing Date
- 2024-06-06
- Publication Date
- 2024-12-06
AI Technical Summary
Traditional feature extractors have increased classification error rates due to data differences in training and practical applications, and low processing capability devices cannot be retrained to overcome the bias problem.
The de-biased network is used to receive basic features, remove biased features through training, and work with pre-trained feature extractors, reducing modifications to feature extractors, saving processing power and avoiding privacy issues.
It realizes refine feature extraction on low-processing equipment, reduces retraining costs, avoids processing and storage of private information, and provides flexibility and real-time operation capabilities.
Smart Images

Figure CN119091478A_ABST
Abstract
Description
Technical Field
[0001] The present invention generally relates to a system for refining feature extraction and a vehicle including the same. In particular, the present invention relates to a debiasing network that operates on the output of a pre-trained feature extractor to remove bias of the extracted features. Background Art
[0002] Traditional feature extractors (such as any neural network) are trained on large datasets. Training is usually done offline, i.e. on a dedicated computing system, and using a dataset prepared specifically for the training phase of the network. As a pure example, a feature extractor can be trained on RGB images that, for example, show faces to train a feature extractor for a facial identification or recognition task.
[0003] However, the actual real-world data fed to the feature extractor in the online phase, i.e. the actual discrimination or recognition task, may be different from the prepared dataset. As a pure example, the prepared dataset used for training is unlikely to include facial images of people to be discriminated or recognized by the trained system. Furthermore, different image sensors may be employed, such as a training dataset that includes only RGB images, while the system employing the trained feature extractor has a near infrared (NIR) image sensor.
[0004] It is therefore an object of the present invention to provide an improved feature extraction system that overcomes the above-mentioned problems. Summary of the invention
[0005] The above objects are solved by the invention as defined in the independent claim. Preferred embodiments are defined by the dependent claims.
[0006] According to a first aspect, in order to better understand the present invention, a system for refining feature extraction includes: a pre-trained feature extractor configured to receive input data and extract basic features from the received data, wherein the extracted basic features allow the received input data to be classified. The pre-trained feature extractor can be a traditional feature extractor, such as a neural network. Pre-training can be performed on training data (dataset) consisting of data pairs and associated class labels. Therefore, the pre-trained feature extractor can be any conventional feature extractor trained in any conventional manner.
[0007] The extracted features may have or include bias due to differences between the training data and the real-world data on which the system operates. Bias is a deviation or error in a result, especially in the result of a subsequent classification or regression of the input data. In other words, the probability of misclassifying the data (i.e., an erroneous result of a subsequent conventional classification) may increase, especially the greater the probability that the training data deviates from the real-world data, the more likely the above probability is to increase.
[0008] To avoid or at least reduce such bias, the system further comprises: a debiasing network configured to receive the base features as input and extract debiasing features therefrom. Further, the debiasing network has an input dimension corresponding to the base features and an output dimension equal to the input dimension. Thus, the output of the debiasing network does not require any changes to subsequent components (such as those developed and implemented for the pre-trained feature extractor).
[0009] The system further includes a classifier configured to classify the deviation features and output a classification result. The classifier may be a traditional classifier or a classifier configured to work with the pre-trained feature extractor or a classifier trained with the pre-trained feature extractor.
[0010] In other words, the debiasing network is a "top setting" for the traditional feature extractor and allows to adjust the final classification or regression of the input data. The debiasing network can be thought of as "prescription glasses" for the feature extractor suitable for real-world data.
[0011] Furthermore, conventionally, in order to remove bias from the results of a feature extractor, the feature extractor must be retrained, for example, on an additional data set (such as real-world data). Such retraining or interval training of the feature extractor requires a large amount of processing power. However, such processing power is not available in many devices that implement or employ feature extractors and classifiers. Therefore, such low-processing power devices are generally unable to overcome the bias problem.
[0012] Employing the debiasing network of the present invention allows for improved and refined feature extraction with less processing power. Since only the layers of the debiasing network need to be trained, the processing power of such a device is typically sufficient to perform such a task.
[0013] As a pure example, pre-trained feature extractors and classifiers can be used for object detection, facial identification, handwriting identification, speech recognition, etc. The detection can be based on sensor data, such as image data from an image sensor, touch input data from a touch input device, audio data from a microphone, etc. The possibility of different sensors being used for the data set on which the feature extractor is pre-trained and the data set is used by the actual system is high. Therefore, there may already be deviations due to different sensors (e.g., RGB image data used for training and NIR sensors using pre-trained feature extractors used in the actual system). Other factors may cause deviations between the training data and the actual real-world data of the current system, such as different groups of facial recognition used for the training data and the actual real-world data. For example, in facial recognition, it is likely that some individuals are not part of the offline training data set, but these individuals need to be identified or identified based on the pre-trained feature extractors.
[0014] By employing a debiasing network, the pre-trained feature extractor does not need to be modified, thereby saving the cost of updating and / or retraining. In addition, privacy issues can be avoided because the actual system running the pre-trained feature extractor does not need to receive training data that includes private information, such as biometric data. The debiasing network can operate on real-world data (i.e., the data of the individuals using the system), thereby being able to confirm the processing of their personal data. In addition, such personal data does not have to be stored for a period of time longer than the usual buffering time (during classification and training the system) or transmitted in a communication network because the debiasing network can operate online (i.e., in real time).
[0015] It should be noted that the present disclosure is independent of the methods, techniques and / or models that underlie the feature extractor. Thus, the pre-trained feature extractor can be treated as a "black box" that remains unchanged in the system, allowing for great flexibility.
[0016] Furthermore, it should be understood that the present disclosure refers to "basic features", but this should not limit the present invention to a single basic feature. Instead, the feature extractor may also extract multiple basic features.
[0017] In a variant of an implementation, the extracted base features can be embedding elements from any type of feature extractor. In addition, the embedding can also be multiple features. An embedding in the context of a feature extractor (such as a neural network) is a mapping of a discrete variable (received data or input data) to a vector of continuous numbers. In other words, an embedding is a low-dimensional learned continuous vector representation of a discrete variable. Neural network embeddings allow reducing the dimensionality of categorical variables and meaningfully representing categories in the transformed space.
[0018] In another implementation variant, the system may further include an updating unit configured to train the debiasing network based on the base features of the pre-trained feature extractor, the debiasing features of the debiasing network, and the expected results of the classifier. That is, the training process is performed by the updating unit equivalent to the training of the feature extractor.
[0019] However, the debiasing network has a lower dimensionality than a pre-trained feature extractor, which simplifies the training of the debiasing network in terms of processing power and processing cost. Therefore, even systems with lower processing power (such as embedded systems) can train / update the debiasing network, thereby achieving an accurate system for extracting features.
[0020] The update unit can be activated during the startup phase of the system (e.g. when the system is used for the first time). Alternatively or additionally, the update unit can be activated at specific intervals (time intervals and / or task-based intervals). This allows improving the system by optimizing the debiasing network and further refining the feature extraction.
[0021] In another implementation variant, the debiasing network may be trained to remove the bias between the base features and the debiased features by subtraction. In other words, the debiasing network learns the bias in such a way that it can be removed by a final subtraction, i.e., the bias is an additive bias. It should be understood that this is only one possibility to implement the update / training process, and that it is equally possible to train the debiasing network by addition.
[0022] In yet another implementation variant, the system may further include a weighting unit configured to weight the output of the debiasing network. The weighting unit may be used to prevent extreme deviations from the underlying features. The operation of the weighting unit may be implemented by a selectable factor or a trainable factor, the latter of which may be trained / updated by an update unit.
[0023] In a specific implementation variant, the weighting unit may be configured to weight intermediate outputs of the debiasing network, such as the results of a specific layer of the debiasing network. For example, the weighting may be applied to most layers of the debiasing network. As a pure example, the outputs of the trainable part of the debiasing network are weighted, while the final fixed layer is applied to the weighted result.
[0024] In another implementation variant, the debiasing network may include one or more batch normalization layers, and / or one or more rectified linear unit function layers, and / or one or more fully connected layers. In other words, the debiasing network may consist of one layer, two layers or more layers, which may include batch normalization (BN) layers, rectified linear unit (Relu) function layers and / or fully connected (FC) layers or any combination thereof. As a pure example, the BN layer of at least one block, and the Relu function layer and the FC layer form (at least part of) the debiasing network.
[0025] Alternatively, the debiasing network or part thereof may comprise an autoencoder. An autoencoder is a special neural network architecture that can be divided into two parts: an encoder and a decoder. The encoder converts the input data (e.g., the embedding of a feature extractor) into a low-dimensional embedding. The decoder converts the low-dimensional embedding back into the encoded representation. In other words, the dimension of the embedding is temporarily reduced in a variant of this implementation of the debiasing network. As a pure example, at least two blocks of BN layers, Relu function layers and FC layers form (at least part of) a debiasing network, wherein the first block reduces the dimension of the embedding and the second block recreates the input dimension.
[0026] According to a second aspect, for a better understanding of the present invention, a vehicle includes: at least one sensor configured to output sensor data, and a system for refining feature extraction according to the first aspect or one of its variants. Specifically, a pre-trained feature extractor of the system receives the output sensor data and extracts basic features therefrom.
[0027] Therefore, since the computing units in the vehicle are usually energy-saving devices or small devices with less processing power, the vehicle can still be equipped with a feature extractor to perform specific tasks in the vehicle, and this provides robust results. In addition, the same system can be implemented in different types of vehicles, especially vehicles with different types of sensors (such as cameras, microphones, etc.) for sensing similar items. Similarly, the same system can be implemented into vehicles sold to different regions of the world, where specific languages are spoken (related to voice recognition), or users belong to specific ethnic groups (related to facial recognition and user authentication).
[0028] In a variant of an implementation, the system is implemented on an engine control unit or an infotainment system of the vehicle. Thus, even a low-power processing unit already installed on the vehicle can be equipped to perform high-reliability feature extraction.
[0029] The present invention is not limited to the aspects and variants in the described form and order. Specifically, the description of each aspect and variant should not be interpreted as a specific restricted grouping of features. It should be understood that the present invention also encompasses the combination of each aspect and variant. Therefore, each variant or optional feature can be combined with any other aspect, variant, optional feature or even combination thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] The present disclosure will be further described below with reference to exemplary embodiments shown in the accompanying drawings, in which:
[0031] Figure 1 A system for refining feature extraction is schematically shown;
[0032] Figure 2 The debiasing network is schematically shown in more detail;
[0033] Figure 3 An exemplary update to a debiasing network is schematically shown;
[0034] Figure 4 schematically illustrates an exemplary feature extraction; and
[0035] Figure 5 A vehicle including a system for refined feature extraction is schematically illustrated. DETAILED DESCRIPTION
[0036] In the following description, for the purpose of explanation rather than limitation, specific details are given in order to provide a thorough understanding of the present invention. It will be apparent to those skilled in the art that the present invention can be implemented in other implementations that depart from these specific details.
[0037] Figure 1 A system 100 for refining feature extraction is schematically illustrated. The schematic diagram of the system 100 is broken down into components used during development of the system 100 and components used during deployment of the system 100.
[0038] For example, a pre-trained feature extractor 130, such as a neural network (e.g., a convolutional neural network CNN, including VGG 16 / 19, InceptionV3, ResNet34 / 50, SENet, MobileNet (V2), EfficientNetB0 / 1 / … / 7, ConvNeXt, etc.) can form the core of the system 100. The feature extractor 130 is pre-trained, that is, it has been trained to perform specific feature extraction, such as classification or regression of images, audio input, handwriting, etc. The pre-training is performed based on an external dataset 120. The dataset can be a large publicly available dataset, such as a collection of images, speech, sound, handwriting samples, etc. Examples of these datasets are ImageNet, VGGFace2, MS-Celeb-1M, CASIA-WebFace, IJB-B, TIMIT, IAM, etc. As a pure example, the external dataset 120 can include multiple facial images and corresponding classifications / labels, such as female / male, wearing glasses, having a beard, hair color, biometric information / category, etc. The feature extractor 130 is trained on this data to perform correct classification / regression for any input image.
[0039] Since such pre-training of the feature extractor 130 is computationally intensive and requires large processing power, it is pre-performed by a correspondingly powerful computing system. Figure 1 The portion of system 100 indicated as "developed" in reflects the establishment of standard / conventional feature extraction.
[0040] The pre-trained model (feature extractor) 130 may then be implemented (downloaded or otherwise stored) in the system 100 for deployment. At run time ("deployment"), real-world data is provided to the pre-trained feature extractor 130. For example, the system 100 may include a sensor 110 (illustratively shown as a camera) that generates input data for the system 100. The feature extractor 130 extracts basic features from the input data. The basic features may refer to any number of features or embeddings (depending on the type and dimensionality of the pre-trained feature extractor 130).
[0041] The present disclosure contemplates that the output of the feature extractor 130 includes errors or deviations. For example, the camera 110 (or other sensor) may be a different camera 110 or a different type / kind of camera 110 than the camera used to generate the dataset 120. Likewise, the types of data (images, sounds, voices, handwriting, etc.) in the dataset 120 may not cover the real-world data sensed and provided by the sensor 110. For example, although the external dataset 120 may include a large number of images showing faces, none of these images may actually show the face of a person sitting in front of the camera 110 to be recognized or authenticated. Therefore, the feature extractor 130 may produce errors in its results, and it is not possible to retrain the feature extractor 130.
[0042] therefore, Figure 1 The output of the feature extractor 130 is identified as including a bias, for example, as a biased embedding 140. The system 100 also includes a debiasing network 150 configured to receive the base features of the feature extractor 130 and extract debiased features therefrom. The debiasing network 150 has as input dimensions corresponding to the base features 140 extracted by the feature extractor 130. In addition, the output dimension of the debiasing network 150 is equal to the input dimension.
[0043] This allows the deviation features to be further processed 170 in the same manner as the base features 140. As a pure example, the further processing 170 may be configured to classify or label the base features 140, but in the system 100, the further processing 170 operates on / utilizes the debiased features from the debiasing network 150.
[0044] Figure 2 The debiasing network 150 is schematically shown in more detail. As a pure example, Figure 2 A debiasing network 150 is shown consisting of six layers. It should be understood that the debiasing network 150 may include any number of layers necessary to debias the base features 140. Furthermore, as another pure example, the debiasing network 150 may be composed of blocks or groups of layers having similar functions. Figure 2 Two blocks of three layers each are shown exemplarily, wherein the first layer is a batch normalization layer (BN), the second layer is a rectified linear unit function layer (Relu), and the third layer is a fully connected layer (FC). It should be understood that these layers are only mentioned exemplarily, and one or more layers may be omitted, repeated, or provided in any combination.
[0045] The debiasing network 150 is trained in such a way that the bias is considered as an additive bias and the debiasing network 150 removes the bias by subtracting ("-"). It should be understood that this is only one possibility for training the debiasing network 150 and removing the bias.
[0046] In order to avoid extreme deviations from the base features 140 (biased embedding), a weighting unit 156 may be provided, which is configured to weight the output of the debiasing network 150. For example, the weighting unit 156 may affect the subtraction, for example by multiplying the subtraction by a factor "α".
[0047] The system 100 may include an updating unit 160 configured to train the debiasing network 150 based on the base features 140 of the feature extractor 130, the debiasing features 154 of the debiasing network 150, and the expected results of the classifier 170. The updating unit 160 may be operated at an initial stage of system deployment and / or at specific intervals during operation of the system 100.
[0048] Figure 3 An exemplary update of the debiasing network 150 is schematically shown. Since the pre-trained feature extractor 130 is not changed during the training phase of the system 100, the update unit 160 only modifies the debiasing network 150. Figure 3 , as shown by the arrow looping back to the biased embedding 140 , since this is the unchanged starting point for the rerun of the debiasing network 150 .
[0049] Figure 4 An exemplary feature extraction during "deployment" is schematically shown. In other words, Figure 4 Components of the system 100 implemented for runtime are shown. Specifically, the system 100 includes a pre-trained feature extractor 130, a debiasing network 150, and a classifier 170. The classifier 170 is configured to classify the deviation features (output of the deviation network 150) and output the classification results or regression results of the deviation features.
[0050] The system 100 may be coupled with a sensor 110 that generates data that is fed to a pre-trained feature extractor 130. The sensor data is the data to be classified or labeled by the classifier 170.
[0051] Figure 5 A vehicle 1 is schematically shown, comprising a system 100 for refining feature extraction. The vehicle 1 may be a car operated by a driver 5. As a pure example, the system 100 may be used to identify the driver 5, to identify the posture of the driver 5, to identify the voice input of the driver 5, etc. To this end, the vehicle comprises a sensor 110, which may be a camera, a microphone, a touch screen or a touch panel, etc.
[0052] System 100 (see example Figure 1 ) can be implemented on the engine control unit 105 of the vehicle 1. Alternatively or additionally, the system 100 can be implemented in the infotainment system 107 of the vehicle 1. The engine control unit 105 as well as the infotainment system 107 are typically low-processing-power devices, since the intended tasks (engine control, media playback, navigation, etc.) are not processing-intensive. Since the debiasing network 150 operates on a lower dimension than the initial dimension of the feature extractor 130, the debiasing network 150 can be run on such low-processing-power processing devices (105, 107). Furthermore, even the update unit 160 can be run on such low-processing-power processing devices 105, 107.
[0053] It should be understood that other processing devices with low processing capabilities may be used to deploy the system 100. For example, mobile phones, tablet computers, laptop computers, home appliances, etc. may be processing devices with low processing capabilities that may be enhanced by installing the system 100.
[0054] It is believed that the advantages of the technology described herein will be fully understood from the foregoing description, and it is apparent that the form, structure and arrangement of the exemplary aspects thereof may be variously changed without departing from the scope of the invention or sacrificing all of its advantageous effects. As the technology presented herein may vary in many aspects, it will be recognized that the disclosure should be limited only by the scope of the following claims.
Claims
1. A system (100) for refining feature extraction, the system comprising: a pre-trained feature extractor (130) configured to receive input data and extract basic features (140) from the received input data, wherein the extracted basic features (140) allow classification of the received input data; a debiasing network (150) configured to receive the base features (140) as input and extract debiasing features (154) therefrom; and A classifier (170), configured to classify the deviation feature and output a classification result; The debiasing network (150) has an input dimension corresponding to the basic features (140) and an output dimension equal to the input dimension.
2. The system according to claim 1, further comprising: An updating unit (160) is configured to train the debiasing network (150) based on the base features (140) of the pre-trained feature extractor, the debiasing features (154) of the debiasing network, and the expected results of the classifier.
3. The system according to claim 1 or 2, wherein: The debiasing network is trained to remove the bias between the base features and the debiased features by subtraction.
4. The system according to any one of claims 1 to 3, further comprising: A weighting unit (156) configured to weight the output of the debiasing network (150).
5. The system according to any one of claims 1 to 4, wherein: The debiasing network (150) includes one or more batch normalization layers, and / or one or more rectified linear unit function layers, and / or one or more fully connected layers, or Wherein, the debiasing network includes an autoencoder.
6. A vehicle (1), comprising: at least one sensor (110) configured to output sensor data; and The system (100) for refined feature extraction according to any one of claims 1 to 5, wherein the pre-trained feature extractor (130) of the system (100) receives the output sensor data and extracts the basic features therefrom.
7. The vehicle according to claim 6, wherein: The system (100) is implemented on an engine control unit (105) or an infotainment system (107) of the vehicle.
