Apparatus and method to improve zero sample classification

Abstract

A computer-implemented method of classifying a sensor signal (30) by a sensor signal encoder and a text encoder, where the encoder is configured to encode its input into a potential representation, comprising the step of encoding, by the sensor signal encoder, the sensor signal (30) as a first potential representation (e (x)). A plurality of textual cues (tij) are generated, wherein several textual cues (tii <-cj) characterizing a corresponding class are instantiated for each class (cj). Encoding, by the text encoder, the generated text cues (tij) as a second potential representation # imgabs0 #, determines a class query (qj) for each class by taking a weighted average on the second representation # imgabs1 # corresponding to the same class (ci). A similarity between the first potential representation (e (x)) and each class query (qj) is calculated. The sensor signals (30) are assigned to the class with the highest similarity.

Description

Technical Field

[0001] The present invention relates to a method for improving zero-shot classification by a combination of image and text encoders, wherein the weights of the cues of the text encoder are optimized, as well as a method for operating an actuator, a computer program and a machine-readable storage medium and a system. Background Art

[0002] Zero-shot classifiers built on models like CLIP (https: / / arxiv.org / abs / 2103.00020) or CoCa (https: / / arxiv.org / abs / 2205.01917) have shown strong domain generalization due to the different training data distributions of CLIP and CoCa. They are constructed based on a set of cue templates (parameterized by class names) that cover possible variations of the domain.

[0003] These hint templates can be hand-crafted (https: / / arxiv.org / abs / 2103.00020), generated by large language models (https: / / arxiv.org / abs / 2210.07183), or randomly generated (https: / / arxiv.org / abs / 2306.07282). By instantiating the templates by inserting the class name, a set of text queries for each class is obtained. These are encoded by the CLIP text encoder, and the resulting encoded queries (per class) are simply averaged. For the data to be classified (usually images, but also other modalities as in methods such as ImageBind https: / / arxiv.org / abs / 2305.05665), the data is first encoded by the encoder of the corresponding modality (e.g., an image encoder), and then the cosine similarity of the encoded data to each (average) class query is calculated. The data is then assigned to the class with the largest similarity.

[0004] Advantages of the present invention

[0005] The improvement over the prior art is given at least by the following: the inventors propose not to simply average the encoded textual prompts (of each class), but to take a weighted average, where a weight is automatically determined for each datum. The weights are determined so that prompt templates whose (cross-class) embedding is closer to the embedding of the corresponding datum receive a higher weight than those prompt templates that are less similar. This can be intuitively motivated by a closer textual encoding corresponding to a textual prompt that better describes the datum than a textual prompt with a lower similarity.

[0006] Therefore, the present invention improves the performance of zero-shot classifiers on many types of data with very little inference time overhead, and is still completely zero-shot because it works without requiring any labels or other data points from the target domain. It makes no assumptions about the underlying multi-model model and is therefore widely applicable. Summary of the invention

[0007] In a first aspect, a method for classifying a sensor signal by means of a sensor signal encoder and a text encoder according to claim 1 is proposed.

[0008] The encoder is configured to encode its input into a latent representation. A latent representation can be understood as a point in a latent space. A latent space - also called a latent feature space or an embedding space - is an embedding within a manifold where items that are similar to each other are located closer to each other in the latent space. Both the sensor signal encoder and the text encoder can share the same latent space.

[0009] The classification method starts with generating a plurality of textual prompts, wherein for each class, several textual prompts characterizing the corresponding class are instantiated. This step can be implemented by manually writing prompts and / or by automatically generating them, for example, as described above in the background technology section. The textual prompts can be text for processing by the base model.

[0010] This is followed by the text encoder encoding the generated text prompt into a second latent representation.

[0011] This is followed by determining the class query for each class by averaging over the second representations corresponding to the same class. Compared to the prior art, this step differs in that the averaging is performed as a weighted average, where each second representation corresponding to the same class is weighted by a weight.

[0012] This is followed by computing the (cosine) similarity between the first latent representation and each class query, and assigning the sensor signal to the class with the highest similarity.

[0013] Preferably, the first potential representation (e (x) ) and the second potential representation is normalized.

[0014] Preferably, the instantiation of the prompt is generated by inserting the class name into a predefined template of the text prompt. In addition, class attributes can be inserted into the template. Note that other known methods from the literature for filling in the text prompt template can be utilized.

[0015] The sensor signal may be an image. The term image basically includes any information distribution arranged in a two-dimensional or multi-dimensional grid. The information may be, for example, the intensity values ​​of image pixels captured with any imaging modality such as an optical camera, a thermal imaging camera or ultrasound. However, any other data such as audio data, radar data or LIDAR data may also be converted into an image and then classified in the same way, or may be provided as an additional input to the encoder.

[0016] Note that classification can be performed based on low-level features (e.g., edges or pixel attributes of an image), in particular the determination of a latent representation.

[0017] Preferably, the encoder is an encoder of a base model, in particular CLIP or CoCa.

[0018] The assigned class (also referred to as classification) may then be used to provide an actuator control signal for controlling the actuator.

[0019] In a further aspect of the present invention, a control system / method for operating an actuator is provided. The control system / method is configured to operate the actuator according to an actuator control signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Embodiments of the present invention will be discussed in more detail with reference to the following figures. The figures show:

[0021] Figure 1 is a schematic flow chart of an embodiment of the present invention;

[0022] Figure 2 is a control system for controlling a robot that is at least partially autonomous;

[0023] Figure 3 It is the control system that controls the manufacturing machines;

[0024] Figure 4 It is the control system that controls the access control system;

[0025] Figure 5 It is the control system that controls the supervisory system;

[0026] Figure 6 It is the control system that controls the automated personal assistant;

[0027] Figure 7 It is a control system that controls the imaging system;

[0028] Figure 8 It is the training system used to control the classifier. DETAILED DESCRIPTION

[0029] Figure 1In a first embodiment, a known method 20 for zero-shot classification is shown.

[0030] Let us consider the classification task X→(c1, .., c C ), where X corresponds to the input domain, such as an image or a radar point cloud of a certain resolution, and there are C classes. We assume that there exists a multimodal joint embedding space E and a corresponding embedding function E X : X→ε, which maps the input data x∈X to the embedding space E, and maps the text prompt to the embedding space ε of E T :T→ε. Let there be K text prompt templates (e.g., manually designed, generated by a large language model, or randomly enhanced), all parameterized by the class name c. The zero-shot classifier is then obtained via the following steps:

[0031] S21: Encode the data x of interest into

[0032] S22: Generate K·C text prompts in Refers to the class name c by inserting j To instantiate the template

[0033] S23: Encode all text prompts

[0034] S24: Average over all text-encoded templates corresponding to a class to obtain the class query

[0035] S25: Calculate the cosine similarity q between x and all class queries j :s j =e (x) ·q j .

[0036] S26: Assign x to class c using the following formula j :

[0037] exist Figure 1 In the second embodiment, an improvement in zero-shot classification is shown. For the improvement, step S24 replaces the unweighted average of the text encoding Using weighted average Among them, preferably This can be done by setting w = softmax(ρ) and preferably to achieve.

[0038] In general, ρ can be initialized with zero, which corresponds to the unweighted mean of the text encoding. Then, the gradient is calculated where s = q × e x , And the operator × refers to the (batched) inner product.

[0039] For a learning rate α, one can run R steps (usually α=0.8 and R=1):

[0040] Therefore, logsumexp(s) is maximized after R steps of gradient ascent with respect to ρ. The motivation for maximizing logsumexp(s) is that this corresponds to assigning higher weights to prompt templates with encodings that are more similar to the encoded data, since similarity in the embedding space corresponds to textual prompts that better describe the data.

[0041] This modification of the known zero-shot classifier can be called "auto-tuning" because its weights w are automatically adapted for each input.

[0042] In a preferred embodiment of an "auto-tuning" classifier, We can use the inner product e (xt) =e (x) ×e (t) The closed form is:

[0043]

[0044] where δ ij is the Kronecker delta function, where δ ii =1 and for i≠j, δ ij =0.

[0045] The automatic tuning averaging by inserting text encoding is now Figure 1 In the second embodiment, step S25 comprises determining the weight w as described above. i , and then by Determine the weighted average, where w i =softmax(ρ).

[0046] In the details of step S25, at the beginning, the auto-tuning takes the average setting ρ=0, and then runs the loop for R steps with the following sub-steps to determine the weight w i :

[0047] a. Set w_i = softmax(ρ);

[0048] b. Take the weighted average of all text encoding templates corresponding to a class to obtain the class query

[0049] c. Calculate the cosine similarity q between x and all class queriesj :s j =e (x) ·q j ;

[0050] d. Calculate the gradient

[0051] e. Update ρ using gradient ascent and learning rate:

[0052] In a preferred embodiment, a heuristic for automatically selecting the learning rate can be used. Note that in the zero-shot setting, by definition, there is no labeled data for which free parameters (like the learning rate) can be tuned. Therefore, the free parameters are preferably chosen globally in a dataset-independent manner. However, a global choice of the learning rate α can be problematic because the gradient The scale of depends on the dataset, and the learning rate will have to be adapted accordingly. Therefore, it is suggested to use a different parameterization, where the free parameters are easier to interpret and α is a derived quantity. Specifically, one can control the entropy of the query weights w The standard, untuned uniform weights have maximum entropy log2K and one can set the target entropy to β*log2K, where the entropy reduction factor β is a new free parameter. The preferred value is β=0.85. Then one can use a method for α∈[0,10 10 ]Find entropy(w)-β*log2K=0 (where The preferred root finding method is the bisection method (https: / / en.wikipedia.org / wiki / Bisection_method). The root α is the determined learning rate.

[0053] Figure 2 1 shows an embodiment of an actuator with a control system 40. The actuator and its environment will be collectively referred to as the actuator system. At preferably uniformly spaced distances, the sensor 30 senses the condition of the actuator system. The sensor 30 may include several sensors. Preferably, the sensor 30 is an optical sensor that captures an image of the environment. The output signal S of the sensor 30 (or, in the case where the sensor 30 includes multiple sensors, the output signal S of each sensor) encoding the sensed condition is transmitted to the control system 40.

[0054] Thus, the control system 40 receives a stream of sensor signals S. It then calculates a series of actuator control commands A depending on the stream of sensor signals S, and then transmits the series of actuator control commands A to the actuator unit 10, which converts the control commands A into mechanical movements or changes in physical quantities. For example, the actuator unit 10 can convert the control commands A into electrical, hydraulic, pneumatic, thermal, magnetic and / or mechanical movements or changes. Specific but non-limiting examples include electric motors, electroactive polymers, hydraulic cylinders, piezoelectric actuators, pneumatic actuators, servo mechanisms, solenoids, stepper motors, etc.

[0055] The control system 40 receives a stream of sensor signals S of the sensor 30 in an optional receiving unit 50. The receiving unit 50 transforms the sensor signal S into an input signal x. Alternatively, without the receiving unit 50, each sensor signal S can be taken directly as an input signal x. The input signal x can be given, for example, as an extract from the sensor signal S. Alternatively, the sensor signal S can be processed to generate the input signal x. The input signal x comprises image data corresponding to an image recorded by the sensor 30. In other words, the input signal x is provided as a function of the sensor signal S.

[0056] The input signal x is then passed to a zero-shot classification method, such as Figure 1 A second exemplary embodiment of a classifier, referred to hereinafter as classifier 60 , may be given, for example, by an artificial neural network.

[0057] The classifier 60 is parameterized by the parameter φ stored in and provided by the parameter storage device St1.

[0058] The classifier 60 determines the output signal y from the input signal x. The output signal y comprises information assigning one or more labels to the input signal x. The output signal y is transmitted to an optional conversion unit 80, which converts the output signal y into a control command A. The actuator control command A is then transmitted to the actuator unit 10 for controlling the actuator unit 10 accordingly. Alternatively, the output signal y can be taken directly as the control command A.

[0059] The actuator unit 10 receives the actuator control command A, is controlled accordingly and performs an action corresponding to the actuator control command A. The actuator unit 10 may include control logic that transforms the actuator control command A into another control command that is then used to control the actuator 10 .

[0060] In further embodiments, the control system 40 may include the sensor 30. In still further embodiments, the control system 40 may include the actuator 10, alternatively or additionally.

[0061] In one embodiment, the classifier 60 may be designed to identify lanes on the road ahead, for example by classifying the road surface and markings on said road, and identifying lanes as patches of the road surface between said markings. Based on the output of the navigation system, a suitable lane may then be selected for following the selected path, and depending on the current lane and said target lane, it may then be decided whether the vehicle 60 is to switch lanes or stay in said current lane. The control command A may then be calculated by, for example, retrieving a predefined motion pattern from a database corresponding to said identified motions.

[0062] Similarly, when a road sign or a traffic light is identified, depending on the identification type of the road sign or the identification state of the traffic light, the corresponding constraints on the possible movement modes of the vehicle 60 can then be retrieved from, for example, a database, the future path of the vehicle 60 commensurate with the constraints can be calculated, and the actuator control command A can be calculated to steer the vehicle, such as to execute the trajectory.

[0063] Similarly, when pedestrians and / or vehicles are identified, the projected future behavior of the pedestrians and / or vehicles can be estimated, and based on the estimated future behavior, a trajectory can then be selected to, for example, avoid a collision with the pedestrians and / or vehicles, and the actuator control command A can be calculated to steer the vehicle to, for example, execute the trajectory.

[0064] Furthermore, the control system 40 may include a processor 45 (or multiple processors) and at least one machine-readable storage medium 46 having stored thereon instructions that, if executed, cause the control system 40 to perform a method according to an aspect of the present invention.

[0065] exist Figure 2 In a preferred embodiment of the present invention, the control system 40 is used to control an actuator, which is an at least partially autonomous robot, such as an at least partially autonomous vehicle 100.

[0066] The sensors 30 may include one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more LiDAR sensors and / or one or more position sensors (such as, for example, GPS). Some or all of these sensors are preferably, but not necessarily, integrated in the vehicle 100.

[0067] Alternatively or additionally, the sensor 30 may include an information system for determining the state of the actuator system. One example of such an information system is a weather information system that determines the current or future state of the weather in the environment 20.

[0068] For example, using the input signal x, the classifier 60 can, for example, detect an object in the vicinity of the at least partially autonomous robot. The output signal y may include information characterizing where the object is located in the vicinity of the at least partially autonomous robot. A control command A may then be determined based on this information, for example to avoid a collision with the detected object.

[0069] The actuator unit 10, which is preferably integrated in the vehicle 100, may be given by the brakes, propulsion system, engine, transmission or steering of the vehicle 100. An actuator control command A may be determined such that the actuator (or actuators) unit 10 is controlled such that the vehicle 100 avoids a collision with said detected object. The detected objects may also be classified according to what the classifier 60 considers them to be most likely to be, for example pedestrians or trees, and the actuator control command A may be determined depending on the classification.

[0070] In a further embodiment, the at least partially autonomous robot may be provided by another mobile robot (not shown), which may move, for example, by flying, swimming, diving or walking. The mobile robot may in particular be an at least partially autonomous lawnmower, or an at least partially autonomous cleaning robot. In all the above embodiments, an actuator command control A may be determined such that a propulsion unit and / or a steering and / or a braking of the mobile robot is controlled such that the mobile robot may avoid a collision with the identified object.

[0071] In a further embodiment, the at least partially autonomous robot may be provided by a gardening robot (not shown) which uses a sensor 30, preferably an optical sensor, to determine the state of plants in the environment 20. The actuator unit 10 may be a nozzle for spraying chemicals. Depending on the identified type of plant and / or the identified state, the actuator control command A may be determined so that the actuator unit 10 sprays the plant with an appropriate amount of an appropriate chemical.

[0072] In a further embodiment, the at least partially autonomous robot may be provided by a household appliance (not shown), such as, for example, a washing machine, a stove, an oven, a microwave or a dishwasher. The sensor 30, such as an optical sensor, may detect the state of an object to be subjected to a treatment by the household appliance. For example, in the case where the household appliance is a washing machine, the sensor 30 may detect the state of the laundry inside the washing machine. The actuator control signal A may then be determined depending on the detected laundry material.

[0073] Figure 3 , an embodiment is shown in which a control system 40 is used to control a manufacturing machine 11 of a manufacturing system 200, such as a solder mounter, a punch cutter, a cutting machine or a gun drill, for example, as part of a production line.

[0074] The sensor 30 may be given by an optical sensor which captures, for example, properties of the manufactured product 12. The classifier 60 may determine from these captured properties the state of the manufactured product 12. The actuator unit 10 controlling the manufacturing machine 11 may then be controlled depending on the determined state of the manufactured product 12 for subsequent manufacturing steps of the manufactured product 12. Alternatively, it is conceivable that, depending on the determined state of the manufactured product 12, the actuator unit 10 is controlled during the manufacture of a subsequent manufactured product 12.

[0075] Figure 4 An embodiment is shown in which the control system controls an access control system 300. The access control system may be designed to physically control access. For example, it may include a door 401. The sensor 30 is configured to detect scenes relevant to the decision whether access is granted or not. For example, it may be an optical sensor for providing image or video data for detecting a face. The classifier 60 may be configured to interpret the image or video data, for example by matching the identity with known persons stored in a database, thereby determining the identity of the person. Then, depending on the interpretation of the classifier 60, for example, depending on the determined identity, an actuator control signal A may be determined. The actuator unit 10 may be a lock that grants access or does not grant access depending on the actuator control signal A. Non-physical, logical access control is also possible.

[0076] Figure 5 An embodiment is shown in FIG. 4, in which a control system 40 controls a supervisory system 400. This embodiment is largely similar to Figure 5 The embodiments shown in are the same as those of the embodiments shown in . Therefore, only the different aspects will be described in detail. The sensor 30 is configured to detect a scene under supervision. The control system does not necessarily control the actuator 10, but rather controls the display 10a. For example, the machine learning system 60 can determine the classification of a scene, such as whether a scene detected by the optical sensor 30 is suspicious. The actuator control signal A transmitted to the display 10a can then, for example, be configured so that the display 10a adjusts the displayed content depending on the determined classification, such as highlighting an object that is considered suspicious by the machine learning system 60.

[0077] Figure 6 , in which the control system 40 is used to control the automated personal assistant 250. The sensor 30 may be an optical sensor, for example, for receiving a video image of the gesture of the user 249. Alternatively, the sensor 30 may also be an audio sensor, for example, for receiving a voice command of the user 249.

[0078] The control system 40 then determines an actuator control command A for controlling the automated personal assistant 250. The actuator control command A is determined based on the sensor signal S of the sensor 30. The sensor signal S is transmitted to the control system 40. For example, the classifier 60 can be configured to, for example, implement a gesture recognition algorithm to identify the gesture made by the user 249. The control system 40 can then determine the actuator control command A for transmission to the automated personal assistant 250. It then transmits the actuator control command A to the automated personal assistant 250.

[0079] For example, the actuator control command A may be determined from the identified user gesture recognized by the classifier 60. It may then include information that causes the automated personal assistant 250 to retrieve information from a database and output the retrieved information in a form suitable for the user 249 to receive.

[0080] In a further embodiment, it is contemplated that, instead of the automated personal assistant 250, the control system 40 controls a home appliance (not shown) that is controlled according to the identified user gesture. The home appliance may be a washing machine, a stove, an oven, a microwave or a dishwasher.

[0081] Figure 7 , an embodiment of a control system 40 for controlling an imaging system 500 is shown, the imaging system 500 being, for example, an MRI device, an x-ray imaging device, or an ultrasound imaging device. The sensor 30 may be, for example, an imaging sensor. The machine learning system 60 may then determine a classification of all or part of the sensed image. The actuator control signal A may then be selected based on the classification to control the display 10a. For example, the machine learning system 60 may interpret an area of ​​the sensed image as a potential anomaly. In this case, the actuator control signal A may be determined so that the display 10a displays the imaging and highlights the possible abnormal area.

[0082] Figure 8 An embodiment of a system 500 is shown in FIG. The system 500 comprises a provider system 51 that provides input images. The input images are fed to an encoder 52 that determines a latent representation from them. The latent representation is supplied to an evaluator 53 that determines a classification as discussed above in S26.

[0083] The processes performed by training device 500 may be implemented as a computer program stored on machine-readable storage medium 54 and executed by processor 55 .

[0084] The term "computer" encompasses any device for the processing of predefined computing instructions. These computing instructions may be in the form of software, or in the form of hardware, or in the form of a mixture of software and hardware.

[0085] It should be further understood that the processes may not only be implemented entirely in software, as described, but they may also be implemented in hardware or in a hybrid form of software and hardware.

Claims

1. A computer-implemented method (20) for classifying a sensor signal (30) by means of a sensor signal encoder and a text encoder, wherein the encoder is configured to encode its input into a latent representation, comprising the steps of: The sensor signal (30) is encoded (S21) by a sensor signal encoder into a first latent representation (e (x) ); Generate (S22) a plurality of text prompts (t ij ), where for each class (c j ) generates several textual hints representing the corresponding classes (t i ←c j ); Each generated text prompt (t ij ) is encoded (S23) as the second potential representation By i ) The average is taken to determine (S24) the class query (q j ); Calculate (S25) the first potential representation (e (x) ) and each class query (q j ) between them; and assigning (S26) the sensor signal (30) to the class having the highest similarity; Features In the second representation Weighted average To implement, the weighted average is implemented by using weights (w i ) for each second representation Weighted addition is performed.

2. The method according to claim 1, characterized in that: The weight (w) is determined by normalizing the predetermined value (ρ) with the softmax function i ).

3. The method according to claim 2, characterized in that By maximizing the first latent representation (e (x) ) and each class query (q j ) to determine the value (ρ), and to The gradient is determined, and it consists in updating the value (ρ) by the gradient.

4. The method according to claim 3, characterized in that: The gradient is determined for the term as follows: where δ ij is the Kronecker delta function, where δ ii =1 and for i≠j, δ ij =0.

5. A method according to any one of the preceding claims, wherein: By passing the class name (c j ) and / or class description into a predefined template for a text prompt to generate an instantiation of the prompt.

6. The method according to any one of claims 1 to 4, wherein: The sensor signal encoder and text encoder are basic models.

7. A method according to any one of the preceding claims, wherein: The sensor signal is a digital image, in particular a video, radar, LiDAR, ultrasound, motion, thermal image, and / or the class is a technical class, in particular a class of an object.

8. Method according to any of the preceding claims, further comprising the step of determining the actuator control signal (A) in dependence on the assigned class of the sensor signal (30).

9. The method according to claim 8, wherein: The actuator (10) controls an at least partially autonomous robot (100) and / or a manufacturing machine (200) and / or an access control system (300).

10. A computer program configured to cause a computer to carry out the method according to any one of claims 1 to 9 and all the steps thereof, when the computer program is executed by a processor (45, 145).

11. A machine-readable storage medium (46, 146) having stored thereon a computer program according to claim 10.

12. An apparatus (140) configured to carry out the method according to any one of claims 1 to 9.

Images