Multi-modal learning method and system for partial labels, and storage medium

Abstract

The invention discloses a partial-label-oriented multi-modal learning method and system and a storage medium, and the method comprises the following steps: S1, processing a training data set comprising an image and corresponding text description, and obtaining a training data set with partial labels; s2, according to the processed training data set, constructing a partial-label-oriented multi-modal learning model for cross-modal retrieval; s3, inputting a data mode needing to be retrieved into the multi-mode learning model, and calculating the similarity between the data mode and the data in the retrieval library; s4, similarity sorting is carried out according to the calculated data similarity, and a cross-modal retrieval result is output; according to the method, the problems of label ambiguity and modal difference faced by cross-modal retrieval with part of labels are solved, and the accuracy of cross-modal retrieval is improved.

Description

Technical Field

[0001] The present invention relates to the field of multimodal learning, and in particular to a multimodal learning method, system and storage medium for partial labels. Background Art

[0002] Multimodal learning is a method of learning using data from different senses or interaction modes. These data modalities may include text, images, audio, video, etc. By integrating multiple data modalities to train the model, multimodal learning can improve the perception and understanding ability of the model and realize cross-modal information interaction and fusion. Cross-modal retrieval is a classic multimodal learning task that can find relevant data from different modalities. In the cross-modal retrieval task for partial labels, each instance is associated with a set of candidate labels. Only one of the candidate label sets is the true label, and the others are all wrong labels, i.e., label ambiguity. This annotation method can significantly reduce the annotation cost, especially when processing large-scale data.

[0003] In recent years, researchers have proposed many cross-modal retrieval methods, which can be generally divided into three categories according to the different data annotation forms: supervised cross-modal retrieval methods, unsupervised cross-modal retrieval methods, and semi-supervised cross-modal retrieval methods. Specifically, 1) supervised cross-modal retrieval methods use annotated labels to learn the common representation of data in different modalities. Although they can provide accurate retrieval results in theory, these methods usually require a large amount of unambiguous annotated data, the annotation cost is high, and they cannot handle the scene with partial label annotation. 2) Unsupervised cross-modal retrieval methods learn modality-specific features by maximizing the correlation between different modalities. Although they can be independent of label information, most of them suffer from insufficient performance due to the lack of supervision information. 3) Semi-supervised cross-modal retrieval methods balance the two pain points of high annotation cost and insufficient supervision information to a certain extent. Although these methods reduce the dependence on a large number of clean labels, they still require a certain amount of clean label information, and cannot handle the label ambiguity problem faced by cross-modal retrieval of partial labels and the need to further narrow the modality difference.

[0004] In summary, cross-modal retrieval for partial tags faces the problems of both label ambiguity and modality differences. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a multimodal learning method, system and storage medium for partial labels, which solves the label ambiguity and modality difference problems of cross-modal retrieval with partial labels and improves the accuracy of cross-modal retrieval.

[0006] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0007] This solution provides a multimodal learning method, system and storage medium for partial labels, including the following steps:

[0008] S1: Processing a training dataset including images and corresponding text descriptions to obtain a training dataset with partial labels;

[0009] S2: Based on the processed training dataset, a multimodal learning model for partial labels is constructed for cross-modal retrieval;

[0010] S3: Input the data modality to be retrieved into the multimodal learning model and calculate its similarity with the data in the retrieval library;

[0011] S4: Sort the similarity according to the calculated data similarity and output the cross-modal retrieval results.

[0012] The beneficial effect of the present invention is to learn discriminative representations from ambiguous partial labels while eliminating the differences between modalities, thereby solving the label ambiguity and modality difference problems of cross-modal retrieval with partial labels and improving the accuracy of cross-modal retrieval.

[0013] Furthermore, step S1 includes the following sub-steps:

[0014] Furthermore, the input space is The label space is represented as Among them, K represents the number of categories in the data set. The expressions of the processed image training set and text description training set are:

[0015]

[0016] Where N represents the number of sample pairs in the training set. 和 表示 The i-th image sample and candidate label set. 和 表示 The th text description sample and candidate label set.

[0017] The beneficial effect of the above further scheme is that after the cross-modal retrieval training set is processed as above, a cross-modal retrieval dataset with partial labels can be obtained, which is used to study cross-modal retrieval methods for partial labels.

[0018] Further, step S2 includes the following sub-steps:

[0019] S21: Extract feature vectors of samples in the input model using the encoder;

[0020] S22: Use the softmax function to obtain the probability distribution of the sample belonging to each label;

[0021] S23: extracting the disambiguated label of each sample during the training process through the obtained sample label probability distribution;

[0022] S24: By considering the loss on both the candidate label set and the non-candidate label set, the model is encouraged to reduce the prediction probability of non-candidate labels while disambiguating the candidate label set;

[0023] S25: Use the labels that the model eliminates ambiguity during training as pseudo labels and perform instance-level cross-modal contrastive learning to eliminate modality differences;

[0024] S26: By maintaining the same number of category prototype vectors as the number of categories for image data and text data respectively, and then performing cross-modal alignment at the prototype level, the cross-modal differences can be further narrowed.

[0025] S27: The above process is executed in a loop, and the update of the model parameters on the training data set is guided by the total loss function.

[0026] Furthermore, the formula for extracting the feature vector in step S21 is:

[0027] q=f(x)

[0028] Among them, q represents the extracted sample feature vector, x represents the sample input to the model, and f represents the feature encoder.

[0029] Furthermore, the formula for obtaining the probability distribution of the sample belonging to each label in step S22 is as follows:

[0030] z=g(q)

[0031] Among them, q represents the extracted sample feature vector, g(·) represents the softmax function, and z represents the probability distribution of the obtained sample belonging to each label.

[0032] Furthermore, the formula for obtaining the disambiguated label in step S23 is as follows:

[0033]

[0034] Among them, z I and z T Represents image samples x I and text description sample x T The probabilities of belonging to the corresponding categories in image and text modalities, respectively, represents the Hadamard product, y I and TRepresents the candidate label sets for image and text modalities respectively. and denote the label confidence after disambiguation of image and text modalities, respectively.

[0035] Furthermore, the disambiguation loss expression of the image and text modalities in step S24 is:

[0036]

[0037] Where ω is a hyperparameter used to balance the disambiguation loss on the candidate label set and the non-candidate label set. f(x) represents the output of the model. p(y=c|x) represents the label confidence given by the model that sample x belongs to category c. p(y=c|x, f(x)) represents the predicted probability after applying the softmax function.

[0038] Therefore, the overall disambiguation loss expression of the model is:

[0039]

[0040] in, and denote the disambiguation loss of image and text modalities respectively, Represents the total disambiguation loss used by the model.

[0041] The beneficial effect of the above further scheme is that it utilizes the information in the candidate label set and the non-candidate label set at the same time, projects the data of different modalities into a common space, effectively solves the ambiguity problem of labels, and reduces the differences between modalities.

[0042] Furthermore, the instance-level cross-modal contrastive learning in step S25 first needs to define a pseudo-label used as supervisory information, and the calculation expression of the pseudo-label is:

[0043]

[0044] in, and denote the confidence lists of labels obtained by image and text modalities respectively, argmax(·) denotes the operation of finding the label with the highest confidence. and Represent the pseudo labels of the acquired image and text modalities respectively.

[0045] In order to maximize the consistency between different modalities in the common space, we maximize the number of samples with the same pseudo labels between different modalities. The distance between instances of The probability of the jth instance belonging to one of the m modes can be defined as:

[0046]

[0047] Where m represents the number of modalities, which is 2, l represents the modality selection, and l is 1 and 2 respectively for image and text modalities. N represents the total number of sample pairs. represents the jth instance belonging to m modes, Representation sample The probability of belonging to the jth instance among m modes, represents the calculation of the similarity between two sample features, τ is the temperature hyperparameter, Indicates the operation of selecting samples from all modalities l with the same pseudo label as the sample. exp indicates exponential operation.

[0048] Therefore, the expression of instance-level cross-modal contrastive learning loss is:

[0049]

[0050] Among them, m represents the number of modes, the size is 2, N represents the total number of sample pairs, Representation sample The probability of the jth instance belonging to one of the m modes, log represents the logarithm operation.

[0051] The beneficial effect of the above further scheme is that it performs instance-level cross-modal contrastive learning in the presence of label ambiguity, significantly reducing the heterogeneity differences between different modalities.

[0052] Furthermore, the cross-modal alignment module in step S26 first needs to maintain a category prototype for each category in the image and text modalities, which is expressed as follows:

[0053]

[0054] in, and Represents the category prototype vector of the Kth category in the image and text modalities, V I and V T List of category prototypes representing image and text modalities respectively.

[0055] The category prototype is updated as follows:

[0056]

[0057] λ(t)=1-t / E

[0058] in, and Represents the category prototype vector of the Kth category in the image and text modalities, qI and q T Represent the feature vectors of image and text modalities respectively, y represents the number of training rounds of the current model, E is the total number of iterations, λ(t) is the dynamic parameter that balances the current prototype vector and feature vector, and denote the pseudo labels of image and text modality samples respectively, and k denotes the kth category.

[0059] The expression of the cross-modal alignment loss at the category prototype level for image and text modalities is:

[0060]

[0061] Among them, q I and q T Represent the feature vectors of image and text modalities respectively, and Represent the k-th category prototype vector for image and text modalities respectively. and denote the cross-modal alignment losses for image and text modalities, respectively.

[0062] The overall cross-modal alignment loss expression is:

[0063]

[0064] in represents the total cross-modal alignment loss, and denote the cross-modal alignment losses for image and text modalities, respectively.

[0065] The beneficial effect of the above further scheme is to perform cross-modal alignment at the category level, further narrowing the cross-modal differences.

[0066] Furthermore, the total loss function expression of the model in step S27 is as follows:

[0067]

[0068] in, is the total loss function of the modality, For the disambiguation loss, is the cross-modal contrastive learning loss, is the cross-modal alignment loss. α and β are hyperparameters.

[0069] The beneficial effect of the above further scheme is that it guides the optimization of the cross-modal retrieval model through disambiguation loss, cross-modal contrastive learning loss, and cross-modal alignment loss, which solves the label ambiguity and modality difference problems faced by the model, so that the trained model can effectively solve the cross-modal retrieval problem under partial label conditions.

[0070] Furthermore, the method of step S3 is: input the image or text modality test sample used for query into the retrieval model trained in the previous step, and calculate the similarity between another modality data in the retrieval library and the query sample.

[0071] Furthermore, the method of step S4 is: sorting according to the calculated data similarity, and outputting the retrieved data with the highest similarity as the retrieval result.

[0072] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0073] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to execute the steps of the above method.

[0074] The beneficial effect of the present invention is to learn discriminative representations from ambiguous partial labels while eliminating inter-modal differences, thereby solving the label ambiguity and modality difference problems of cross-modal retrieval with partial labels and improving the accuracy of cross-modal retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. It should be understood that the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0076] Figure 1 A step diagram of a multimodal learning method, system, and storage medium for partial labels provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0078] like Figure 1 As shown, the present invention provides a multimodal learning method, system and storage medium for partial labels, and the implementation method thereof is as follows:

[0079] S1: Processing a training dataset including images and corresponding text descriptions to obtain a training dataset with partial labels;

[0080] S2: Based on the processed training dataset, a multimodal learning model for partial labels is constructed for cross-modal retrieval;

[0081] S3: Input the data modality to be retrieved into the multimodal learning model and calculate its similarity with the data in the retrieval library;

[0082] S4: Sort the similarity according to the calculated data similarity and output the cross-modal retrieval results.

[0083] In this embodiment, step S2 includes the following sub-steps:

[0084] S21: Extract feature vectors of samples in the input model using the encoder;

[0085] S22: Use the softmax function to obtain the probability distribution of the sample belonging to each label;

[0086] S23: extracting the disambiguated label of each sample during the training process through the obtained sample label probability distribution;

[0087] S24: By considering the loss on both the candidate label set and the non-candidate label set, the model is encouraged to reduce the prediction probability of non-candidate labels while disambiguating the candidate label set;

[0088] S25: Use the labels that the model eliminates ambiguity during training as pseudo labels and perform instance-level cross-modal contrastive learning to eliminate modality differences;

[0089] S26: By maintaining the same number of category prototype vectors for image data and text data respectively, and then performing cross-modal alignment at the prototype level, the cross-modal differences can be further reduced;

[0090] S27: The above process is executed in a loop, and the update of the model parameters on the training data is guided by the total loss function.

[0091] In this embodiment, the formula for extracting the feature vector in step S21 is as follows:

[0092] q=f(x)

[0093] Among them, q represents the extracted sample feature vector, x represents the sample input to the model, and f represents the feature encoder.

[0094] The formula for obtaining the probability distribution of the sample belonging to each label in step S22 is as follows:

[0095] z=g(q)

[0096] Among them, q represents the extracted sample feature vector, g(·) represents the softmax function, and z represents the probability distribution of the obtained sample belonging to each label.

[0097] The formula for obtaining the disambiguated label in step S23 is as follows:

[0098]

[0099] Among them, z I and z T Represents image samples x I and text description sample x T The probabilities of belonging to the corresponding categories in image and text modalities, respectively, represents the Hadamard product, y I and T Represents the candidate label sets for image and text modalities respectively. and denote the label confidence after disambiguation of image and text modalities, respectively.

[0100] The disambiguation loss expression of the image and text modalities in step S24 is:

[0101]

[0102] Where ω is a hyperparameter used to balance the disambiguation loss on the candidate label set and the non-candidate label set. f(x) represents the output of the model. p(y=c|x) represents the label confidence given by the model that sample x belongs to category c. p(y=c|x, f(x)) represents the predicted probability after applying the softmax function.

[0103] The overall disambiguation loss expression of the model is:

[0104]

[0105] in, and denote the disambiguation loss of image and text modalities respectively, Represents the total disambiguation loss used by the model.

[0106] The instance-level cross-modal contrastive learning in step S25 first needs to define pseudo labels used as supervisory information. The pseudo labels are obtained as follows:

[0107]

[0108] in, and denote the confidence lists of labels obtained by image and text modalities respectively, argmax(·) denotes the operation of finding the label with the highest confidence. and Represent the pseudo labels of the acquired image and text modalities respectively.

[0109] In order to maximize the consistency between different modalities in the common space, we maximize the number of samples with the same pseudo labels between different modalities. The distance between instances of The probability of the jth instance belonging to one of the m modes can be defined as follows:

[0110]

[0111] Where m represents the number of modalities, which is 2, l represents the modality selection, and l is 1 and 2 respectively for image and text modalities. N represents the total number of sample pairs. represents the jth instance belonging to m modes, Representation sample The probability of belonging to the jth instance among m modes, represents the calculation of the similarity between two sample features, τ is the temperature hyperparameter, Indicates the operation of selecting samples from all modalities l with the same pseudo label as the sample. exp indicates exponential operation.

[0112] Therefore, the expression of instance-level cross-modal contrastive learning loss is as follows:

[0113]

[0114] Among them, m represents the number of modes, the size is 2, N represents the total number of sample pairs, Representation sample The probability of the jth instance belonging to one of the m modes, log represents the logarithm operation.

[0115] The cross-modal alignment module in step S26 first needs to maintain a category prototype for each category in the image and text modalities, which is expressed as follows:

[0116]

[0117] in, and Represents the category prototype vector of the kth category in the image and text modalities, V I and V T List of category prototypes representing image and text modalities respectively.

[0118] The category prototype is updated as follows;

[0119]

[0120] in, and Represents the category prototype vector of the kth category in the image and text modalities, q I and qT They represent the feature vectors of image and text modalities respectively, t represents the number of training rounds of the current model, E is the total number of iterations, λ(t) is the dynamic parameter that balances the current prototype vector and feature vector, and denote the pseudo labels of image and text modality samples respectively, and k denotes the kth category.

[0121] Therefore, the cross-modal alignment loss expression at the category prototype level for image and text modalities is:

[0122]

[0123] Among them, q I and q T Represent the feature vectors of image and text modalities respectively, and Represent the k-th category prototype vector for image and text modalities respectively. and denote the cross-modal alignment losses for image and text modalities, respectively.

[0124] The overall cross-modal alignment loss expression is as follows:

[0125]

[0126] in represents the total cross-modal alignment loss, and denote the cross-modal alignment losses for image and text modalities, respectively.

[0127] The total loss function expression of the model in step S27 is:

[0128]

[0129] in, is the total loss function of the modality, For the disambiguation loss, is the cross-modal contrastive learning loss, is the cross-modal alignment loss. α and β are hyperparameters.

[0130] In this embodiment, the method of step S3 is: input the image or text modality test sample used for query into the retrieval model trained in the previous step, and calculate the similarity between another modality data in the retrieval library and the query sample.

[0131] In this embodiment, the method of step S4 is: sorting the data according to the calculated data similarity, and outputting the retrieved data with the highest similarity as the retrieval result.

[0132] This embodiment provides a computer device, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the steps of the above-mentioned multimodal learning method for partial labels.

[0133] The computer device may be a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with the user through a keyboard, a mouse, a remote control, a touch pad, or a voice control device.

[0134] The memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of the computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Of course, the memory may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is often used to store the operating system and various application software installed on the computer device, such as the program code of the multimodal learning method for partial labels, etc. In addition, the memory may also be used to temporarily store various types of data that have been output or are to be output.

[0135] The processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor is generally used to control the overall operation of the computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data, such as running the program code of the multimodal learning method for partial labels.

[0136] This embodiment provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes the steps of the above-mentioned multimodal learning method for partial labels.

[0137] The computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor so that the at least one processor performs the steps of the multimodal learning method for partial labels as described above.

[0138] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including a number of instructions for a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the multimodal learning method for partial labels described in the embodiment of the present application.

[0139] The present invention provides a multimodal learning method, system and storage medium for partial labels, which solve the label ambiguity and modality difference problems of cross-modal retrieval with partial labels and improve the accuracy of cross-modal retrieval.

Claims

1. A multimodal learning method, system and storage medium for partial labels, characterized in that: The method comprises: S1: Processing a training dataset including images and corresponding text descriptions to obtain a training dataset with partial labels; S2: Based on the processed training dataset, a multimodal learning model for partial labels is constructed for cross-modal retrieval; S3: Input the data modality to be retrieved into the multimodal learning model and calculate its similarity with the data in the retrieval library; S4: perform similarity sorting based on the calculated data similarity and output cross-modal retrieval results; Step S2 specifically includes: S21: Extract feature vectors of samples in the input model using the encoder; S22: Use the softmax function to obtain the probability distribution of the sample belonging to each label; S23: extracting the disambiguated label of each sample during the training process through the obtained sample label probability distribution; S24: By considering the loss on both the candidate label set and the non-candidate label set, the model is encouraged to reduce the prediction probability of non-candidate labels while disambiguating the candidate label set; S25: Use the labels that the model eliminates ambiguity during training as pseudo labels and perform instance-level cross-modal contrastive learning to eliminate modality differences; S26: By maintaining the same number of category prototype vectors for image data and text data respectively, and then performing cross-modal alignment at the prototype level, the cross-modal differences can be further reduced; S27: executing the above process in a loop, guiding the update of model parameters on the training data set through the total loss function; The disambiguation loss expressions of the image and text modalities in step S24 are as follows: Where ω is a hyperparameter used to balance the disambiguation loss on the candidate label set and the non-candidate label set, f(x) represents the output of the model, p(y=c|x) represents the label confidence given by the model that the sample x belongs to category c, and p(y=c|x, f(x)) represents the predicted probability after applying the softmax function. Therefore, the overall disambiguation loss expression of the model is: in, and denote the disambiguation loss of image and text modalities respectively, represents the total disambiguation loss used by the model; S3: Input the data modality to be retrieved into the multimodal learning model and calculate its similarity with the data in the retrieval library; S4: Sort the similarity according to the calculated data similarity and output the cross-modal retrieval results.

2. The method, system and storage medium for partial labeling according to claim 1, characterized in that: The instance-level cross-modal contrastive learning in step S25 first needs to define a pseudo-label used as supervisory information. The calculation expression of the pseudo-label is: in, and Represents the label confidence list obtained by image and text modality respectively, argmax() represents the operation of finding the label with the highest confidence. and They represent the pseudo labels of the acquired image and text modalities respectively. The expression of the instance-level cross-modal contrastive learning loss is: Among them, m represents the number of modes, the size is 2, N represents the total number of sample pairs, and log represents the logarithmic operation. Representation sample The probability of belonging to the jth instance in m modes is calculated as: Among them, m represents the number of modalities, the size is 2, l represents the modality selection, l takes 1 and 2 to represent image and text modalities respectively, and N represents the total number of sample pairs. represents the jth instance belonging to m modes, Representation sample The probability of belonging to the jth instance among m modes, represents the calculation of the similarity between two sample features, τ is the temperature hyperparameter, It means to select samples from all modalities l with the same pseudo label as the sample, and exp means to perform exponential operation.

3. The method, system and storage medium for partial labeling according to claim 1, characterized in that: The cross-modal alignment module in step S26 first needs to maintain a category prototype for each category in the image and text modalities, which is expressed as follows: in, and Represents the category prototype vector of the Kth category in the image and text modalities, V I and V T A list of category prototypes representing image and text modalities respectively, The class prototype is updated as follows: λ(t)=1-t / E in, and Represents the category prototype vector of the Kth category in the image and text modalities, q I and q T They represent the feature vectors of image and text modalities respectively, t represents the number of training rounds of the current model, E is the total number of iterations, λ(t) is the dynamic parameter that balances the current prototype vector and feature vector, and denote the pseudo labels of the image and text modality samples respectively, k denotes the kth category, and the cross-modal alignment loss at the category prototype level of the image and text modalities is: Among them, q I and q T Represent the feature vectors of image and text modalities respectively, and Represents the k-th category prototype vector of image and text modality respectively, and They represent the cross-modal alignment losses of image and text modalities respectively, and the total cross-modal alignment loss expression is: in represents the total cross-modal alignment loss, and denote the cross-modal alignment losses for image and text modalities, respectively.

4. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 3.

5. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 3.

Images