Method, apparatus, device and storage medium for extracting features

Abstract

According to embodiments of this disclosure, a method, apparatus, device, and storage medium for feature extraction are provided. The method includes: generating fused features for an input sample based on first sample features of an input sample and a reference feature library representing a set of reference samples, wherein the input sample and reference samples in the reference sample library are samples of the same type. The method further includes: updating the first sample features with second sample features of the input sample based on the fused features. The method further includes: determining a processing result associated with the input sample based on the second sample features. In this manner, the representational power of the sample features is improved, thereby contributing to improved accuracy of the processing result.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatus, devices, and computer-readable storage media for extracting features. Background Technology

[0002] Artificial neural networks that mimic the actual neural networks of biological organisms (such as humans) are generally referred to as neural network models or simply neural networks. Neural network models hold broad and attractive prospects in many fields, including image processing and natural language processing. Feature extraction using neural network models has already been applied in many real-world applications. A neural network model can process input samples into features and determine the processing result based on the obtained features. The ability of the extracted features to represent the samples affects the accuracy of the processing result. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for extracting features is provided. The method includes: generating fused features for the input sample based on first sample features of an input sample and a reference feature library representing a reference sample set, wherein the input sample and the reference samples in the reference sample set are samples of the same type. The method further includes: updating the first sample features with second sample features of the input sample based on the fused features. The method further includes: determining a processing result associated with the input sample based on the second sample features.

[0004] In a second aspect of this disclosure, an apparatus for feature extraction is provided. The apparatus includes: a fusion feature generation module configured to generate fusion features for the input sample based on first sample features of an input sample and a reference feature library representing a reference sample set, wherein the input sample and the reference samples in the reference sample set are samples of the same type. The apparatus further includes: a sample feature updating module configured to update the first sample features to second sample features of the input sample based on the fusion features. The apparatus further includes: a processing result generation module configured to determine a processing result associated with the input sample based on the second sample features.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this summary section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0010] Figure 2 A schematic diagram of an example method for extracting features according to embodiments of the present disclosure is shown;

[0011] Figure 3 An example architecture for feature extraction according to some embodiments of this disclosure is shown;

[0012] Figure 4 An example process for generating fusion features according to some embodiments of this disclosure is shown;

[0013] Figure 5 Examples of constructing a reference feature library according to some embodiments of this disclosure are shown;

[0014] Figure 6 A block diagram of an apparatus for feature extraction according to some embodiments of the present disclosure is shown; and

[0015] Figure 7 A block diagram of an apparatus capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0016] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0017] For example, when the received sample includes a facial image, a prompt message is sent to the sample sender and provider to explicitly inform them that the requested operation involves the use of a facial image and requires the provision of authorization information related to the content included in the facial image. This allows the sample sender and provider to autonomously choose, based on the prompt message, whether to provide the facial image and corresponding authorization information to the software or hardware such as electronic devices, applications, servers, or storage media performing the operations of this disclosed technical solution.

[0018] As an optional but non-limiting implementation, the prompt message can be presented as a pop-up window, where the message can be displayed in text format. Furthermore, the pop-up window can also include a selection control for the user to choose "upload authorization information," allowing the sample sender and the provider to provide authorization information via this selection control.

[0019] It is understood that the above notification and authorization process is merely illustrative and does not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0020] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0021] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0022] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0023] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0024] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. In this paper, "model" may also be referred to as a "machine learning model," a "machine learning network," or simply a "network," and these terms are used interchangeably. A model can also include different types of processing units or networks.

[0025] Example Environment

[0026] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In environment 100, input sample 110 is data that is expected to be processed using sample processing model 121 to obtain a corresponding processing result. Input sample 110 can be data in any suitable form. For example, input sample 110 may include media data, such as images, audio, text, etc. As another example, input sample 110 may include digital signals, such as medical signals.

[0027] The processing result 130 corresponds to the processing objective, such as image classification, image recognition, or image feature extraction. The sample processing model 121 is used to achieve the processing objective of the input sample 110, thereby obtaining the processing result 130. Depending on the type of input sample 110 and the specific processing objective, the sample processing model 121 can be a model that implements various functions. For example, if the input sample 110 includes images, the sample processing model 121 can be an image processing model, such as an image classification model, an image recognition model, an image feature extraction model, or a species identification model. Similarly, if the input sample 110 includes text, the sample processing model 121 can be a natural language processing model. If the input sample 110 includes both images and text, the sample processing model 121 can be a cross-modal processing model. The sample processing model 121 can include any suitable algorithm or network, such as, but not limited to, a neural network model.

[0028] The processing result 130 is the content obtained by processing the input sample 110 through the sample processing model 121. The processing result 130 is related to the processing purpose of the input sample 110, such as image classification result, image recognition result, image feature extraction result, etc.

[0029] As an example, input sample 110 includes images, and the processing objective is image classification. Accordingly, sample processing model 121 is an image classification model, and the processing result includes the classification result of the input sample.

[0030] In environment 100, sample processing model 121 can be deployed on computing device 120. Computing device 120 can be any type of device with computing capabilities, such as a terminal device or a server device. Terminal devices can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. Server devices can include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0031] It should be understood that the structure and function of environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. In environment 100, in order to obtain processing result 130, sample processing model 121 needs to extract features from input sample 110.

[0032] Taking neural network models as an example, as briefly mentioned earlier, current neural network models extract only one corresponding feature for a specific individual sample. This approach of single individual input and single feature output does not consider the overall or global features of the relevant dataset or database. This results in weak representational power and low quality of the extracted features, thus affecting the performance of the sample processing model or the accuracy of the processing results. For example, weak representational power of extracted image features will reduce the accuracy of image classification.

[0033] Embodiments of this disclosure propose a scheme for feature extraction. According to various embodiments of this disclosure, fused features for the input sample are generated based on first sample features of the input sample and a reference feature library representing a reference sample set. The input sample and the reference samples in the reference sample set are samples of the same type. The first sample features are updated to second sample features of the input sample using the fused features. Then, a processing result associated with the input sample is determined based on the second features. Thus, the sample features of the input sample are enhanced based on reference features of samples of the same type as the input sample in the reference feature library, improving the representational ability of the sample features. Furthermore, the improved representational ability helps to improve the accuracy of the processing result.

[0034] Example Process

[0035] Figure 2A schematic diagram of an example method 200 for extracting features according to some embodiments of the present disclosure is shown. Method 200 may be implemented, for example, at a computing device 120. Reference is made below for illustration only. Figure 1 Let's describe method 200.

[0036] In box 210, computing device 120 generates fused features for input sample 110 based on first sample features of input sample 110 and a reference feature library representing a reference sample set. Input sample 110 and reference samples in the reference sample set are samples of the same type. For example, input sample 110 and reference samples can both be images. Alternatively, input sample 110 and reference samples can both be natural language text. Yet another example is that input sample 110 and reference samples can both be digital signals.

[0037] In some embodiments, the first sample features of the input sample 110 may be obtained externally, for example, generated by other models or units. In this case, the sample processing model 121 can directly use the first sample features input from the outside to determine the corresponding fused features.

[0038] In some embodiments, the first sample feature of the input sample 110 may be generated by the sample processing model 121. For example, the computing device 120 may feed the received input sample 110 to the sample processing model 121. The sample processing model 121 may process the received input sample 110 to obtain the first sample feature of the input sample 110. In this case, the sample processing model 121 may include a sample feature extraction unit configured to extract features from the input sample.

[0039] Figure 3 An example architecture 300 for feature extraction according to some embodiments of the present disclosure is shown. Architecture 300 illustrates an example implementation of sample processing model 121. Input image 310 can be considered as an example of input sample 110. Figure 3 The input image 310 is shown as a face image, but this is merely exemplary and not intended to limit the scope of this disclosure. Figure 3 In the example, the sample processing model 121 includes a sample feature extraction unit 320 configured to extract features from the input samples. The sample feature extraction unit 320 generates first sample features 330 of the input image 310.

[0040] The sample processing model 121 also utilizes a reference feature library constructed from a reference sample set. In some embodiments, the reference feature library may be stored locally on the device or unit including the sample processing model 121. For example, the reference feature library may be implemented as metadata of the sample processing model 121. In some embodiments, the reference feature library may be stored independently of the sample processing model 121. The sample processing model 121 may retrieve the reference feature library from the device or unit storing the reference feature library by sending a request.

[0041] continue Figure 3 The following examples illustrate the reference feature library. Reference feature library 340 represents reference sample set 350. Reference sample set 350 includes reference samples 351-1, 351-2, 351-3….351-N, which are also collectively referred to or individually as reference samples 351. N is a positive integer. Reference feature library 340 includes reference features 341-1, 341-2, 341-3….341-M, which are also collectively referred to or individually as reference features 341. M is a positive integer.

[0042] In some embodiments, the reference feature library 340 may include features of each reference sample 351 in the reference sample set 350. For example, any reference feature 341 is a feature of a reference sample 351. In this case, M equals N. In some embodiments, the reference feature library 340 may include representative features of the reference samples in the reference sample set 350. For example, the features of all reference samples 351 may be clustered. The features of the resulting cluster centers can be considered representative features and added to the reference feature library 340.

[0043] In some embodiments, the reference feature library may include reference features corresponding to multiple predetermined categories. For example, reference samples in the reference sample set 350 may be classified into multiple predetermined categories. Each reference feature 341 represents a sample of a predetermined category. In this case, the value of M may be equal to the number of multiple predetermined categories. In particular, the reference feature library can be implemented in this manner when the sample processing model 121 is a classification model.

[0044] In some embodiments, the features of the reference samples used to construct the reference feature library can be generated by any known or future-developed feature extraction model or algorithm. In some embodiments, the features of the reference samples used to construct the reference feature library can be generated by the sample processing model 121 during training. That is, the construction of the reference feature library can be combined with the training of the sample processing model 121. The following will refer to Figure 5 Such embodiments are described further.

[0045] Based on the obtained first sample features and the reference feature library, fused features are generated for input sample 110. The first sample features reflect the individual features of the input sample. In contrast, the reference feature library can reflect the group features or overall features of the reference sample set, which can be regarded as a kind of global feature. In this case, the fused features integrate the individual features and the global features.

[0046] continue Figure 3 Examples are used to describe some implementation methods. Figure 3 As shown, a fusion feature 360 ​​is generated based on the first sample feature 330 and the reference feature library 340. The fusion feature 360 ​​can be generated in any suitable manner. In some embodiments, the weighted sum of the first sample feature 330 and each of the reference features 341 can be used as the fusion feature 360. In some embodiments, one or more reference features 341 similar to the first sample feature 330 can be selected from the reference feature library 340. For example, the first sample feature 330 and each of the reference features 341 can be clustered, and reference features 341 belonging to the same cluster as the first sample feature 330 can be selected. Then, the fusion feature 360 ​​can be generated based on the selected one or more reference features 341. For example, the weighted sum of the first sample feature 330 and the selected one or more reference features 341 can be used as the fusion feature 360. In some embodiments, the sample processing model 121 can determine the correlation between the first sample feature 330 and each of the reference features 341 in the reference feature library 340. Then, the fusion feature can be determined based on the correlation and at least one reference feature in the reference feature library 340. In this embodiment, by taking into account the correlation between the first sample features and each reference feature, the fused features can be made to better match the features of the input sample. In this way, the representation quality can be further improved.

[0047] In some embodiments, reference feature 341 is determined based on the features of reference samples 351 included in the same reference sample set. It should be understood that the reference samples included in the same reference sample set correspond to the construction criteria of the reference sample set. For example, multiple reference sample sets are constructed based on the category to which the image belongs. Each reference sample 351-1, 351-2…351-N in the reference sample set 350, which belongs to the category of face images, is a face image. Features of each reference sample can be extracted separately, and reference feature 341 for a reference sample set whose type includes face images can be formed based on the features of each reference sample. In some embodiments, the common parts of the features of each reference sample in the same reference sample set can be determined as reference feature 341. In some embodiments, the cluster centers can be used as reference features 341 after clustering the features of each reference sample in the reference sample set.

[0048] Continue to refer to Figure 2In box 220, computing device 120 updates the first sample features to the second sample features of the input sample based on the fusion features. The fusion features corresponding to the input sample 110 are extracted from the reference feature library. The first sample features are updated based on the fusion features to obtain the second sample features. The second sample features obtained in this way can represent both the individual features of the input sample 110 and the fusion features determined based on the reference feature library. Therefore, the second sample features can represent both the original individual sample features and the group or global features of the reference samples.

[0049] Any suitable quantization representation can be used to represent the first sample features, the fused features, and the second sample features. Such quantization representations can include, but are not limited to, feature vectors, feature matrices, and eigenvalues. In box 230, the first sample features can be updated to the second sample features in any suitable manner. See below for further details. Figure 3 To describe some embodiments.

[0050] In some embodiments, the quantized representations of the first sample feature 330 and the quantized representations of the fused feature 360 ​​can be element-wise added to obtain the quantized representation of the second sample feature 370. For clarity, an example is provided where both the first sample feature 330 and the fused feature 360 ​​are feature matrices. The corresponding elements of the feature matrices of the first sample feature 330 and the fused feature 360 ​​can be element-wise added. The resulting feature matrix is ​​then used as the feature matrix of the second sample feature 370. By using element-wise addition, the fused features can be easily incorporated into the second sample feature.

[0051] Alternatively, in some embodiments, the first sample feature 330 and the fused feature 360 ​​can be concatenated together to form the second sample feature 370. For example, when the first sample feature 330 and the fused feature 360 ​​are quantized into feature vectors, the beginning of the feature vector of the fused feature 360 ​​can be concatenated with the end of the feature vector of the first sample feature 330 to obtain the feature vector of the second sample feature 370. Furthermore, when the quantized representations of the first sample feature 330 and the fused feature 360 ​​have multiple channels, the quantized representations of the first sample feature 330 and the quantized representations of the fused feature 360 ​​can be concatenated by channel.

[0052] In some embodiments, the first sample feature 360 ​​and the fused feature 370 can be fused by early fusion, late fusion, or other methods to obtain the second sample feature 370.

[0053] Continue to refer to Figure 2In box 230, computing device 120 determines a processing result 130 associated with input sample 110 based on the features of the second sample. Processing result 130 may include, for example, image classification result, image recognition result, etc.

[0054] In some embodiments, the second sample features may be further processed to obtain the processing result 130. For example, the second sample features may be processed using the feature processing unit included in the sample processing model 121 to determine the processing result 130 associated with the input sample 110. Alternatively, the second sample features may be processed using a feature processing unit independent of the sample processing model to determine the processing result 130 associated with the input sample 110.

[0055] Alternatively, in some embodiments, the second sample features may not be processed, and may be directly used as the processing result 130. For example, if the sample processing model 121 is a feature extraction model, the second sample features may be used as the processing result 130.

[0056] for Figure 3 For example, after obtaining the second sample feature 370, the second sample feature 370 can be processed using the feature processing unit (not shown in the figure) included in the sample processing model 121. Thus, the processing result for the input image 310, which is a face image, can be obtained.

[0057] In this scenario, the second sample feature reflects both the original individual characteristics of the input sample and reference features representing the same category as the input sample, serving as global features. Therefore, the sample processing model 121 can use the second sample feature, which has a stronger representational ability than the first sample feature, to process the sample, thus improving the accuracy of the processing results.

[0058] Example generation of fused features

[0059] As referenced above Figure 2 As described, in some embodiments, the correlation between the first sample feature and each reference feature in the reference feature library can be determined separately. For example, the feature vector obtained by processing the first sample feature through a first transformation layer and the matrix obtained by processing the reference feature library through a second transformation layer are multiplied and then normalized to generate the correlation between the reference features in the reference feature library and the first sample feature. The correlation can, for example, reflect the similarity between the first sample feature and the reference features. Further, a fusion feature can be determined based on the correlation between at least one reference feature in the reference feature library and the first sample feature.

[0060] Any suitable method can be used to determine the fusion feature based on relevance. For example, the reference feature with the highest relevance can be used as the fusion feature. Alternatively, the fusion feature can be determined based on the weighted sum of the reference feature with the highest relevance and the first sample feature. Yet another method is to set a relevance threshold. Reference features with relevance exceeding the threshold are then combined with the first sample feature to form the fusion feature.

[0061] Specifically, in some embodiments, the individual reference features can be transformed according to parameters associated with the determination of correlation. Based on the correlation, the transformed reference features are combined into a fused feature. For example, the determination of correlation and the transformation of reference features can be performed by different parts of the same algorithm or model.

[0062] In this embodiment, an attention mechanism can be used. (See reference) Figure 4 Let's describe such an example. The first sample feature 330 is input as a query to the transformation layer 410. The transformation layer 410 then generates the transformed first sample feature 420. This process can be described by equation (1):

[0063] Q = W Q *I (1)

[0064] Where vector I represents the first sample feature 330, vector Q represents the transformed first sample feature 420, and W Q This represents the transformation matrix applied by transformation layer 410.

[0065] The reference feature library 340 is input as a key to the transformation layer 441. The transformation layer 441 then generates the transformed feature library 450. The transformed feature library 450 includes the transformed reference features. This process can be described by equation (2):

[0066] K = W K *U (2)

[0067] Where matrix U represents the reference feature library 340, matrix K represents the transformed feature library 450, and W K This represents the transformation matrix applied by transformation layer 441.

[0068] Based on the transformed first sample feature 420 and the transformed feature library 450, the correlation 430 between the first sample feature 330 and each reference feature 341 can be obtained. This process can be described by equation (3):

[0069] S = Softmax(Q*K) (3)

[0070] Where S represents the correlation coefficient 430, and Softmax represents the normalization of the product of the feature vector Q and the matrix K using logistic regression.

[0071] The reference feature library 340 is also input as a value to the transformation layer 442. The transformation layer 442 then generates the transformed feature library 460. The transformed feature library 460 includes the transformed reference features. This process can be described by equation (4):

[0072] V = W V *U (4)

[0073] Where matrix U represents the reference feature library 340, matrix V represents the transformed feature library 460, and W V This represents the transformation matrix applied by transformation layer 442.

[0074] Finally, based on the correlation 430, the transformed reference features in the transformed feature library 460 can be combined into fused features 360. This process can be described by equation (5):

[0075] A = S * V (5)

[0076] Where A represents the fusion feature 360.

[0077] In this example, the parameters of transformation layer 442 (e.g., matrix W) V The parameters can be considered as being associated with the determination of the correlation. It is understood that the values ​​of the parameters of transform layers 410, 441, and 442 can be determined during the training of the sample processing model 121. Therefore, it can be understood that the parameters of transform layer 442 are associated with the determination of the correlation.

[0078] The attention mechanism described above is merely an example of combining reference features based on relevance. In the embodiments of this disclosure, any known or future-developed algorithm or model can be used to combine reference features based on relevance.

[0079] In this embodiment, the correlation between the individual features and the reference features is taken into account when combining reference features with the individual features of the samples. In this way, reference features that are more relevant to the individual features are given greater weight; that is, reference features similar to or close to the individual features are enhanced in the fused features (and thus in the final obtained second sample features). This further improves the expressive power of the obtained sample features. When applied to classification tasks (e.g., image classification tasks), the obtained sample features are better able to express the characteristics of the category to which they belong, thereby contributing to correct classification.

[0080] Example construction based on the sample library

[0081] The following is for reference. Figure 5 This describes an example process for constructing a reference sample library. The current version of the reference sample library 340 can be randomly initialized, or it can be generated based on one or more reference samples 351 in the reference sample set 350.

[0082] Assume the reference sample under consideration is reference sample 351-1. For reference sample 351-1, determine the target reference features associated with it from the current version of the reference feature library 340. The target reference features can be determined using any suitable method.

[0083] In some embodiments, the target reference feature may be a reference feature corresponding to the category to which reference feature 351-1 belongs. For example, the reference samples in the reference sample set 350 may be divided into multiple predetermined categories. Each reference feature 341 represents a sample of a predetermined category. Then, the reference feature representing the category to which reference feature 351-1 belongs can be determined as the target feature. It should be understood that in supervised training, each reference sample has a label, and its target reference feature can be found based on the label of the reference sample.

[0084] Alternatively or additionally, in some embodiments, the target reference feature can be a reference feature whose similarity to sample feature 510 of reference sample 351-1 exceeds a threshold similarity. For example, the similarity between sample feature 510 and each reference feature 341 of the current version can be calculated in the feature space. The reference feature 341 with the highest similarity can be identified as the target reference feature. In this case, the second highest similarity can be considered as the threshold similarity. Alternatively, sample feature 510 can be clustered with all reference features 341 of the current version. The target reference feature can be a reference feature belonging to the same cluster as sample feature 510. This approach can be used to find the target reference feature for unsupervised training.

[0085] Next, the target reference features can be updated based on the sample features 510 of the reference sample 351-1. The target reference features can be updated in any suitable manner. For example, the sample features 510 and the target reference features can be added element-wise to obtain the updated target reference features.

[0086] In some embodiments, the weighted average of the sample features 510 of the reference sample 351-1 and the target reference features can be used as the updated target reference features. For example, a moving weighted average can be used for updating. Let the sample features 510 of the reference sample 351-1 be denoted as I′, and the reference sample 351-1 belong to category y, U y This represents the reference feature for category y (the target reference feature in this example). The target reference feature U can be defined based on the following formula:y Updated to U y ′:

[0087] U y ′=α*U y +(1-α)I′, (6)

[0088] Where α∈[0,1] is a hyperparameter, for example α=0.99.

[0089] Therefore, the reference feature library 340 is updated, resulting in an updated version of the reference feature library 340. The same process can be performed on other reference samples to update the reference feature library 340 until all reference samples have been considered. For example, the reference feature library 340 can be updated next using sample feature 520 of reference sample 351-2. In this way, dynamic updates of the reference features can be achieved, improving the quality of the obtained reference features.

[0090] It should be understood that the process described above can be implemented during the training of the sample processing model 121. In this case, sample features 510 and 520, etc., can be extracted by the sample feature extraction unit (e.g., ...) in the sample processing model 121. Figure 3 The sample feature extraction unit 320 shown in the figure generates the sample feature extraction unit. In addition, in this case, the parameters of the attention mechanism described above (e.g., transformation layers 410, 441 and 442) are also determined through training.

[0091] In some embodiments, a reference feature library may be generated by a terminal device or server independent of the computing device 110. The terminal device or server used to generate the reference feature library may send the reference feature library to the computing device 110 for use after generation. In some embodiments, the computing device 110 may also send a reference feature library acquisition request to the terminal device or server used to generate the reference feature library to obtain the corresponding reference feature library.

[0092] In the embodiments of this disclosure, the sample features of the input sample are enhanced based on the reference features of samples of the same type as the input sample in the reference feature library, thereby improving the representational ability of the sample features.

[0093] Example devices and equipment

[0094] Figure 6 A schematic structural block diagram of a feature extraction apparatus 600 according to certain embodiments of the present disclosure is shown. Apparatus 600 may be implemented as or included in computing device 120. Various modules / components in apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0095] As shown in the figure, the device 600 includes a fusion feature generation module 610, configured to generate fusion features for the input sample based on a first sample feature of the input sample and a reference feature library representing a reference sample set. The input sample and the reference samples in the reference sample set are samples of the same type. The device 600 also includes a sample feature update module 620, configured to update the first sample feature to a second sample feature of the input sample based on the fusion feature. The device 600 further includes a processing result generation module 630, configured to determine a processing result associated with the input sample based on the second sample feature.

[0096] In some embodiments, the fusion feature generation module 610 further includes: a correlation determination module configured to determine the correlation between the first sample feature and each reference feature in the reference feature library; and a fusion feature determination module configured to determine the fusion feature based on the correlation and at least one reference feature in the reference feature library.

[0097] In some embodiments, the fusion feature determination module is further configured to: transform each reference feature according to parameters associated with the determination of correlation; and combine the transformed reference features into a fusion feature based on the correlation.

[0098] In some embodiments, the input samples in the fusion feature generation module 610 include an input image and the reference samples include a reference image.

[0099] In some embodiments, the processing result in the processing result generation module 630 includes classifying the input sample into one of a plurality of predetermined categories, and the reference feature library includes reference features corresponding to the plurality of predetermined categories respectively.

[0100] In some embodiments, the apparatus 600 further includes a reference feature library determination module configured to determine a reference feature library by iteratively performing the following operations: for a given reference sample in the reference sample set, determining a target reference feature associated with the given reference sample from the current version of the reference feature library; and generating an updated version of the reference feature library by updating the target reference feature based on the sample features of the given reference sample.

[0101] In some embodiments, the target reference features in the reference feature library determination module include at least one of the following: reference features corresponding to the category to which the given reference sample belongs, and reference features whose similarity to the sample features of the given reference sample exceeds a threshold similarity.

[0102] In some embodiments, the target reference features in the reference feature library determination module are updated based on a weighted average of the sample features of the second reference sample and the target reference features.

[0103] In some embodiments, the sample feature update module 620 is further configured to perform element-wise addition on the quantized representation of the first sample feature and the quantized representation of the fused feature to obtain the quantized representation of the second sample feature.

[0104] Figure 7 A block diagram is shown illustrating a computing device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that... Figure 7 The computing device 700 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 7 The computing device 700 shown can be used to implement Figure 1 120 computing devices.

[0105] like Figure 7 As shown, computing device 700 is in the form of a general-purpose computing device. Components of computing device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage devices 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processing unit 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 700.

[0106] Computing device 700 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to computing device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within computing device 700.

[0107] The computing device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 7As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0108] The communication unit 740 enables communication with other computing devices via a communication medium. Additionally, the components of the computing device 700 can function as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device 700 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or another network node.

[0109] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Computing device 700 can also communicate as needed with one or more external devices (not shown) via communication unit 740. These external devices, such as storage devices, display devices, etc., can communicate with one or more devices that enable user interaction with computing device 700, or with any device (e.g., network card, modem, etc.) that enables computing device 700 to communicate with one or more other computing devices. Such communication can be performed via input / output (I / O) interface (not shown).

[0110] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0111] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0112] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0113] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0114] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0115] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for extracting features, comprising: Based on the first sample features of the input sample and the reference feature library representing the reference sample set, a fusion feature is generated for the input sample, wherein the input sample and the reference samples in the reference sample set are samples of the same type. Based on the fusion features, the first sample features are updated to the second sample features of the input sample; and Based on the second sample features, determine the processing result associated with the input sample; The reference feature library is determined iteratively by performing the following operations: For a given reference sample in the reference sample set, determine the target reference feature associated with the given reference sample from the current version of the reference feature library; as well as An updated version of the reference feature library is generated by updating the target reference features based on the sample features of the given reference sample.

2. The method according to claim 1, wherein generating the fusion feature comprises: Determine the correlation between the first sample feature and each reference feature in the reference feature library; as well as The fusion feature is determined based on the correlation and at least one reference feature in the reference feature library.

3. The method of claim 2, wherein determining the fusion feature comprises: The reference features are transformed according to the parameters associated with the determination of the correlation. as well as Based on the correlation, the transformed reference features are combined into the fused feature.

4. The method of claim 1, wherein the input sample comprises an input image and the reference sample comprises a reference image.

5. The method of claim 4, wherein the processing result includes classifying the input sample into one of a plurality of predetermined categories, and the reference feature library includes reference features corresponding to the plurality of predetermined categories respectively.

6. The method of claim 1, wherein the target reference feature comprises at least one of the following: Reference features corresponding to the category to which the given reference sample belongs. Reference features whose similarity to the sample features of the given reference sample exceeds a threshold similarity.

7. The method of claim 1, wherein the target reference feature is updated based on a weighted average of the sample features of the given reference sample and the target reference feature.

8. The method of claim 1, wherein updating the first sample feature to the second sample feature comprises: The quantized representation of the first sample feature and the quantized representation of the fused feature are added element-wise to form the quantized representation of the second sample feature.

9. An apparatus for extracting features, comprising: The fusion feature generation module is configured to generate fusion features for the input sample based on the first sample features of the input sample and the reference feature library representing the reference sample set, wherein the input sample and the reference samples in the reference sample set are samples of the same type. The sample feature update module is configured to update the first sample feature to the second sample feature of the input sample based on the fused feature; as well as The processing result generation module is configured to determine the processing result associated with the input sample based on the second sample features; The device further includes: The reference feature library determination module is configured to determine the feature by iteratively performing the following operations: For a given reference sample in the reference sample set, determine the target reference feature associated with the given reference sample from the current version of the reference feature library; as well as An updated version of the reference feature library is generated by updating the target reference features based on the sample features of the given reference sample.

10. The apparatus of claim 9, further comprising: The correlation determination module is configured to determine the correlation between the first sample feature and each reference feature in the reference feature library; as well as The fusion feature determination module is configured to determine the fusion feature based on the correlation and at least one reference feature in the reference feature library.

11. The apparatus of claim 10, wherein the fusion feature determination module is further configured to: Transform each reference feature according to the parameters associated with the determination of correlation; and Based on correlation, the transformed reference features are combined into a fused feature.

12. The apparatus of claim 9, wherein the input sample comprises an input image and the reference sample comprises a reference image.

13. The apparatus of claim 12, wherein the processing result includes classifying the input sample into one of a plurality of predetermined categories, and the reference feature library includes reference features corresponding to the plurality of predetermined categories respectively.

14. The apparatus of claim 9, wherein the target reference feature in the reference feature library determination module comprises at least one of the following: Reference features corresponding to the category to which the given reference sample belongs. Reference features whose similarity to the sample features of the given reference sample exceeds a threshold similarity.

15. The apparatus of claim 9, wherein the target reference feature is updated based on a weighted average of the sample features of the given reference sample and the target reference feature.

16. The apparatus of claim 9, wherein the sample feature update module is further configured to: The quantized representation of the first sample feature and the quantized representation of the fused feature are added element-wise to form the quantized representation of the second sample feature.

17. An electronic device comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 8.

18. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 8.

Images