Feature determination method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a feature determination method and device, equipment, a storage medium and a program product. The method includes determining, using a visual encoder, a visual feature representation of an image, the visual feature representation having a first dimension; dividing the visual feature representation into a plurality of sub-feature representations according to dimensions; for each sub-feature representation in the plurality of sub-feature representations, quantitative feature representations matched with the sub-feature representations are determined from codebooks corresponding to the sub-feature representations in the plurality of codebooks, and each codebook comprises a plurality of quantitative feature representations; and determining a quantized visual feature representation of the image by splicing the plurality of quantized feature representations corresponding to the plurality of sub-feature representations, the quantized visual feature representation having the first dimension.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computer technology, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for feature determination. Background Art

[0002] In recent years, with the rapid growth of multimodal language models, autoregressive modeling has extended its advantages from the language field to the visual field. For visual understanding, multimodal language models have shown superior performance in tasks such as image captioning and visual question answering. In the field of visual generation, autoregressive methods have also shown scalability and tend to catch up with diffusion models in terms of generation quality. How to unify visual understanding and visual generation within the framework of a single multimodal language model has become a matter of concern. Summary of the invention

[0003] In a first aspect of the present disclosure, a feature determination method is provided. The method includes: using a visual encoder to determine a visual feature representation of an image, the visual feature representation having a first dimension; dividing the visual feature representation into multiple sub-feature representations according to the dimension; for each sub-feature representation in the multiple sub-feature representations, from a codebook corresponding to the sub-feature representation in multiple codebooks, determining a quantized feature representation matching the sub-feature representation, each codebook including multiple quantized feature representations; and determining a quantized visual feature representation of the image by splicing multiple quantized feature representations corresponding to the multiple sub-feature representations, the quantized visual feature representation having a first dimension.

[0004] In a second aspect of the present disclosure, a device for feature determination is provided. The device includes: a visual feature representation determination module, configured to determine a visual feature representation of an image using a visual encoder, wherein the visual feature representation has a first dimension; a sub-feature representation division module, configured to divide the visual feature representation into multiple sub-feature representations according to the dimension; a quantitative feature representation determination module, configured to determine, for each of the multiple sub-feature representations, a quantitative feature representation matching the sub-feature representation from a codebook corresponding to the sub-feature representation in multiple codebooks, wherein each codebook includes multiple quantitative feature representations; and a quantitative visual feature representation determination module, configured to determine a quantitative visual feature representation of the image by splicing multiple quantitative feature representations corresponding to the multiple sub-feature representations, wherein the quantitative visual feature representation has a first dimension.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the electronic device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0008] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 2 A determination process for determining a quantitative visual feature representation according to some embodiments of the present disclosure is shown;

[0012] Figure 3A An architectural diagram of a first multi-head attention module according to some embodiments of the present disclosure is shown;

[0013] Figure 3B shows an architecture diagram of a second multi-head attention module according to some embodiments of the present disclosure;

[0014] Figure 4 The present invention shows a training process of a visual encoder, a visual decoder and multiple codebooks according to some embodiments of the present disclosure;

[0015] Figure 5 A schematic diagram showing a performance for visual question answering according to some embodiments of the present disclosure;

[0016] Figure 6 A flow chart showing a method for determining features according to some embodiments of the present disclosure is shown;

[0017] Figure 7 An apparatus for feature determination according to some embodiments of the present disclosure is shown; and

[0018] Figure 8 A block diagram of an electronic device is shown in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0021] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0022] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0023] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that execute operations of the technical solution of the present disclosure based on the prompt message.

[0024] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0025] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0026] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0027] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes input from the previous layer.

[0028] Generally, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also called the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association between input and output (also called input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. The testing stage can sometimes be integrated into the training stage. In the application or inference stage, the trained model can be used to process the actual model input based on the parameter values ​​obtained through training to determine the corresponding model output.

[0029] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In environment 100, electronic device 110 applies visual encoder model 105 to perform feature extraction of visual data. Visual encoder model 105 is configured to generate quantized visual feature representation 114 based on image 112. In some examples, quantized visual feature representation 114 is a discrete feature representation.

[0030] In some embodiments, the visual encoder model 105 may compress the image 112 into a quantized visual feature representation 114 in a low-dimensional implicit space, thereby compressing the image 112 and reducing the amount of data of the image 112 .

[0031] In some embodiments, a reconstructed image for image 112 may be generated from quantized visual feature representation 114 using visual decoder model 106 .

[0032] It should be noted that the input of the visual decoder model 106 is not limited to the quantized visual feature representation 114 output by the visual encoder model 105, and the visual decoder model 106 can generate images based on any feature representation.

[0033] In environment 100, electronic device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including mobile phone, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, media computer, multimedia tablet, personal communication system (PCS) device, personal navigation device, personal digital assistant (PDA), audio / video player, digital camera / camcorder, positioning device, television receiver, radio broadcast receiver, e-book device, game device or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, electronic device 110 can also support any type of interface for user (such as "wearable" circuit, etc.). Feature determination model 105, for example, can be implemented in various types of computing systems / servers that can provide computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like.

[0034] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0035] As mentioned above, the development of multimodal language models has sparked interest in unifying visual generation and visual understanding within the framework of a single multimodal language model. One related work adopts the contrastive language-image pre-training (CLIP) model as a visual tokenizer, which has been shown to be beneficial for visual understanding tasks. However, due to the continuous nature of CLIP tokens, it is challenging to incorporate visual generation into an autoregressive framework. Therefore, these methods usually rely on external diffusion models to synthesize images. To address this issue, another study chose the vector quantized variational autoencoder (VQVAE) tokenizer, which converts images into discrete codes, similar to the language tokenization process. This enables unified modeling of visual and language sequences with the same next token prediction loss. However, compared with multimodal language models for understanding, these methods show poor visual understanding capabilities because the vector quantized (VQ) tokens are not naturally aligned with the language feature space.

[0036] In the field of visual generation, image tokenization plays an important role in encoding raw pixels into compact implicit features for generative modeling. Among various tokenizers, vector quantization tokenizers are more widely used due to their discrete implicit space and compatibility with autoregressive and mask generation models. Some related works propose to discretize feature vectors by mapping continuous tokens to the nearest neighbors in a learnable codebook.

[0037] In the field of visual understanding, the success of language models has catalyzed the development of multimodal language models, which have demonstrated outstanding capabilities in visual-language tasks that require high-level understanding and reasoning. As a key component of multimodal language models, choosing an effective visual word segmenter has been a topic of extensive research. A common choice for a visual word segmenter is a pre-trained CLIP model, which is aligned to the language during the pre-training stage. Alternatively, a self-supervised learning model can be used as a visual word segmenter. However, these word segmenters mainly encode images into continuous tokens, which poses a challenge to the unified modeling of visual tokens and text tokens. To address these challenges, some related works have explored discretizing CLIP tokens or adopting VQVAE encoders for word segmentation in multimodal language models. However, these schemes may harm the performance of visual understanding tasks.

[0038] In order to solve the problem that the above-mentioned word segmenter cannot extract feature representation well, a feature determination scheme is proposed in the embodiment of the present disclosure. Specifically, a visual feature representation of an image is determined by using a visual encoder, and the visual feature representation has a first dimension; the visual feature representation is divided into multiple sub-feature representations according to the dimension; for each sub-feature representation in the multiple sub-feature representations, a codebook corresponding to the sub-feature representation in multiple codebooks is determined to match the sub-feature representation, and each codebook includes multiple quantitative feature representations; and by splicing multiple quantitative feature representations corresponding to the multiple sub-feature representations, a quantitative visual feature representation of the image is determined, and the quantitative visual feature representation has a first dimension.

[0039] According to the scheme disclosed in the present invention, each of the multiple codebooks can be used to determine the quantized feature representations corresponding to the sub-feature representations of the image, and the optimization problems associated with large codebooks are avoided when the size of a single codebook remains unchanged. In this way, the dimension of the quantized visual feature representation of the image is scaled proportionally to the number of codebooks, thereby improving the representation ability of the quantized visual feature representation.

[0040] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0041] Figure 2 FIG. 2 shows a determination process 200 for determining a quantitative visual feature representation 114 according to some embodiments of the present disclosure. Figure 2 As shown, using visual encoder 210, visual feature representation 215 of image 112 can be determined. Visual feature representation 215 can have a first dimension (e.g., 64 dimensions). In some examples, visual feature representation 215 can be represented by continuous word units.

[0042] After the visual feature representation 215 is determined, the visual feature representation 215 may be divided into a plurality of sub-feature representations 215 - 1 to 215 - n according to dimensions.

[0043] In some embodiments, the visual feature representation 215 may be evenly divided in dimension to obtain a plurality of sub-feature representations 215-1 to 215-n. (also called visual feature representation 215) can be evenly divided into n-dimensional blocks {f1,f2,…,f n},in For example, the dimension corresponding to the visual feature representation 215 is 64 dimensions, and the visual feature representation 215 can be divided into four 16-dimensional sub-feature representations.

[0044] Alternatively, or additionally, the visual feature representation 215 may be unevenly divided in dimension to obtain a plurality of sub-feature representations 215-1 to 215-n. For example, the 64-dimensional visual feature representation 215 is divided into four sub-feature representations with dimensions of 16, 18, 14, and 16, respectively. The specific dimensional division method may depend on the specific configuration requirements, and the embodiments of the present disclosure are not limited thereto.

[0045] After the division obtains a plurality of sub-feature representations 215-1 to 215-n, for each of the plurality of sub-feature representations, a codebook corresponding to the sub-feature representation in a plurality of codebooks 220-1 to 220-n (collectively referred to as a plurality of codebooks 220 for ease of description) is used to determine a quantized feature representation that matches the sub-feature representation. Each codebook may include a plurality of quantized feature representations. The quantized feature representations in the plurality of codebooks 220 are learnable and can be determined during the training process of the visual encoder 210. For example, in order to determine a quantized feature representation that matches the sub-feature representation 215-1, first, a quantized feature representation 225-1 that matches the sub-feature representation 215-1 can be determined from a codebook (e.g., codebook 220-1) corresponding to the sub-feature representation 215-1 in the plurality of codebooks. In another example, in order to determine the quantized feature representation that matches the sub-feature representation 215-2, a codebook (e.g., codebook 220-2) corresponding to the sub-feature representation 215-2 may be selected from multiple codebooks to determine the quantized feature representation 225-2 that matches the sub-feature representation 215-2. This process is repeated in this way until the quantized feature representation 225-n corresponding to the sub-feature representation 215-n is determined.

[0046] In some embodiments, each codebook in the plurality of codebooks 220 corresponds to a corresponding dimension interval divided from the first dimension. For example, the first dimension is 64 dimensions, there are 4 codebooks in total, codebook 220-1 corresponds to dimensions 1 to 16 in the first dimension, codebook 220-2 corresponds to dimensions 17 to 32 in the first dimension, and so on. In some embodiments, according to the dimension interval of the sub-feature representation, a codebook corresponding to the sub-feature representation can be determined, and a quantized feature representation matching the sub-feature representation can be determined from the corresponding codebook. In one example, the dimension interval of the sub-feature representation 215-1 is the 1st to 16th dimensions, and the codebook 220-1 corresponding to the sub-feature representation 215-1 can be determined. Then, a quantized feature representation 225-1 matching the sub-feature representation 215-1 can be determined from the codebook 220-1.

[0047] After determining the quantitative feature representations 225-1 to 225-n corresponding to the multiple sub-feature representations 215-1 to 215-n, the multiple quantitative feature representations can be spliced ​​to determine the quantitative visual feature representation 225 of the image 205. The quantitative visual feature representation 225 has a first dimension. In some examples, the quantitative visual feature representation 225 can be represented by discrete word units. Splicing multiple quantitative feature representations can be as follows:

[0048]

[0049] in represents a discrete implicit vector (also called quantized visual feature representation 225), Q represents a code index query operation, and Z i represents the i-th codebook, Q(Z i ,f i ) represents the i-th quantitative feature representation.

[0050] In some embodiments, multiple quantized feature representations 225-1 to 225-n may be concatenated according to corresponding dimension intervals to obtain a quantized visual feature representation 225. In one example, the dimension intervals corresponding to the quantized feature representations 225-1 to 225-4 are the 1st to 16th dimensions, the 17th to 32nd dimensions, the 33rd to 48th dimensions, and the 49th to 64th dimensions, respectively. The quantized feature representations 225-1 to 225-4 are concatenated according to the corresponding dimension intervals to obtain a 64-dimensional quantized visual feature representation 225. In this way, the dimension of the quantized visual feature representation increases as the number of codebooks increases, thereby improving the representation capability of the quantized visual feature representation.

[0051] In some embodiments, the visual encoder 210 may be used to extract an intermediate visual feature representation 220 of the image 112. The dimension corresponding to the intermediate visual feature representation 212 is the second dimension, and the second dimension (e.g., 768 dimensions) is greater than the first dimension (e.g., 64 dimensions). Then, the first multi-head attention module 214 (also referred to as a dimension compression module) may be used to perform a dimensionality reduction operation on the intermediate visual feature representation 212 to obtain a visual feature representation 215 having a first dimension.

[0052] The following will refer to Figure 3A A dimensionality reduction operation is described that is performed on the intermediate visual feature representation 212 . Figure 3A FIG. 2 shows an architecture diagram of the first multi-head attention module 214 according to some embodiments of the present disclosure. Figure 3AAs shown, the dimension of the intermediate visual feature representation 212 input to the first multi-head attention module 214 is N×C (as an example of the second dimension), and the dimension of the feature representation output by the linear layer 305 is N×h×c, where C=h×c. The linear layer 310 and the average pooling layer 315 can compress the dimension of the intermediate visual feature representation from the second dimension to the first dimension (e.g., N×c) to obtain the visual feature representation 215. In this way, the relative density of the visual feature representation can be increased by compressing the intermediate visual feature representation, thereby reducing the quantization error. The compressed visual feature representation 215 having the first dimension can continue to be quantized through multiple codebooks 220 to obtain a quantized visual feature representation 225.

[0053] The quantized feature representation 225 can represent the visual feature information of the image in a smaller dimension. In some embodiments, the visual decoder 235 is used to generate a target image based on the quantized visual feature representation 225. In some examples, a dimensionality increase operation can be performed on the quantized visual feature representation 225 to obtain a target quantized feature representation 230, and then the visual decoder 235 can decode the target quantized feature representation 230 to obtain the target image.

[0054] In some embodiments, a language model (not shown) may be used to generate a visual understanding of the image 112 based on the quantized visual feature representation 225. For example, the language model may generate a descriptive text “flowers and grass” for the image 112 based on the quantized visual feature representation 225.

[0055] In some embodiments, the quantized visual feature representation 225 may be continued using a language model, and then the visual decoder 235 may generate an image related to the image 112 from the continued quantized visual feature representation 225 .

[0056] Continue to refer Figure 2, the visual decoder 235 can be used to generate a reconstructed image from the quantized feature representation. In some embodiments, an intermediate quantized feature representation for image generation can be obtained, and the intermediate quantized feature representation has a first dimension. In some examples, the intermediate quantized feature representation can be a quantized visual feature representation 225 of the image 205. Alternatively or additionally, the intermediate quantized feature representation can be a quantized feature representation generated by a content generation model (e.g., a language model). In the following, the intermediate quantized feature representation input to the visual decoder 235 is taken as an example to illustrate that the quantized visual feature representation 225 of the image 112 is input. Before being input to the visual decoder 235, the second multi-head attention module 226 (also called a dimension expansion module) is first used to perform a dimensionality increase operation on the intermediate quantized feature representation (i.e., the quantized visual feature representation 225) having a first dimension to obtain a target quantized feature representation 230 having a second dimension. Using the visual decoder 235, a target image can be generated based on the target quantized feature representation 230.

[0057] The following will refer to Figure 3B Describes the dimensionality increase operation performed on the intermediate quantized feature representation. Figure 3B FIG. 2 shows an architecture diagram of the second multi-head attention module 226 according to some embodiments of the present disclosure. Figure 3B As shown, the dimension of the intermediate quantized feature representation input to the second multi-head attention module 226 is N×c (as an example of the first dimension), and the linear layer 354 and the linear layer 356 expand the dimension of the intermediate quantized feature representation from the first dimension to the second dimension (e.g., N×C) to obtain the target quantized feature representation 230. According to an embodiment of the present disclosure, the multi-head attention mechanism is used to reduce the dimension of the feature representation and then increase the dimension, which can effectively improve the representation ability of the target quantized feature representation.

[0058] The following will refer to Figure 4 The training process of the visual encoder 210, the visual decoder 235, and the plurality of codebooks 220 is described. Figure 4 FIG. 4 shows a training process 400 of the visual encoder 210, the visual decoder 235, and the plurality of codebooks 220 according to some embodiments of the present disclosure. Figure 4 As shown, using the visual encoder 210 being trained, a sample visual feature representation 410 of the sample image 405 can be determined. Based on the multiple codebooks being trained, a sample quantized visual feature representation 415 corresponding to the sample visual feature representation 410 can be determined.

[0059] In some embodiments, visual generation and visual understanding typically place different demands on a visual tokenizer (e.g., visual encoder 210). For example, visual generation emphasizes lossless compression for accurate reconstruction, while visual understanding prioritizes semantically meaningful and distinguishable features. Therefore, an image-text contrast loss can be used to enhance high-level semantic information in feature representations. In order to determine the contrast loss, a text encoder 420 can be used to determine a first text feature representation 422 of a positive sample text 418 and a second text feature representation of a negative sample text (not shown in the figure), respectively, wherein the positive sample text matches the sample image and the negative sample text does not match the sample image. For example, the sample image 405 is an oil painting of a rural landscape, the positive sample text 418 can be "an oil painting depicting a rural landscape", and the negative sample text can be "a sketch depicting a person". Then, the contrast loss can be determined based on the difference between the sample quantized visual feature representation 410 and the first text feature representation 422, and the difference between the sample quantized visual feature representation 410 and the second text feature representation. 425. The loss value of the contrast loss 425 is positively correlated with the difference between the sample quantized visual feature representation 410 and the first text feature representation 422, and is negatively correlated with the difference between the sample quantized visual feature representation 410 and the second text feature representation. Next, the visual encoder 210, the visual decoder 235, and the multiple codebooks 220 are updated based on the first training objective, and the first training objective is configured to reduce or minimize the contrast loss. In this way, the visual encoder 210, the visual decoder 235, and the multiple codebooks 220 are trained based on the reconstruction loss between the image and the text, and the quantized visual feature representation of the image can enhance the representation of high-level semantic information used for visual understanding.

[0060] In some embodiments, a reconstruction loss (e.g., a reconstruction loss based on a VQVAE) can preserve low-level information in the feature representation. To determine the reconstruction loss, a reconstructed image 430 corresponding to the sample image 405 can be generated based on the sample quantized visual feature representation 415 using the visual decoder 235. A reconstruction loss 435 is determined based on the difference between the sample image 405 and the reconstructed image 430. In some examples, the reconstruction loss 435 (given by ) can include pixel-level reconstruction loss (denoted by ), perceived loss (denoted by denoted by ), the discriminator loss for enhancing the reconstruction fidelity (denoted by denoted by), entropy loss used to encourage codebook utilization (denoted by ) and a vector quantization loss that minimizes the distance between the output of the visual encoder and its nearest code entry (denoted by denoted by ). The reconstruction loss can be expressed as follows:

[0061]

[0062] Where λ represents the weight factor for the corresponding loss term.

[0063] After the loss function is constructed, the visual encoder, the visual decoder, and the plurality of codebooks are also updated based on the second training objective, and the second training objective is configured to reduce or minimize the combination of the contrast loss and the reconstruction loss. The combination of the contrast loss and the reconstruction loss can be expressed as follows:

[0064]

[0065] in represents the contrast loss, λ contra represents the weight corresponding to the contrast loss. In one example, λ contra Can be set to 1. According to an embodiment of the present disclosure, the reconstruction loss can retain low-level information used for visual generation, and by combining the contrast loss and the reconstruction loss, the quantized visual feature representation of the image can simultaneously enhance the representation of high-level semantic information used for visual understanding and the representation of information used for visual generation, thereby unifying visual understanding and visual generation in a single multimodal language model. It should be noted that for the convenience of description, the determination of the quantitative feature representation using each of the multiple codebooks and the manipulation of the dimensionality increase and decrease of the feature representation are not repeated in process 400, and the relevant description can be found in process 200.

[0066] In some embodiments, the text encoder 420 can be jointly trained with the visual encoder and the visual decoder. That is, the text encoder 420 can also be jointly updated based on the first training target or the second training target. During the training process, the parameters of the text encoder 420 are updated, so that a more accurate text feature representation can be determined.

[0067] Figure 5 Schematic diagram 500 of a performance for visual question answering according to some embodiments of the present disclosure is shown. Figure 5 As shown in the figure, after experiments, histograms 502, 504 and 506 represent the accuracy of the quantized feature representation using the related technology in the visual question answering task. Histograms 510 and 520 represent the accuracy of the quantized feature representation using the embodiment of the present disclosure (i.e., multi-codebook and multi-head attention mechanism) in the visual question answering task. Figure 5It can be seen that according to some embodiments of the present disclosure, the quantitative visual features of the image can be extracted more accurately using a multi-codebook, and the reconstruction loss and contrast loss can be constructed based on the quantitative visual features. Based on the reconstruction loss and contrast loss, the text encoder, visual encoder and visual decoder can be jointly trained, and the accuracy of the quantitative feature representation extracted after training in the visual question answering task is significantly higher than the accuracy of the related technology. Furthermore, a multi-head attention mechanism can be added to perform dimensionality increase and dimensionality reduction operations on the feature representation extracted during the training process, so that the accuracy of the quantitative feature representation in the visual question answering task is further improved. According to the disclosed embodiments, a more accurate quantitative feature representation for visual understanding can be extracted using a multi-codebook and a multi-head attention mechanism, thereby achieving a higher accuracy.

[0068] Figure 6 FIG. 6 is a flowchart of a feature determination method 600 according to some embodiments of the present disclosure. The method 600 may be implemented in Figure 1 The electronic device 110 is referred to as Figure 1 The method 600 is described with reference to the environment 100 of FIG.

[0069] At block 610 , the electronic device 110 determines a visual feature representation of an image using a visual encoder, the visual feature representation having a first dimension.

[0070] At block 620 , the electronic device 110 divides the visual feature representation into a plurality of sub-feature representations by dimension.

[0071] At block 630 , the electronic device 110 determines, for each of the multiple sub-feature representations, a quantized feature representation that matches the sub-feature representation from a codebook corresponding to the sub-feature representation from a plurality of codebooks, each codebook including a plurality of quantized feature representations.

[0072] In block 640 , the electronic device 110 determines a quantized visual feature representation of the image by concatenating a plurality of quantized feature representations corresponding to the plurality of sub-feature representations, wherein the quantized visual feature representation has a first dimension.

[0073] In some embodiments, method 600 further includes: generating, using a visual decoder, a target image based on the quantized visual feature representation.

[0074] In some embodiments, the visual encoder, visual decoder and multiple codebooks are trained in the following manner: using the visual encoder being trained, determining a sample visual feature representation of a sample image; based on the multiple codebooks being trained, determining a sample quantized visual feature representation corresponding to the sample visual feature representation; using a text encoder, determining a first text feature representation of a positive sample text and a second text feature representation of a negative sample text, respectively, wherein the positive sample text matches the sample image and the negative sample text does not match the sample image; determining a contrast loss based on the difference between the sample quantized visual feature representation and the first text feature representation, and the difference between the sample quantized visual feature representation and the second text feature representation; and updating the visual encoder, visual decoder and multiple codebooks based on a first training objective, the first training objective being configured to reduce or minimize the contrast loss.

[0075] In some embodiments, updating the visual encoder, the visual decoder, and the plurality of codebooks further comprises: generating, using the visual decoder, a reconstructed image corresponding to the sample image based on the sample quantized visual feature representation; determining a reconstruction loss based on the difference between the sample image and the reconstructed image; and also updating the visual encoder, the visual decoder, and the plurality of codebooks based on a second training objective, the second training objective being configured to reduce or minimize a combination of contrast loss and reconstruction loss.

[0076] In some embodiments, the text encoder is jointly updated based on the first training objective or the second training objective.

[0077] In some embodiments, dividing the visual feature representation into a plurality of sub-feature representations according to a dimension includes: dividing the visual feature representation evenly in the dimension to obtain a plurality of sub-feature representations.

[0078] In some embodiments, each codebook in the plurality of codebooks corresponds to a corresponding dimensional interval divided from a first dimension, and wherein determining a quantized feature representation matching the sub-feature representation comprises: determining a codebook corresponding to the sub-feature representation according to the dimensional interval of the sub-feature representation; and determining a quantized feature representation matching the sub-feature representation from the corresponding codebook.

[0079] In some embodiments, determining the quantized visual feature representation includes: concatenating multiple quantized feature representations according to corresponding dimension intervals to obtain the quantized visual feature representation.

[0080] In some embodiments, determining the visual feature representation includes: using a visual encoder to extract an intermediate visual feature representation of the image, wherein the dimension corresponding to the intermediate visual feature representation is the second dimension, and the second dimension is greater than the first dimension; and using a first multi-head attention module to perform a dimensionality reduction operation on the intermediate visual feature representation to obtain a visual feature representation having a first dimension.

[0081] In some embodiments, method 600 also includes: obtaining an intermediate quantitative feature representation for image generation, the intermediate quantitative feature representation having a first dimension; using a second multi-head attention module to perform a dimensionality increase operation on the intermediate quantitative feature representation to obtain a target quantitative feature representation, the dimension corresponding to the target quantitative feature representation is a second dimension; and using a visual decoder to generate a target image based on the target quantitative feature representation.

[0082] In some embodiments, a dimensionality reduction operation is performed on the intermediate visual feature representation to obtain the visual feature representation, including: compressing the dimension of the intermediate visual feature representation from the second dimension to the first dimension through a first linear layer and an average pooling layer in a first multi-head attention module to obtain the visual feature representation.

[0083] In some embodiments, performing a dimensionality increase operation on the intermediate quantitative feature representation to obtain a target quantitative feature representation includes: expanding the dimension of the intermediate quantitative feature representation from a first dimension to a second dimension through a second linear layer and a third linear layer in a second multi-head attention module to obtain a target quantitative feature representation.

[0084] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 7 The apparatus 700 is shown as an apparatus for feature determination according to some embodiments of the present disclosure. The apparatus 700 may be implemented as or included in the electronic device 110. Each module / component in the apparatus 700 may be implemented by hardware, software, firmware or any combination thereof.

[0085] like Figure 7 As shown, the device 700 includes a visual feature representation determination module 710, which is configured to use a visual encoder to determine a visual feature representation of an image, wherein the visual feature representation has a first dimension; a sub-feature representation division module 720, which is configured to divide the visual feature representation into multiple sub-feature representations according to the dimension; a quantitative feature representation determination module 730, which is configured to determine, for each sub-feature representation in the multiple sub-feature representations, a quantitative feature representation matching the sub-feature representation from a codebook corresponding to the sub-feature representation in multiple codebooks, each codebook including multiple quantitative feature representations; and a quantitative visual feature representation determination module 740, which is configured to determine a quantitative visual feature representation of the image by splicing multiple quantitative feature representations corresponding to the multiple sub-feature representations, wherein the quantitative visual feature representation has a first dimension.

[0086] In some embodiments, the apparatus 700 further includes a target image generation module configured to generate a target image based on the quantized visual feature representation using a visual decoder.

[0087] In some embodiments, the device 700 also includes a training module, which is configured to determine a sample visual feature representation of a sample image using a visual encoder being trained; determine a sample quantized visual feature representation corresponding to the sample visual feature representation based on multiple codebooks being trained; determine a first text feature representation of a positive sample text and a second text feature representation of a negative sample text using a text encoder, respectively, wherein the positive sample text matches the sample image and the negative sample text does not match the sample image; determine a contrast loss based on the difference between the sample quantized visual feature representation and the first text feature representation, and the difference between the sample quantized visual feature representation and the second text feature representation; and update the visual encoder, the visual decoder and the multiple codebooks based on a first training objective, and the first training objective is configured to reduce or minimize the contrast loss.

[0088] In some embodiments, the training module is further configured to generate, using a visual decoder, a reconstructed image corresponding to a sample image based on a sample quantized visual feature representation; determine a reconstruction loss based on a difference between the sample image and the reconstructed image; and update the visual encoder, the visual decoder, and the plurality of codebooks based on a second training objective, wherein the second training objective is configured to reduce or minimize a combination of contrast loss and reconstruction loss.

[0089] In some embodiments, the text encoder is jointly updated based on the first training objective or the second training objective.

[0090] In some embodiments, the sub-feature representation division module 720 is further configured to evenly divide the visual feature representation in dimensions to obtain a plurality of sub-feature representations.

[0091] In some embodiments, each codebook in the plurality of codebooks corresponds to a corresponding dimension interval divided from the first dimension. The quantitative feature representation determination module 730 is further configured to determine a codebook corresponding to the sub-feature representation according to the dimension interval of the sub-feature representation; and determine a quantitative feature representation matching the sub-feature representation from the corresponding codebook.

[0092] In some embodiments, the quantitative visual feature representation determination module 740 is further configured to concatenate multiple quantitative feature representations according to corresponding dimension intervals to obtain a quantitative visual feature representation.

[0093] In some embodiments, the visual feature representation determination module 710 is further configured to utilize a visual encoder to extract an intermediate visual feature representation of the image, wherein the dimension corresponding to the intermediate visual feature representation is a second dimension, and the second dimension is greater than the first dimension; and to utilize a first multi-head attention module to perform a dimensionality reduction operation on the intermediate visual feature representation to obtain a visual feature representation having a first dimension.

[0094] In some embodiments, the device 700 also includes a dimension upgrading module, which is configured to obtain an intermediate quantitative feature representation for image generation, wherein the intermediate quantitative feature representation has a first dimension; using a second multi-head attention module, performing a dimension upgrading operation on the intermediate quantitative feature representation to obtain a target quantitative feature representation, wherein the dimension corresponding to the target quantitative feature representation is a second dimension; and using a visual decoder, generating a target image based on the target quantitative feature representation.

[0095] In some embodiments, the visual feature representation determination module 710 is further configured to compress the dimension of the intermediate visual feature representation from the second dimension to the first dimension through a first linear layer and an average pooling layer in a first multi-head attention module to obtain a visual feature representation.

[0096] In some embodiments, the dimension increasing module is further configured to expand the dimension of the intermediate quantitative feature representation from the first dimension to the second dimension through a second linear layer and a third linear layer in a second multi-head attention module to obtain a target quantitative feature representation.

[0097] The units and / or modules included in the device 700 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units and / or modules in the device 700 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0098] It should be understood that one or more steps in the above method can be performed by a suitable electronic device or a combination of electronic devices. Such an electronic device or a combination of electronic devices may include, for example, Figure 1 The electronic device 110 in.

[0099] Figure 8 8 is a block diagram of an electronic device 800 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 8 The electronic device 800 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 8 The electronic device 800 shown can be used to implement Figure 1 Electronic device 110 or Figure 7 Device 700.

[0100] like Figure 8 As shown, the electronic device 800 is in the form of a general electronic device. The components of the electronic device 800 may include, but are not limited to, one or more processors or processing units 810, a memory 820, a storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. The processing unit 810 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 820. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 800.

[0101] The electronic device 800 typically includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by the electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 820 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 830 can be a removable or non-removable medium, and can include a machine-readable medium, such as a flash drive, a disk, or any other medium, which can be used to store information and / or data and can be accessed within the electronic device 800.

[0102] The electronic device 800 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 8 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 820 may include a computer program product 825 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0103] The communication unit 840 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 800 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 800 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0104] The input device 850 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 860 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 800 may also communicate with one or more external devices (not shown) through the communication unit 840 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 800, or communicate with any device that allows the electronic device 800 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0105] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0106] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0107] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0108] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0109] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as an update, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the function involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0110] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the marketplace, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A feature determination method, comprising: Determining, using a visual encoder, a visual feature representation of the image, the visual feature representation having a first dimension; Dividing the visual feature representation into a plurality of sub-feature representations according to the dimension; For each sub-feature representation of the plurality of sub-feature representations, Determine a quantized feature representation matching the sub-feature representation from a codebook corresponding to the sub-feature representation among a plurality of codebooks, each codebook including a plurality of quantized feature representations; as well as The quantized visual feature representation of the image is determined by concatenating the multiple quantized feature representations respectively corresponding to the multiple sub-feature representations, wherein the quantized visual feature representation has the first dimension.

2. The method according to claim 1, further comprising: A target image is generated based on the quantized visual feature representation using a visual decoder.

3. The method according to claim 2, wherein the visual encoder, the visual decoder and the plurality of codebooks are trained by: Determining a sample visual feature representation of a sample image using the visual encoder being trained; Determining, based on the plurality of codebooks being trained, a sample quantized visual feature representation corresponding to the sample visual feature representation; Determine, by using a text encoder, a first text feature representation of a positive sample text and a second text feature representation of a negative sample text, respectively, wherein the positive sample text matches the sample image, and the negative sample text does not match the sample image; Determining a contrast loss based on a difference between the sample quantized visual feature representation and the first text feature representation, and a difference between the sample quantized visual feature representation and the second text feature representation; as well as The visual encoder, the visual decoder, and the plurality of codebooks are updated based on a first training objective, wherein the first training objective is configured to reduce or minimize the contrastive loss.

4. The method of claim 3, wherein updating the visual encoder, the visual decoder, and the plurality of codebooks further comprises: generating, using the visual decoder, a reconstructed image corresponding to the sample image based on the sample quantized visual feature representation; determining a reconstruction loss based on a difference between the sample image and the reconstructed image; as well as The visual encoder, the visual decoder, and the plurality of codebooks are also updated based on a second training objective configured to reduce or minimize a combination of the contrastive loss and the reconstruction loss.

5. The method according to claim 3 or 4, wherein the text encoder is jointly updated based on the first training objective or the second training objective.

6. The method according to claim 1, wherein dividing the visual feature representation into the plurality of sub-feature representations by dimension comprises: The visual feature representation is evenly divided in dimensions to obtain the multiple sub-feature representations.

7. The method of claim 1, wherein each codebook in the plurality of codebooks corresponds to a corresponding dimension interval divided from the first dimension, and wherein determining the quantized feature representation matching the sub-feature representation comprises: Determining a codebook corresponding to the sub-feature representation according to the dimensional interval of the sub-feature representation; as well as The quantized feature representation matching the sub-feature representation is determined from the corresponding codebook.

8. The method of claim 7, wherein determining the quantitative visual feature representation comprises: The multiple quantitative feature representations are concatenated according to corresponding dimension intervals to obtain the quantitative visual feature representation.

9. The method of claim 1 , wherein determining the visual feature representation comprises: Extracting an intermediate visual feature representation of the image using the visual encoder, wherein the dimension corresponding to the intermediate visual feature representation is a second dimension, and the second dimension is greater than the first dimension; and Using a first multi-head attention module, a dimensionality reduction operation is performed on the intermediate visual feature representation to obtain the visual feature representation having the first dimension.

10. The method according to claim 9, further comprising: Obtaining an intermediate quantitative feature representation for image generation, wherein the intermediate quantitative feature representation has the first dimension; Using a second multi-head attention module, performing a dimensionality increase operation on the intermediate quantitative feature representation to obtain a target quantitative feature representation, wherein the dimension corresponding to the target quantitative feature representation is the second dimension; and A target image is generated based on the target quantitative feature representation using a visual decoder.

11. The method according to claim 9, wherein performing a dimensionality reduction operation on the intermediate visual feature representation to obtain the visual feature representation comprises: Through the first linear layer and the average pooling layer in the first multi-head attention module, the dimension of the intermediate visual feature representation is compressed from the second dimension to the first dimension to obtain the visual feature representation.

12. The method according to claim 10, wherein performing a dimensionality increase operation on the intermediate quantitative feature representation to obtain the target quantitative feature representation comprises: By means of a second linear layer and a third linear layer in the second multi-head attention module, the dimension of the intermediate quantitative feature representation is expanded from the first dimension to the second dimension to obtain the target quantitative feature representation.

13. An apparatus for feature determination, comprising: a visual feature representation determination module configured to determine a visual feature representation of an image using a visual encoder, wherein the visual feature representation has a first dimension; A sub-feature representation division module, configured to divide the visual feature representation into a plurality of sub-feature representations according to a dimension; a quantized feature representation determination module configured to determine, for each of the multiple sub-feature representations, a quantized feature representation matching the sub-feature representation from a codebook corresponding to the sub-feature representation in a plurality of codebooks, each codebook including a plurality of quantized feature representations; as well as The quantitative visual feature representation determination module is configured to determine the quantitative visual feature representation of the image by splicing multiple quantitative feature representations corresponding to the multiple sub-feature representations respectively, and the quantitative visual feature representation has the first dimension.

14. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to perform the method according to any one of claims 1 to 12 when executed by the at least one processing unit.

15. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 12 when executed by a processor.

16. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images