Training method of visual encoder, visual encoding method and device

Abstract

The present disclosure provides a training method of a visual encoder, a visual encoding method and device, and relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, deep learning, large model and the like. The specific implementation scheme is as follows: dividing a sample image into a plurality of image blocks, the sample image including an image of any size; determining a fusion feature of each image block according to the plurality of image blocks and position information of each image block in the sample image; inputting the fusion feature of each image block into a conversion module of a visual encoder, determining attention parameters of the plurality of image blocks by the conversion module; and predicting an image feature of the sample image based on the attention parameters and the fusion feature of the plurality of image blocks by the visual encoder; and adjusting parameters of the visual encoder based on the image feature.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of computer vision, deep learning, and large models. Background Technology

[0002] In current large-scale model applications, visual tasks such as image captioning and visual question answering (VQA) are common tasks in multimodal large-scale models. These tasks typically require using a visual encoder to extract key visual features from an image to obtain the image's feature encoding. Improving the accuracy of image feature encoding is a key technical problem to be solved. Summary of the Invention

[0003] This disclosure provides a method for training a visual encoder, a visual encoding method, and an apparatus.

[0004] According to one aspect of this disclosure, a method for training a visual encoder is provided, comprising:

[0005] The sample image is divided into multiple image blocks, which include images of arbitrary size;

[0006] Based on multiple image patches and their location information in the sample image, the fusion features of each image patch are determined.

[0007] The fusion features of each image patch are input into the transformation module of the visual encoder, which determines the attention parameters of multiple image patches; and the visual encoder predicts the image features of the sample image based on the attention parameters and fusion features of multiple image patches.

[0008] Adjust the parameters of the visual encoder based on image features.

[0009] According to another aspect of this disclosure, a visual encoding method is provided, comprising:

[0010] The image to be encoded is divided into multiple image blocks, which include images of arbitrary size;

[0011] Based on multiple image blocks and the position information of each image block in the image to be encoded, the fusion features of each image block are determined;

[0012] The fusion features of each image patch are input into the conversion module of the visual encoder, which determines the attention parameters of multiple image patches. The visual encoder then predicts the image features of the image to be encoded based on the attention parameters and fusion features of the multiple image patches.

[0013] According to another aspect of this disclosure, a training apparatus for a visual encoder is provided, comprising:

[0014] The first partitioning module is used to divide the sample image into multiple image blocks, which include images of arbitrary size.

[0015] The first determining module is used to determine the fusion features of each image patch based on multiple image patches and the position information of each image patch in the sample image;

[0016] The second determining module is used to input the fusion features of each image patch into the conversion module of the visual encoder, the conversion module determines the attention parameters of multiple image patches, and the visual encoder predicts the image features of the sample image based on the attention parameters and fusion features of multiple image patches.

[0017] The parameter adjustment module is used to adjust the parameters of the visual encoder based on image features.

[0018] According to another aspect of this disclosure, a visual encoding device is provided, comprising:

[0019] The second partitioning module is used to divide the image to be encoded into multiple image blocks, which include images of arbitrary size.

[0020] The third determining module is used to determine the fusion features of each image block based on multiple image blocks and the position information of each image block in the image to be encoded.

[0021] The fourth determination module is used to input the fusion features of each image patch into the conversion module of the visual encoder. The conversion module determines the attention parameters of multiple image patches, and the visual encoder predicts the image features of the image to be encoded based on the attention parameters and fusion features of multiple image patches.

[0022] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0023] At least one processor; and

[0024] The memory is communicatively connected to the at least one processor; wherein,

[0025] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0026] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0027] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0028] This disclosure divides a sample image into multiple image blocks, and then determines the image features of the sample image based on these multiple image blocks and their positional information. The parameters of the visual encoder are then adjusted based on these image features. Because the sample image is divided during the training of the visual encoder, the image size is not limited during image recognition. The visual encoder can encode features from images of any size, avoiding image distortion and loss of detail caused by image scaling, thereby improving the accuracy of image feature encoding.

[0029] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0030] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0031] Figure 1 This is a flowchart illustrating the implementation of a training method for an image encoder according to an embodiment of the present disclosure.

[0032] Figure 2 This is a schematic diagram illustrating the division of an input image into multiple image blocks according to an embodiment of the present disclosure;

[0033] Figure 3 This is a schematic diagram of a visual encoder according to an embodiment of the present disclosure;

[0034] Figure 4 This is a schematic diagram of a partitioned window according to an embodiment of the present disclosure;

[0035] Figure 5 This is a flowchart illustrating the implementation of a visual encoding method according to an embodiment of the present disclosure;

[0036] Figure 6 This is a schematic diagram of the structure of a training device 600 for a visual encoder according to an embodiment of the present disclosure;

[0037] Figure 7 This is a schematic diagram of the structure of a video encoding apparatus 700 according to an embodiment of the present disclosure;

[0038] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0039] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0040] The term "and / or" in this disclosure indicates that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The term "at least one" in this document means any combination of at least two of a plurality of options, such as including at least one of A, B, and C, which can mean including any one or more elements selected from the set of A, B, and C. The terms "first" and "second" in this document refer to and distinguish multiple similar technical terms, and do not imply a specific order or a limitation to only two. For example, "first feature" and "second feature" refer to two types / two features; the first feature can be one or more, and the second feature can also be one or more.

[0041] The network architecture used in mainstream multimodal large-scale models typically consists of three parts: a visual encoder (ImageEncoder), a visual-language alignment layer (Adapter), and a language model. Generally, a fixed-size image is input to the visual encoder, which extracts feature information from the image to obtain a fixed-dimensional visual feature sequence. This visual feature sequence is then mapped by the language alignment layer (Adapter) to obtain visual embedding features. Similarly, text is input to the embedding model to obtain text embedding features. Finally, the visual and text embedding features are concatenated and fed together into the language model, which generates the output result through autoregression. This output result is text information generated based on the input text and image. For example, in an image description task, the input text includes instructions to describe the image, and the output text includes descriptive information about the input image generated based on the image and the input text.

[0042] Existing multimodal large-scale visual encoders rely on pre-trained visual models to obtain robust visual features; the input image for these encoders is a fixed-size image. Typically, the original image needs to be scaled to the required size for the visual encoder; this approach can easily cause image distortion or loss of detail, negatively impacting the final recognition performance of the visual encoder and leading to inaccurate image features extracted.

[0043] For example, when processing images of text lines, directly scaling the text line image to a fixed size will cause the text to become distorted. When the visual encoder performs image recognition and image feature extraction on the scaled image, it will affect the model's recognition performance.

[0044] To address the aforementioned issues, this disclosure proposes a training method for a visual encoder. The visual encoder trained using this method can extract image features from images of arbitrary sizes, enabling multi-model large-scale models to implement applications such as image description and visual question answering.

[0045] Figure 1 This is a flowchart illustrating the implementation of a training method for an image encoder according to an embodiment of the present disclosure, including:

[0046] S110. Divide the sample image into multiple image blocks, where the sample image includes images of arbitrary size;

[0047] S120. Determine the fusion features of each image patch based on multiple image patches and their position information in the sample image.

[0048] S130. The fusion features of each image patch are input into the conversion module of the visual encoder, which determines the attention parameters of multiple image patches; and the visual encoder predicts the image features of the sample image based on the attention parameters and fusion features of multiple image patches.

[0049] S140. Adjust the parameters of the visual encoder based on the image features.

[0050] As can be seen from the above, this disclosure proposes an image encoder that supports arbitrary size (i.e., arbitrary resolution) and a method for training the image encoder. First, the original image of arbitrary size is preprocessed to obtain multiple image blocks. Then, based on these multiple image blocks and their positional information, the image features of the entire image are determined, thereby obtaining the image features of the entire image. Since the input image is first divided, the size of the image to be recognized is not limited. This image encoder can encode image features for images of arbitrary size, avoiding image distortion and loss of detail caused by image scaling, thus improving the accuracy of image feature encoding.

[0051] In some implementations, step S110 includes:

[0052] Determine the image patch size;

[0053] Based on the image patch size, the sample image is divided into multiple image patches so that multiple image patches cover the sample image, and there is no overlap between any two image patches.

[0054] An image patch is a concept of a local region. In this embodiment of the disclosure, an image patch focuses on a smaller area compared to the entire image.

[0055] In natural language processing, a token refers to a word, phrase, or symbol in text. In deep learning models, especially when processing sequential data, text can be segmented into a series of tokens. Similar to text segmentation in natural language processing, embodiments of this disclosure divide an image into multiple image patches, each corresponding to a token. Different tokens represent image patches at different locations within the entire image. The visual extractor learns the representations of these tokens to understand the image content and extract image features, thereby converting the raw pixels in the image into image feature vectors in the representation space.

[0056] Figure 2 This is a schematic diagram illustrating the division of an input image into multiple image blocks according to an embodiment of the present disclosure. For example... Figure 2 As shown, the original image has dimensions h*w, where h represents the height of the original image (in pixels) and w represents the width of the original image (in pixels). Figure 2 In the example shown, the image patch is square, with a size of s*s. Here, s represents the number of pixels, and each image patch is a region in the original image, with a height of s pixels and a width of s pixels. The original image is divided into multiple image patches, all of which cover the entire original image, and there is no overlap between the image patches; that is, there is no intersection between any two image patches.

[0057] For example, if the size of the original image is 448*672 and the size of an image block is 8*8, then the entire original image can be divided into (448 / 8)*(672 / 8)=4704 image blocks. The original image is divided into 84 horizontal blocks and 56 vertical blocks.

[0058] In this embodiment, the size of the image patch can be set according to specific circumstances. For example, for larger sample images, the image patch size is larger; for smaller sample images, the image patch size is smaller. This solution divides the sample image based on the image patch size, which can divide the sample image into image patches that are adapted to the processing capabilities of the visual encoder, so that the visual encoder can determine the image features of the sample image and improve the accuracy of image feature determination.

[0059] Figure 3This is a schematic diagram of a visual encoder according to an embodiment of the present disclosure. As shown in Figure 3, the visual encoder includes an embedding module 310, a position encoding module 320, a transformer module 330, and a feature fusion module 340.

[0060] based on Figure 3 In some implementations of the structure shown, during the training process of the visual encoder, the fusion features of each image patch are determined based on multiple image patches and their position information in the sample image, including:

[0061] Multiple image patches are input into the embedding module of the visual encoder to obtain image embedding features of multiple image patches;

[0062] The position information of each image patch in the sample image is input into the position encoding module of the visual encoder to obtain the coordinate embedding features of multiple image patches;

[0063] Based on the image embedding feature and the coordinate embedding feature, the fusion features of each image patch are obtained.

[0064] In one example, the sample image is first divided into multiple image patches. These multiple image patches can be flattened in 2D dimension to obtain a two-dimensional sequence of dimensions [grid_h*grid_w, 3*size*size]. Here, grid_h and grid_w represent the number of image patches obtained in the vertical and horizontal directions respectively when the original image [h,w] is divided according to the size*size, i.e., grid_h = h / size, grid_w = w / size; 3 represents the 3 channels in the image.

[0065] Based on the two-dimensional sequence described above, the encoded image embedding features can be obtained by inputting the embedding module 310 of the visual encoder; for example, multiple image embedding features can be obtained, each carrying image information from an image block.

[0066] Furthermore, since the sample image has been flattened in two dimensions from its original state, the two-dimensional flattening process loses the original two-dimensional spatial information in the sample image. Therefore, the position encoding module of the visual encoder is used to encode the position information of the image block in the sample image in two-dimensional coordinates. The position information is then transformed by encoding to obtain the encoded coordinate embedding features. For example, multiple coordinate embedding features are obtained, and each coordinate embedding feature carries one.

[0067] In some examples, the position information of an image patch in the sample image is represented by the image patch identifier (id). For example, the ID is a data of length N bits. The high N1 bits of the N bits represent the vertical position of the corresponding image patch in the sample image, and the low N2 bits represent the horizontal position of the corresponding image patch in the sample image.

[0068] In some examples, the position encoding module described above can be a Rotary Position Embedding (RoPE) module.

[0069] The RoPE module aims to effectively integrate location information into feature representations without disrupting the temporal and spatial structure of the original data. The core idea of ​​RoPE is to encode location information into each dimension through rotation, enabling the model to capture the relative positional information of elements in the sequence. Specifically, RoPE performs a rotation transformation on the query and key vectors of each token in the input sequence. The transformed vectors not only contain information about the element itself but also incorporate location information. The query vector can be image feature information, and the key vector can be important positions or information within the image feature information. The implementation of RoPE mainly includes the following steps:

[0070] (1) Generate a rotation matrix based on the length of the input sequence and the dimension of the vector. Each element of this matrix is ​​a complex number, with the real and imaginary parts calculated by the cosine and sine functions, respectively. The angle parameter is related to the position of the element in the sequence and the dimension of the vector.

[0071] (2) Multiply the query vector and key vector of each element in the input sequence by the corresponding rotation matrix to obtain the transformed vector. This step performs a rotation transformation on the vector, incorporating positional information into the vector.

[0072] Through the above process, the embedding module 310 outputs the image embedding features of multiple image blocks, and the position encoding module 320 outputs the coordinate embedding features of multiple image blocks. The two are then fused to obtain the fused features of each image block.

[0073] For example, the output of the embedding module is represented as {P1, P2, ..., Pn}, where P1, P2, and Pn represent the image embedding features of the 1st, 2nd, and nth image patches, respectively; the output of the position encoding module is represented as {i1, i2, ..., in}, where i1, i2, and in represent the coordinate embedding features of the 1st, 2nd, and nth image patches, respectively. Fusing the image embedding features and the embedding module yields {P1+i1, P2+i2, ..., Pn+in}, where P1+i1, P2+i2, and Pn+in represent the fused features of the 1st, 2nd, and nth image patches, respectively. It is evident that the fused features of the image patches include image embedding features representing the image detail information within the image patch, and coordinate embedding features representing the positional information of the image patch within the entire image; therefore, they can contain complete information representing the image patch.

[0074] Then, the fused features of multiple image patches are input into the Transformer module 330, which determines the attention parameters of the multiple image patches.

[0075] For example, this attention parameter can include a self-attention parameter. Self-attention is a mechanism that allows a model to focus on and capture the relationships between elements within a sequence when processing input data. Unlike traditional attention mechanisms, self-attention does not rely on external information but directly analyzes the interdependencies within the sequence. This mechanism is particularly effective in handling dependencies and has therefore become an important component of models such as the Transformer.

[0076] In this embodiment of the disclosure, in the Transformer module 330, a self-attention mechanism is used to capture the degree of correlation between the fusion features of image patches at different locations in the fusion features of multiple input image patches, thereby improving the representation ability and generalization performance of the Transformer module 330.

[0077] In some implementations, the Transformer module 330 includes a plurality of sequentially connected Transformer Blocks; the fused features of the plurality of image patches are input to a first-level Transformer Block, which extracts features based on the fused features and inputs the extracted features to a second-level Transformer Block; this process continues until the last-level Transformer Block outputs the attention parameters of the plurality of image patches, and predicts the image features of the sample image based on the attention parameters of the plurality of image patches and the fused features. Figure 3As shown, in some examples, each transformation block includes a multi-head attention unit, an add & normalization unit, and a feedforward unit.

[0078] Since the sample image size can be arbitrary, dividing a large sample image results in a large number of image patches. In this case, if the Transformer module 330 uses a self-attention mechanism to analyze the interdependencies between different image patches to determine the attention parameters of each image patch, the model will consume a large amount of GPU memory during training, and the training speed will also be slow. For example, if the number of image patches is grid_h*grid_w, where grid_h = h / size and grid_w = w / size, the computational complexity is O(grid_h*grid_w)*O(grid_h*grid_w), which is quite large. To reduce the computational complexity of the model during training, reduce the consumption of GPU memory resources, and improve the training speed, this embodiment of the disclosure can divide the sample image into multiple windows, each containing multiple image patches; when determining the attention parameters of the image patches, local attention parameters are first calculated within each window; then, based on the local attention parameters within each window, the global attention parameters of multiple image patches are determined.

[0079] In one example, the fused features of each image patch are input into the Transformer module of the visual encoder. The Transformer module determines the attention parameters for multiple image patches, including:

[0080] Define multiple windows, each containing multiple image blocks;

[0081] For each window, the fusion features of the image patches contained in the window are input into the conversion module, which then determines the local attention parameters for each window.

[0082] The local attention parameters corresponding to each window are input into the conversion module, which then determines the global attention parameters for multiple image patches.

[0083] In some implementations, multiple windows are defined, each containing multiple image blocks, including:

[0084] Determine the window size;

[0085] Multiple windows are determined based on the window size, each containing multiple image blocks, so that the multiple windows cover the sample image, and the image blocks contained in any two windows are different.

[0086] In one example, the window size can represent N1 image blocks in the horizontal direction and N2 image blocks in the vertical direction. Figure 4 This is a schematic diagram of a partitioned window according to an embodiment of the present disclosure. Figure 4 In the example shown, N1 = N2 = 2, meaning that each window contains 2*2 image blocks.

[0087] As can be seen, the embodiments of this disclosure use windows to segment sample images, which enables the fusion of spatially adjacent features, thus greatly reducing the number of tokens after fusion; for example, if using Figure 4 If the window size shown is adjusted, the number of fused tokens will be reduced to 1 / 4 of the original number. This reduces the computational complexity of the model during training, decreases the consumption of GPU memory resources, and increases training speed.

[0088] In some examples, the visual encoder also includes a feature fusion module, which predicts image features of the sample image based on attention parameters and fused features from multiple image patches, including:

[0089] The weights corresponding to each image patch are determined based on the attention parameters of multiple image patches.

[0090] Based on the weights and fusion features corresponding to each image patch, the visual coding features of the sample image are determined.

[0091] A feature fusion module of a visual encoder is used to fuse visual encoded features to obtain image features of the sample image.

[0092] By determining the weights corresponding to image patches based on their attention parameters, and then using these weights to determine visual coding features, the determined visual coding features can be correlated with the relationships between image patches, thus improving the accuracy of the visual coding features. Furthermore, the feature fusion module can reduce dimensionality by fusing features from adjacent image patches to reduce the number of tokens, thereby reducing the computational load of the model.

[0093] This disclosure also proposes a visual encoding method. Figure 5 This is a flowchart illustrating the implementation of a visual encoding method according to an embodiment of the present disclosure, including:

[0094] S510. Divide the image to be encoded into multiple image blocks, the image to be encoded including images of arbitrary size;

[0095] S520. Determine the fusion features of each image block based on multiple image blocks and the position information of each image block in the image to be encoded.

[0096] S530. The fusion features of each image patch are input into the conversion module of the visual encoder, and the conversion module determines the attention parameters of the multiple image patches; and the visual encoder predicts the image features of the image to be encoded based on the attention parameters and fusion features of the multiple image patches.

[0097] The visual encoder can be trained based on any of the training methods described in the above embodiments.

[0098] In some implementations, step S520 includes:

[0099] Based on multiple image patches and their position information within the image to be encoded, the fusion features of each image patch are determined, including:

[0100] Multiple image patches are input into the embedding module of the visual encoder to obtain image embedding features of multiple image patches;

[0101] The position information of each image patch in the image to be encoded is input into the position encoding module of the visual encoder to obtain the coordinate embedding features of multiple image patches;

[0102] Based on image embedding features and coordinate embedding features, the fusion features of each image patch are obtained.

[0103] In some implementations, step S530, which involves inputting the fused features of each image patch into a conversion module of the visual encoder, and having the conversion module determine the attention parameters of the multiple image patches, includes:

[0104] Define multiple windows, each containing multiple image blocks;

[0105] For each window, the fusion features of the image patches contained in the window are input into the conversion module, and the conversion module determines the local attention parameters corresponding to each window.

[0106] The local attention parameters corresponding to each window are input into the conversion module, which then determines the global attention parameters for multiple image patches.

[0107] In one example, multiple windows are defined, each containing multiple image patches, including:

[0108] Determine the window size;

[0109] Multiple windows are determined based on the window size, and each window contains multiple image blocks, so that the multiple windows cover the image to be encoded, and the image blocks contained in any two windows are different.

[0110] This disclosure also proposes a training device for a visual encoder. Figure 6 This is a schematic diagram of a training apparatus 600 for a visual encoder according to an embodiment of the present disclosure, comprising:

[0111] The first partitioning module 610 is used to divide the sample image into multiple image blocks, the sample image including images of arbitrary size;

[0112] The first determining module 620 is used to determine the fusion features of each image block based on multiple image blocks and the position information of each image block in the sample image;

[0113] The second determining module 630 is used to input the fusion features of each image patch into the conversion module of the visual encoder, the conversion module determines the attention parameters of multiple image patches, and the visual encoder predicts the image features of the sample image based on the attention parameters and fusion features of multiple image patches.

[0114] The parameter adjustment module 640 is used to adjust the parameters of the visual encoder based on image features.

[0115] In some implementations, the first determining module 620 is used for:

[0116] Multiple image patches are input into the embedding module of the visual encoder to obtain image embedding features of multiple image patches;

[0117] The position information of each image patch in the sample image is input into the position encoding module of the visual encoder to obtain the coordinate embedding features of multiple image patches;

[0118] Based on image embedding features and coordinate embedding features, the fusion features of each image patch are obtained.

[0119] In some implementations, the first partitioning module 610 is used for:

[0120] Determine the image patch size;

[0121] Based on the image patch size, the sample image is divided into multiple image patches so that multiple image patches cover the sample image, and there is no overlap between any two image patches.

[0122] In some implementations, the second determining module 630 is used for:

[0123] Define multiple windows, each containing multiple image blocks;

[0124] For each window, the fusion features of the image patches contained in the window are input into the conversion module, which then determines the local attention parameters corresponding to each window.

[0125] The local attention parameters corresponding to each window are input into the conversion module, which then determines the global attention parameters for multiple image patches.

[0126] In some implementations, the second determining module 630 is used for:

[0127] Determine the window size;

[0128] Multiple windows are defined based on the window size, and each window contains multiple image patches, so that the multiple windows cover the sample image, and the image patches contained in any two windows are different.

[0129] In some implementations, the second determining module 630 is used for:

[0130] The weights corresponding to each image patch are determined based on the attention parameters of multiple image patches.

[0131] Based on the weights and fusion features corresponding to each image patch, the visual coding features of the sample image are determined.

[0132] A feature fusion module of a visual encoder is used to fuse visual encoded features to obtain image features of the sample image.

[0133] This disclosure also proposes a video encoding device. Figure 7 This is a schematic diagram of the structure of a video encoding apparatus 700 according to an embodiment of the present disclosure, comprising:

[0134] The second partitioning module 710 is used to divide the image to be encoded into multiple image blocks, the image to be encoded including images of arbitrary size;

[0135] The third determining module 720 is used to determine the fusion features of each image block based on multiple image blocks and the position information of each image block in the image to be encoded.

[0136] The fourth determining module 730 is used to input the fusion features of each image patch into the conversion module of the visual encoder, the conversion module determines the attention parameters of multiple image patches, and the visual encoder predicts the image features of the image to be encoded based on the attention parameters and fusion features of the multiple image patches.

[0137] The visual encoder can be trained based on any of the devices provided in this disclosure.

[0138] In some implementations, the third determining module 720 is used for:

[0139] Multiple image patches are input into the embedding module of the visual encoder to obtain image embedding features of multiple image patches;

[0140] The position information of each image patch in the image to be encoded is input into the position encoding module of the visual encoder to obtain the coordinate embedding features of multiple image patches;

[0141] Based on image embedding features and coordinate embedding features, the fusion features of each image patch are obtained.

[0142] In some implementations, the fourth determining module 730 is used for:

[0143] Define multiple windows, each containing multiple image blocks;

[0144] For each window, the fusion features of the image patches contained in the window are input into the conversion module, which then determines the local attention parameters corresponding to each window.

[0145] The local attention parameters corresponding to each window are input into the conversion module, which then determines the global attention parameters for multiple image patches.

[0146] In some implementations, the fourth determining module 730 is used for:

[0147] Determine the window size;

[0148] Multiple windows are determined based on the window size, and each window contains multiple image blocks, so that the multiple windows cover the image to be encoded, and the image blocks contained in any two windows are different.

[0149] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0150] The acquisition, storage, and application of personal information by users involved in this technical solution comply with relevant laws and regulations and do not violate public order and good morals.

[0151] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0152] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0153] like Figure 8As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0154] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0155] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as detection methods. For example, in some embodiments, the detection method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the detection method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the detection method by any other suitable means (e.g., by means of firmware).

[0156] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0157] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0158] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0159] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0160] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0161] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0162] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0163] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training a visual encoder, comprising: The sample image is divided into multiple image blocks, and the sample image includes images of arbitrary size; Based on the plurality of image blocks and the position information of each image block in the sample image, the fusion features of each image block are determined; Multiple windows are defined, each containing multiple image patches, and the image patches are spatially adjacent. For each window, the fusion features of the image patches contained in the window are input into a conversion module, which determines the local attention parameters corresponding to each window. The local attention parameters corresponding to each window are also input into the conversion module, which determines the global attention parameters of the multiple image patches. Based on the global attention parameters of the multiple image patches, the weights corresponding to each image patch are determined. According to the weights corresponding to each image patch and the fusion features, the visual encoding features of the sample image are determined. The feature fusion module of the visual encoder is used to perform feature fusion on the visual encoding features to obtain the image features of the sample image. The image features include image feature vectors. The parameters of the visual encoder are adjusted based on the image features.

2. The method according to claim 1, wherein, The step of determining the fusion features of each image patch based on the plurality of image patches and the position information of each image patch in the sample image includes: The plurality of image blocks are input into the embedding module of the visual encoder to obtain the image embedding features of the plurality of image blocks; The position information of each image patch in the sample image is input into the position encoding module of the visual encoder to obtain the coordinate embedding features of the multiple image patches; Based on the image embedding features and the coordinate embedding features, the fusion features of each image block are obtained.

3. The method according to claim 1 or 2, wherein, The process of dividing the sample image into multiple image blocks includes: Determine the image patch size; Based on the image patch size, the sample image is divided into multiple image patches such that the multiple image patches cover the sample image, and there is no overlap between any two image patches.

4. The method according to claim 1, wherein, The determination of multiple windows, each containing multiple image blocks, includes: Determine the window size; Multiple windows are determined based on the window size, and each window contains multiple image blocks, so that the multiple windows cover the sample image, and the image blocks contained in any two windows are different.

5. A visual encoding method, comprising: The image to be encoded is divided into multiple image blocks, and the image to be encoded includes images of arbitrary size; Based on the plurality of image blocks and the position information of each image block in the image to be encoded, the fusion features of each image block are determined; The fusion features of each image patch are input into the conversion module of the visual encoder, the conversion module determines the attention parameters of the multiple image patches, and the visual encoder predicts the image features of the image to be encoded based on the attention parameters of the multiple image patches and the fusion features. The visual encoder is trained based on the training method described in any one of claims 1-4.

6. The method according to claim 5, wherein, The step of determining the fusion features of each image block based on the plurality of image blocks and the position information of each image block in the image to be encoded includes: The plurality of image blocks are input into the embedding module of the visual encoder to obtain the image embedding features of the plurality of image blocks; The position information of each image block in the image to be encoded is input into the position encoding module of the visual encoder to obtain the coordinate embedding features of the multiple image blocks; Based on the image embedding features and the coordinate embedding features, the fusion features of each image block are obtained.

7. The method according to claim 5 or 6, wherein, The step of inputting the fused features of the various image patches into the conversion module of the visual encoder, and having the conversion module determine the attention parameters of the multiple image patches, includes: Multiple windows are defined, and each window contains multiple image blocks; For each window, the fusion features of the image patches contained in the window are input into the conversion module, and the conversion module determines the local attention parameters corresponding to each window. The local attention parameters corresponding to each window are input into the conversion module, and the conversion module determines the global attention parameters of the multiple image blocks.

8. The method according to claim 7, wherein, The determination of multiple windows, each containing multiple image blocks, includes: Determine the window size; Multiple windows are determined based on the window size, and each window contains multiple image blocks, so that the multiple windows cover the image to be encoded, and the image blocks contained in any two windows are different.

9. A training device for a visual encoder, comprising: The first partitioning module is used to divide the sample image into multiple image blocks, wherein the sample image includes images of arbitrary size; The first determining module is used to determine the fusion features of each image block based on the plurality of image blocks and the position information of each image block in the sample image; The second determining module is used to determine multiple windows, each window containing multiple image blocks, the multiple image blocks being spatially adjacent; for each window, the fusion features of the image blocks contained in the window are input to a conversion module, the conversion module determines the local attention parameters corresponding to each window; the local attention parameters corresponding to each window are input to the conversion module, the conversion module determines the global attention parameters of the multiple image blocks; and based on the global attention parameters of the multiple image blocks, the weights corresponding to each image block are determined; according to the weights corresponding to each image block and the fusion features, the visual encoding features of the sample image are determined; the feature fusion module of the visual encoder is used to perform feature fusion on the visual encoding features to obtain the image features of the sample image; the image features include image feature vectors; The parameter adjustment module is used to adjust the parameters of the visual encoder based on the image features.

10. The apparatus according to claim 9, wherein, The first determining module is used for: The plurality of image blocks are input into the embedding module of the visual encoder to obtain the image embedding features of the plurality of image blocks; The position information of each image patch in the sample image is input into the position encoding module of the visual encoder to obtain the coordinate embedding features of the multiple image patches; Based on the image embedding features and the coordinate embedding features, the fusion features of each image block are obtained.

11. The apparatus according to claim 9 or 10, wherein, The first partitioning module is used for: Determine the image patch size; Based on the image patch size, the sample image is divided into multiple image patches such that the multiple image patches cover the sample image, and there is no overlap between any two image patches.

12. The apparatus according to claim 9, wherein, The second determining module is used for: Determine the window size; Multiple windows are determined based on the window size, and each window contains multiple image blocks, so that the multiple windows cover the sample image, and the image blocks contained in any two windows are different.

13. A video encoding apparatus, comprising: The second partitioning module is used to divide the image to be encoded into multiple image blocks, wherein the image to be encoded includes images of arbitrary size; The third determining module is used to determine the fusion features of each image block based on the plurality of image blocks and the position information of each image block in the image to be encoded; The fourth determining module is used to input the fusion features of each image patch into the conversion module of the visual encoder, the conversion module determines the attention parameters of the multiple image patches, and the visual encoder predicts the image features of the image to be encoded based on the attention parameters of the multiple image patches and the fusion features. The visual encoder is trained based on the training device described in any one of claims 9-12.

14. The apparatus according to claim 13, wherein, The third determining module is used for: The plurality of image blocks are input into the embedding module of the visual encoder to obtain the image embedding features of the plurality of image blocks; The position information of each image block in the image to be encoded is input into the position encoding module of the visual encoder to obtain the coordinate embedding features of the multiple image blocks; Based on the image embedding features and the coordinate embedding features, the fusion features of each image block are obtained.

15. The apparatus according to claim 13 or 14, wherein, The fourth determining module is used to: determine multiple windows, each window containing multiple image blocks; For each window, the fusion features of the image patches contained in the window are input into the conversion module, and the conversion module determines the local attention parameters corresponding to each window. The local attention parameters corresponding to each window are input into the conversion module, and the conversion module determines the global attention parameters of the multiple image blocks.

16. The apparatus according to claim 15, wherein, The fourth determining module is used for: Determine the window size; Multiple windows are determined based on the window size, and each window contains multiple image blocks, so that the multiple windows cover the image to be encoded, and the image blocks contained in any two windows are different.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images