Training method of visual encoder, and visual encoding method and device

Abstract

The invention provides a training method of a visual encoder and a visual encoding method and device, and relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like. According to the specific implementation scheme, a sample image is divided into a plurality of image blocks, wherein the sample image comprises images of any size; according to the plurality of image blocks and the position information of each image block in the sample image, determining fusion features of each image block; inputting the fusion feature of each image block into a conversion module of a visual encoder, and determining attention parameters of the plurality of image blocks by the conversion module; a visual encoder predicts image features of the sample image based on the attention parameters and the fusion features of the plurality of image blocks; parameters of the visual encoder are adjusted based on the image features.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, especially computer vision, deep learning, large models and other technical fields. Background Art

[0002] In current large-scale model applications, visual tasks such as image captioning and visual question answering (VQA) are common tasks for multimodal large models. These tasks usually require the use of visual encoders to extract key visual features in images to obtain feature encoding of the image. How to improve the accuracy of image feature encoding is a technical problem to be solved. Summary of the invention

[0003] The present disclosure provides a training method for a visual encoder, a visual encoding method and a device.

[0004] According to one aspect of the present disclosure, a method for training a visual encoder is provided, comprising:

[0005] Dividing a sample image into a plurality of image blocks, the sample image including an image of any size;

[0006] Determining fusion features of each image block according to the plurality of image blocks and position information of each image block in the sample image;

[0007] The fused features of each image block are input into a conversion module of a visual encoder, and the conversion module determines attention parameters of the multiple image blocks; and the visual encoder predicts image features of the sample image based on the attention parameters and fused features of the multiple image blocks;

[0008] Adjust the parameters of the visual encoder based on image features.

[0009] According to another aspect of the present disclosure, a visual encoding method is provided, comprising:

[0010] Dividing a to-be-encoded image into a plurality of image blocks, the to-be-encoded image including an image of any size;

[0011] Determining a fusion feature of each image block according to the plurality of image blocks and position information of each image block in the image to be encoded;

[0012] The fused features of each image block are input into the conversion module of the visual encoder, and the conversion module determines the attention parameters of the multiple image blocks; and the visual encoder predicts the image features of the image to be encoded based on the attention parameters and fused features of the multiple image blocks.

[0013] According to another aspect of the present disclosure, a training device for a visual encoder is provided, comprising:

[0014] A first division module, used to divide a sample image into a plurality of image blocks, wherein the sample image includes an image of any size;

[0015] A first determination module, used to determine the fusion feature of each image block according to the plurality of image blocks and the position information of each image block in the sample image;

[0016] A second determination module is used to input the fusion features of each image block into a conversion module of a visual encoder, and the conversion module determines the attention parameters of the multiple image blocks; and the visual encoder predicts the image features of the sample image based on the attention parameters and the fusion features of the multiple image blocks;

[0017] The parameter adjustment module is used to adjust the parameters of the visual encoder based on image features.

[0018] According to another aspect of the present disclosure, there is provided a visual encoding device, comprising:

[0019] A second division module, used to divide the image to be encoded into a plurality of image blocks, wherein the image to be encoded includes an image of any size;

[0020] A third determination module, used to determine the fusion feature of each image block according to the plurality of image blocks and the position information of each image block in the image to be encoded;

[0021] The fourth determination module is used to input the fusion features of each image block into the conversion module of the visual encoder, and the conversion module determines the attention parameters of the multiple image blocks; and the visual encoder predicts the image features of the image to be encoded based on the attention parameters and fusion features of the multiple image blocks.

[0022] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0023] at least one processor; and

[0024] a memory communicatively connected to the at least one processor; wherein,

[0025] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0026] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0027] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0028] The present invention divides a sample image to obtain a plurality of image blocks, and then determines the image features of the sample image based on the plurality of image blocks and the position information of each image block, and adjusts the parameters of the visual encoder based on the image features. Since the sample image is divided in the process of training the visual encoder, the size of the image is not restricted in the process of recognizing the image, and the visual encoder can perform feature encoding on images of any size, avoiding image distortion and detail loss caused by image scaling, thereby improving the accuracy of image feature encoding.

[0029] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0031] Figure 1 is a flowchart of an implementation method of an image encoder training method according to an embodiment of the present disclosure;

[0032] Figure 2 is a schematic diagram of dividing an input image into multiple image blocks according to an embodiment of the present disclosure;

[0033] Figure 3 is a structural schematic diagram of a visual encoder according to an embodiment of the present disclosure;

[0034] Figure 4 is a schematic diagram of dividing windows according to an embodiment of the present disclosure;

[0035] Figure 5 is a flowchart of an implementation of a visual encoding method according to an embodiment of the present disclosure;

[0036] Figure 6 is a structural schematic diagram of a visual encoder training device 600 according to an embodiment of the present disclosure;

[0037] Figure 7 is a structural diagram of a video encoding device 700 according to an embodiment of the present disclosure;

[0038] Figure 8 A schematic block diagram of an example electronic device 800 that may be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0039] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0040] The “and / or” in the embodiments of the present disclosure indicates that three relationships may exist. For example, A and / or B may indicate the following three situations: A exists alone, A and B exist at the same time, and B exists alone. The term “at least one” herein indicates any combination of at least two of any one or more of a plurality of types. For example, at least one of A, B, and C may indicate any one or more elements selected from the set consisting of A, B, and C. The terms “first” and “second” herein refer to and distinguish between multiple similar technical terms, and do not mean to limit the order or to limit the meaning to only two. For example, the first feature and the second feature refer to two types / two features. The first feature may be one or more, and the second feature may also be one or more.

[0041] The network structure used by the existing mainstream multimodal large models usually includes three parts: visual encoder (ImageEncoder), visual language alignment layer (Adapter) and language model (Language Model). Generally speaking, a fixed-size image is input into the visual encoder, and the visual encoder extracts the feature information in the image to obtain a visual feature sequence of fixed dimension; then the visual feature sequence is mapped through the language alignment layer (Adapter) to obtain visual embedding features; similarly, the text is input into the embedding model to obtain text embedding features; finally, the visual embedding features and text embedding features are spliced ​​and input into the language model together, and the language model is autoregressively generated to obtain the output result, which is the text information generated according to the input text and image. For example, for the image description task, the input text includes instruction information for describing the image, and the output text includes description information for the input image generated according to the image and the input text.

[0042] The existing multimodal large-model visual encoder relies on a pre-trained visual model to obtain robust visual features; the input image of the visual encoder is an image of a fixed size. Usually, the original image needs to be scaled to the size required by the visual encoder; this method can easily cause image distortion or loss of detail information, which can easily affect the final recognition effect of the visual encoder, that is, cause the image features extracted by the visual encoder to be inaccurate.

[0043] For example, when processing text line images, directly scaling the text line images to a fixed size will cause the text to deform. The visual encoder performs image recognition and image feature extraction on the scaled image, which will affect the model recognition effect.

[0044] In order to solve the above problems, the embodiments of the present disclosure propose a training method for a visual encoder. The visual encoder trained by this method can extract image features of images of any size for multi-model large models to realize applications such as image description and visual question answering.

[0045] Figure 1 is a flowchart of an implementation method of an image encoder training method according to an embodiment of the present disclosure, comprising:

[0046] S110, dividing a sample image into a plurality of image blocks, the sample image including an image of any size;

[0047] S120, determining a fusion feature of each image block according to the plurality of image blocks and position information of each image block in the sample image;

[0048] S130, inputting the fusion features of each image block into a conversion module of a visual encoder, and the conversion module determines attention parameters of the multiple image blocks; and the visual encoder predicts image features of the sample image based on the attention parameters and the fusion features of the multiple image blocks;

[0049] S140. Adjust parameters of the visual encoder based on the image features.

[0050] As can be seen from the above content, the disclosed embodiment proposes an image encoder supporting any size (i.e., any resolution) and a method for training the image encoder, by first preprocessing the original image of any size to obtain multiple image blocks, and then determining the image features of the entire image based on the multiple image blocks and the position information of each image block, thereby obtaining the image features of the entire image. Since the input image is first divided, the size of the image to be identified is not limited, and the image encoder can encode the image features of images of any size, avoiding image distortion and detail loss due to image scaling, thereby improving the accuracy of image feature encoding.

[0051] In some implementations, step S110 includes:

[0052] Determine the image block size;

[0053] Based on the image block size, the sample image is divided into a plurality of image blocks, so that the plurality of image blocks cover the sample image and no intersection area exists between any two image blocks.

[0054] An image block is a concept of a local area. In the embodiment of the present disclosure, an image block focuses on a smaller area compared to the entire image.

[0055] In natural language processing, a token refers to a word, phrase, or symbol in a text. In deep learning models, especially when processing sequence data, text can be segmented into a series of tokens. Similar to the division of text in the field of natural language processing, the disclosed embodiment divides an image into multiple image blocks, each of which corresponds to a token. Different tokens represent image blocks at different positions in the entire image. The visual extractor understands the image content and extracts image features by learning the representation of these tokens, thereby converting the original pixels in the image into image feature vectors representing the space.

[0056] Figure 2 FIG. 1 is a schematic diagram of dividing an input image into a plurality of image blocks according to an embodiment of the present disclosure. Figure 2 As shown in , the size of the original image is h*w, where h represents the height of the original image (specifically, the number of pixels) and w represents the width of the original image (specifically, the number of pixels). Figure 2 In the example shown, the image block is a square, and the size of the image block is s*s. Among them, s represents the number of pixels, and each image block is an area in the original image, and the height of the area is s pixels and the width is s pixels. The original image is divided into multiple image blocks, all of which cover the entire original image, and there is no overlap between the image blocks, that is, there is no intersection area between any two image blocks.

[0057] For example, the size of the original image is 448*672, and the size of an image block is 8*8, then the entire original image can be divided into (448 / 8)*(672 / 8)=4704 image blocks, and the original image is divided into 84 image blocks in the horizontal direction and 56 image blocks in the vertical direction.

[0058] In the embodiment of the present solution, the size of the image block can be set according to the specific situation. For example, for a sample image with a larger size, the image block size is larger; for a sample image with a smaller size, the image block size is smaller. The present solution divides the sample image based on the image block size, and can divide the sample image into image blocks adapted to the processing capability of the visual encoder, so that the visual encoder can determine the image features of the sample image and improve the accuracy of determining the image features.

[0059] Figure 3It is a structural diagram of a visual encoder according to an embodiment of the present disclosure. As shown in 3, the visual encoder includes an embedding module 310, a position encoding module 320, a transformation module 330 and a feature fusion module 340.

[0060] based on Figure 3 In some embodiments, during the training of the visual encoder, the fusion features of each image block are determined according to the multiple image blocks and the position information of each image block in the sample image, including:

[0061] Inputting the multiple image blocks into the embedding module of the visual encoder to obtain image embedding features of the multiple image blocks;

[0062] Inputting the position information of each image block in the sample image into the position encoding module of the visual encoder to obtain the coordinate embedding features of multiple image blocks;

[0063] Based on the image embedding feature and the coordinate embedding feature, a fusion feature of each image block is obtained.

[0064] In one example, a sample image is first divided to obtain multiple image blocks. After the multiple image blocks are divided, the sample image can be flattened in the 2D dimension to obtain a two-dimensional sequence of dimension [grid_h*grid_w,3*size*size]. Among them, grid_h and grid_w respectively represent the number of image blocks obtained in the vertical and horizontal directions when the original image [h,w] is divided according to the size of size*size, that is, grid_h=h / size, grid_w=w / size; 3 represents the 3 channels in the image.

[0065] Based on the above two-dimensional sequence, by inputting the embedding module 310 of the visual encoder, the encoded image embedding features can be obtained; for example, multiple image embedding features are obtained, each image embedding feature carries the image information in an image block.

[0066] Furthermore, since the sample image has been two-dimensionally flattened from its original state, the two-dimensional flattening process loses the original spatial two-dimensional information in the sample image. Therefore, the position encoding module of the visual encoder is used to encode the position information of the image block in the sample image in two-dimensional coordinates, and the position information is transformed by encoding to obtain the encoded coordinate embedding feature; for example, multiple coordinate embedding features are obtained, each of which carries one.

[0067] In some examples, the position information of the image block in the sample image is represented by an identifier (ID) of the image block. For example, the ID is data with a length of N bits, and the high-order N1 bits of the N bits represent the vertical position of the corresponding image block in the sample image; the low-order N2 bits of the N bits represent the horizontal position of the corresponding image block in the sample image.

[0068] In some examples, the position encoding module may be a rotational position encoding (Rotary Position Embedding, RoPE) module.

[0069] The RoPE module aims to effectively integrate position information into feature representation without destroying the temporal and spatial structure of the original data. The core idea of ​​RoPE is to encode position information into each dimension by rotation, so that the model can capture the relative position information of elements in the sequence. Specifically, RoPE rotates the query vector and key vector of each token in the input sequence. The transformed vector not only contains the information of the element itself, but also incorporates the position information. Among them, the query vector can be image feature information, and the key vector can be important positions or information in the image feature information. The implementation of RoPE mainly includes the following steps:

[0070] (1) Generate a rotation matrix based on the length of the input sequence and the dimension of the vector. Each element of this matrix is ​​a complex number, the real part and imaginary part are calculated by the cosine and sine functions respectively, and the angle parameter is related to the position of the element in the sequence and the dimension of the vector.

[0071] (2) Multiply the query vector and key vector of each element in the input sequence by the corresponding rotation matrix to obtain the transformed vector. This step is to perform a rotation transformation on the vector to incorporate the position information into the vector.

[0072] Through the above process, the embedding module 310 outputs the image embedding features of multiple image blocks, and the position encoding module 320 outputs the coordinate embedding features of multiple image blocks, and then the two are fused to obtain the fused features of each image block.

[0073] For example, the output of the embedding module is represented as: {P1, P2,…, Pn}, where P1, P2, and Pn represent the image embedding features of the first, second, and nth image blocks, respectively; the output of the position encoding module is represented as: {i1, i2,…, in}, where i1, i2, and in represent the coordinate embedding features of the first, second, and nth image blocks, respectively. The image embedding features and the embedding module are fused to obtain {P1+i1, P2+i2,…, Pn+in}, where P1+i1, P2+i2, and Pn+in represent the fused features of the first, second, and nth image blocks, respectively. It can be seen that the fused features of the image block include the image embedding features used to represent the image detail information in the image block, and the coordinate embedding features used to represent the position information of the image block in the entire image, and therefore can contain the complete information representing the image block.

[0074] Afterwards, the fused features of the multiple image blocks are input into the Transformer module 330, and the Transformer module 330 determines the attention parameters of the multiple image blocks.

[0075] For example, the attention parameter may include a self-attention parameter. The self-attention mechanism is a mechanism that allows the model to self-focus on the input data and capture the relationship between elements within the sequence. Unlike traditional attention mechanisms, the self-attention mechanism does not rely on external information, but directly analyzes the interdependencies within the sequence. This mechanism is particularly effective when there are dependencies, so it has become an important part of models such as Transformer.

[0076] In the embodiment of the present disclosure, in the transformer module 330, the self-attention mechanism is used to capture the correlation between the fusion features of image blocks at different positions in the fusion features of multiple input image blocks, thereby improving the representation ability and generalization performance of the transformer module 330.

[0077] In some embodiments, the transformer module 330 includes a plurality of transformer blocks connected in sequence; the fused features of the plurality of image blocks are input into a first-level transformer block, the first-level transformer block extracts features based on the fused features, and inputs the extracted features into a second-level transformer block; and so on and so forth until the last-level transformer block outputs the attention parameters of the plurality of image blocks, and predicts the image features of the sample image based on the attention parameters and fused features of the plurality of image blocks. Figure 3As shown, in some examples, each conversion block includes a multi-head attention unit, a normalization (Add&Norm) unit, and a feedforward (FeedForward) unit.

[0078] Since the size of the sample image can be of any size, when the sample image with a larger size is divided, the number of image blocks obtained is larger. In this case, if the transformer module 330 adopts a self-attention mechanism to analyze the interdependence between different image blocks to determine the attention parameters of each image block, the model will occupy a larger amount of video memory resources during training, and the training speed will also be slower. For example, the number of image blocks is grid_h*grid_w, where grid_h=h / size, grid_w=w / size, then the computational complexity is o(grid_h*grid_w)*o(grid_h*grid_w), which is relatively large. In order to reduce the computational complexity of the model during training, reduce the occupation of video memory resources, and increase the training speed, the embodiment of the present disclosure can divide the sample image into multiple windows, each window containing multiple image blocks; when determining the attention parameters of the image blocks, first calculate the local attention parameters in each window; then determine the global attention parameters of multiple image blocks based on the local attention parameters in each window.

[0079] In one example, the fusion features of each image block are input into the Transformer module of the visual encoder, and the Transformer module determines the attention parameters of multiple image blocks, including:

[0080] Determine a plurality of windows, each window containing a plurality of image blocks;

[0081] For each window, the fusion features of the image blocks contained in the window are input into the conversion module, and the conversion module determines the local attention parameters corresponding to each window;

[0082] The local attention parameters corresponding to each window are input into the conversion module, and the conversion module determines the global attention parameters of multiple image blocks.

[0083] In some implementations, determining a plurality of windows, each window containing a plurality of image blocks, comprises:

[0084] Determine the window size;

[0085] A plurality of windows are determined based on the window size, each window including a plurality of image blocks, so that the plurality of windows cover the sample image, and the image blocks included in any two windows are different.

[0086] In one example, the window size may indicate that it includes N1 image blocks in the horizontal direction and N2 image blocks in the vertical direction. Figure 4 is a schematic diagram of dividing windows according to an embodiment of the present disclosure, Figure 4 In the example shown, N1=N2=2, that is, each window includes 2*2 image blocks.

[0087] It can be seen that the embodiment of the present disclosure uses a window to segment the sample image, which can fuse the features adjacent to each other in spatial position, and the number of tokens after fusion will be greatly reduced; for example, if Figure 4 If the window size is smaller than that shown in the figure, the number of fused tokens will be reduced to 1 / 4 of the original number. Therefore, the computational complexity of the model during training can be reduced, the usage of video memory resources can be reduced, and the training speed can be increased.

[0088] In some examples, the visual encoder further includes a feature fusion module, wherein the visual encoder predicts image features of the sample image based on attention parameters and fusion features of multiple image blocks, including:

[0089] Based on the attention parameters of multiple image blocks, determine the weight corresponding to each image block;

[0090] Determine the visual coding features of the sample image according to the weights and fusion features corresponding to each image block;

[0091] The feature fusion module of the visual encoder is used to fuse the visual encoding features to obtain the image features of the sample image.

[0092] Based on the attention parameters of the image blocks, the weights corresponding to the image blocks are determined, and the visual coding features are determined based on the weights, so that the determined visual coding features can be related to the correlation between the image blocks, thereby improving the accuracy of the visual coding features. In addition, the feature fusion module can play a role in dimensionality reduction, fusing the features of adjacent image blocks to reduce the number of tokens, thereby reducing the amount of calculation of the model.

[0093] The present disclosure also provides a visual encoding method. Figure 5 is a flowchart of a visual encoding method according to an embodiment of the present disclosure, including:

[0094] S510, dividing an image to be encoded into a plurality of image blocks, where the image to be encoded includes an image of any size;

[0095] S520, determining a fusion feature of each image block according to the plurality of image blocks and position information of each image block in the image to be encoded;

[0096] S530, inputting the fusion features of each image block into a conversion module of a visual encoder, and the conversion module determines attention parameters of the multiple image blocks; and the visual encoder predicts the image features of the image to be encoded based on the attention parameters and the fusion features of the multiple image blocks;

[0097] The visual encoder may be trained based on any of the training methods in the above-mentioned implementations.

[0098] In some implementations, step S520 includes:

[0099] According to the plurality of image blocks and the position information of each image block in the image to be encoded, the fusion feature of each image block is determined, including:

[0100] Inputting the multiple image blocks into the embedding module of the visual encoder to obtain image embedding features of the multiple image blocks;

[0101] Inputting the position information of each image block in the image to be encoded into the position encoding module of the visual encoder to obtain the coordinate embedding features of the multiple image blocks;

[0102] Based on the image embedding features and coordinate embedding features, the fusion features of each image block are obtained.

[0103] In some embodiments, the step S530 of inputting the fusion features of each image block into a conversion module of the visual encoder, and the conversion module determining the attention parameters of the multiple image blocks includes:

[0104] Determine a plurality of windows, each window containing a plurality of image blocks;

[0105] For each window, the fusion features of the image blocks contained in the window are input into the conversion module, and the conversion module determines the local attention parameters corresponding to each window;

[0106] The local attention parameters corresponding to each window are input into the conversion module, and the conversion module determines the global attention parameters of multiple image blocks.

[0107] In one example, a plurality of windows are determined, each of which contains a plurality of image blocks, including:

[0108] Determine the window size;

[0109] A plurality of windows are determined based on the window size, each window including a plurality of image blocks, so that the plurality of windows cover the image to be encoded, and the image blocks included in any two windows are different.

[0110] The disclosed embodiment also provides a visual encoder training device, Figure 6 is a structural diagram of a visual encoder training device 600 according to an embodiment of the present disclosure, comprising:

[0111] A first division module 610 is used to divide a sample image into a plurality of image blocks, where the sample image includes an image of any size;

[0112] A first determination module 620, configured to determine a fusion feature of each image block according to the plurality of image blocks and position information of each image block in the sample image;

[0113] A second determination module 630 is used to input the fusion features of each image block into a conversion module of a visual encoder, and the conversion module determines the attention parameters of the multiple image blocks; and the visual encoder predicts the image features of the sample image based on the attention parameters and the fusion features of the multiple image blocks;

[0114] The parameter adjustment module 640 is used to adjust the parameters of the visual encoder based on the image features.

[0115] In some implementations, the first determining module 620 is configured to:

[0116] Inputting the multiple image blocks into the embedding module of the visual encoder to obtain image embedding features of the multiple image blocks;

[0117] Inputting the position information of each image block in the sample image into the position encoding module of the visual encoder to obtain the coordinate embedding features of multiple image blocks;

[0118] Based on the image embedding features and coordinate embedding features, the fusion features of each image block are obtained.

[0119] In some implementations, the first partitioning module 610 is configured to:

[0120] Determine the image block size;

[0121] Based on the image block size, the sample image is divided into a plurality of image blocks, so that the plurality of image blocks cover the sample image and no intersection area exists between any two image blocks.

[0122] In some implementations, the second determination module 630 is configured to:

[0123] Determine a plurality of windows, each window containing a plurality of image blocks;

[0124] For each window, the fusion features of the image blocks contained in the window are input into the conversion module, and the conversion module determines the local attention parameters corresponding to each window;

[0125] The local attention parameters corresponding to each window are input into the conversion module, which determines the global attention parameters of multiple image blocks.

[0126] In some implementations, the second determination module 630 is configured to:

[0127] Determine the window size;

[0128] A plurality of windows are determined based on the window size, each window includes a plurality of image blocks, so that the plurality of windows cover the sample image, and the image blocks included in any two windows are different.

[0129] In some implementations, the second determination module 630 is configured to:

[0130] Based on the attention parameters of multiple image blocks, determine the weight corresponding to each image block;

[0131] Determine the visual coding features of the sample image according to the weights and fusion features corresponding to each image block;

[0132] The feature fusion module of the visual encoder is used to fuse the visual encoding features to obtain the image features of the sample image.

[0133] The present disclosure also provides a video encoding device. Figure 7 is a schematic structural diagram of a video encoding device 700 according to an embodiment of the present disclosure, comprising:

[0134] A second division module 710, configured to divide the image to be encoded into a plurality of image blocks, wherein the image to be encoded includes an image of any size;

[0135] A third determination module 720, configured to determine a fusion feature of each image block according to the plurality of image blocks and position information of each image block in the image to be encoded;

[0136] A fourth determination module 730 is used to input the fusion features of each image block into a conversion module of a visual encoder, and the conversion module determines the attention parameters of the multiple image blocks; and the visual encoder predicts the image features of the image to be encoded based on the attention parameters and the fusion features of the multiple image blocks;

[0137] Among them, the visual encoder can be trained based on any device provided in the present disclosure.

[0138] In some implementations, the third determination module 720 is configured to:

[0139] Inputting the multiple image blocks into the embedding module of the visual encoder to obtain image embedding features of the multiple image blocks;

[0140] Inputting the position information of each image block in the image to be encoded into the position encoding module of the visual encoder to obtain the coordinate embedding features of the multiple image blocks;

[0141] Based on the image embedding features and coordinate embedding features, the fusion features of each image block are obtained.

[0142] In some implementations, the fourth determination module 730 is configured to:

[0143] Determine a plurality of windows, each window containing a plurality of image blocks;

[0144] For each window, the fusion features of the image blocks contained in the window are input into the conversion module, and the conversion module determines the local attention parameters corresponding to each window;

[0145] The local attention parameters corresponding to each window are input into the conversion module, which determines the global attention parameters of multiple image blocks.

[0146] In some implementations, the fourth determination module 730 is configured to:

[0147] Determine the window size;

[0148] A plurality of windows are determined based on the window size, each window including a plurality of image blocks, so that the plurality of windows cover the image to be encoded, and the image blocks included in any two windows are different.

[0149] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0150] In the technical solution disclosed in the present invention, the acquisition, storage and application of personal information of users involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0151] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0152] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0153] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0154] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0155] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as detection methods. For example, in some embodiments, the detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the detection method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the detection method in any other appropriate manner (e.g., by means of firmware).

[0156] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0157] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0158] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0159] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0160] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0161] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0162] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0163] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a visual encoder, comprising: Dividing a sample image into a plurality of image blocks, wherein the sample image includes an image of any size; Determining a fusion feature of each of the image blocks according to the plurality of image blocks and position information of each image block in the sample image; Inputting the fusion features of the respective image blocks into a conversion module of the visual encoder, and determining the attention parameters of the plurality of image blocks by the conversion module; and predicting the image features of the sample image by the visual encoder based on the attention parameters of the plurality of image blocks and the fusion features; Parameters of the visual encoder are adjusted based on the image features.

2. The method according to claim 1, wherein: The determining, according to the plurality of image blocks and the position information of each image block in the sample image, a fusion feature of each image block comprises: Inputting the plurality of image blocks into an embedding module of the visual encoder to obtain image embedding features of the plurality of image blocks; Inputting the position information of each image block in the sample image into the position encoding module of the visual encoder to obtain the coordinate embedding features of the multiple image blocks; Based on the image embedding features and the coordinate embedding features, fusion features of each of the image blocks are obtained.

3. The method according to claim 1 or 2, wherein: The step of dividing the sample image into a plurality of image blocks comprises: Determine the image block size; Based on the image block size, the sample image is divided into a plurality of image blocks, so that the plurality of image blocks cover the sample image and no intersection area exists between any two of the image blocks.

4. The method according to claim 1 or 2, wherein: The step of inputting the fusion features of the respective image blocks into a conversion module of the visual encoder, and determining the attention parameters of the plurality of image blocks by the conversion module, comprises: Determine a plurality of windows, each of the windows containing a plurality of the image blocks; For each of the windows, the fusion features of the image blocks contained in the window are input into the conversion module, and the conversion module determines the local attention parameters corresponding to each of the windows; The local attention parameters corresponding to each of the windows are input into the conversion module, and the conversion module determines the global attention parameters of the multiple image blocks.

5. The method according to claim 4, wherein: The determining of a plurality of windows, each of which contains a plurality of image blocks, comprises: Determine the window size; A plurality of windows are determined based on the window size, each of the windows includes a plurality of the image blocks, so that the plurality of windows cover the sample image, and the image blocks included in any two of the windows are different.

6. The method according to claim 1 or 2, wherein: The predicting, by the visual encoder based on the attention parameters of the plurality of image blocks and the fusion features, the image features of the sample image comprises: Determining a weight corresponding to each of the image blocks based on the attention parameters of the multiple image blocks; Determining the visual coding features of the sample image according to the weights corresponding to the image blocks and the fusion features; The feature fusion module of the visual encoder is used to perform feature fusion on the visual coding features to obtain image features of the sample image.

7. A visual encoding method, comprising: Dividing a to-be-encoded image into a plurality of image blocks, wherein the to-be-encoded image includes an image of any size; Determining a fusion feature of each of the image blocks according to the plurality of image blocks and position information of each image block in the image to be encoded; The fused features of the respective image blocks are input into a conversion module of a visual encoder, and the conversion module determines attention parameters of the plurality of image blocks; and the visual encoder predicts image features of the image to be encoded based on the attention parameters of the plurality of image blocks and the fused features.

8. The method according to claim 7, wherein: The determining, according to the plurality of image blocks and the position information of each image block in the image to be encoded, a fusion feature of each image block comprises: Inputting the plurality of image blocks into an embedding module of the visual encoder to obtain image embedding features of the plurality of image blocks; Inputting the position information of each image block in the image to be encoded into the position encoding module of the visual encoder to obtain the coordinate embedding features of the multiple image blocks; Based on the image embedding features and the coordinate embedding features, fusion features of each of the image blocks are obtained.

9. The method according to claim 7 or 8, wherein: The step of inputting the fusion features of the respective image blocks into a conversion module of the visual encoder, and determining the attention parameters of the plurality of image blocks by the conversion module, comprises: Determine a plurality of windows, each of the windows containing a plurality of the image blocks; For each of the windows, the fusion features of the image blocks contained in the window are input into the conversion module, and the conversion module determines the local attention parameters corresponding to each of the windows; The local attention parameters corresponding to each of the windows are input into the conversion module, and the conversion module determines the global attention parameters of the multiple image blocks.

10. The method according to claim 9, wherein: The determining of a plurality of windows, each of which contains a plurality of image blocks, comprises: Determine the window size; A plurality of windows are determined based on the window size, each of the windows comprising a plurality of the image blocks, so that the plurality of windows cover the image to be encoded, and the image blocks contained in any two of the windows are different.

11. A training device for a visual encoder, comprising: A first division module, used to divide a sample image into a plurality of image blocks, wherein the sample image includes an image of any size; A first determining module, configured to determine a fusion feature of each image block according to the plurality of image blocks and position information of each image block in the sample image; a second determination module, configured to input the fusion features of the respective image blocks into a conversion module of the visual encoder, and the conversion module determines the attention parameters of the plurality of image blocks; and the visual encoder predicts the image features of the sample image based on the attention parameters of the plurality of image blocks and the fusion features; A parameter adjustment module is used to adjust the parameters of the visual encoder based on the image features.

12. The device according to claim 11, wherein: The first determining module is used to: Inputting the plurality of image blocks into an embedding module of the visual encoder to obtain image embedding features of the plurality of image blocks; Inputting the position information of each image block in the sample image into the position encoding module of the visual encoder to obtain the coordinate embedding features of the multiple image blocks; Based on the image embedding features and the coordinate embedding features, fusion features of each of the image blocks are obtained.

13. The device according to claim 11 or 12, wherein: The first division module is used for: Determine the image block size; Based on the image block size, the sample image is divided into a plurality of image blocks, so that the plurality of image blocks cover the sample image and no intersection area exists between any two of the image blocks.

14. The device according to claim 11 or 12, wherein: The second determining module is used to: Determine a plurality of windows, each of the windows containing a plurality of the image blocks; For each of the windows, the fusion features of the image blocks contained in the window are input into the conversion module, and the conversion module determines the local attention parameters corresponding to each of the windows; The local attention parameters corresponding to each of the windows are input into the conversion module, and the conversion module determines the global attention parameters of the multiple image blocks.

15. The device according to claim 14, wherein: The second determining module is used to: Determine the window size; A plurality of windows are determined based on the window size, each of the windows includes a plurality of the image blocks, so that the plurality of windows cover the sample image, and the image blocks included in any two of the windows are different.

16. The device according to claim 11 or 12, wherein: The second determining module is used to: Determining a weight corresponding to each of the image blocks based on the attention parameters of the multiple image blocks; Determining the visual coding features of the sample image according to the weights corresponding to the image blocks and the fusion features; The feature fusion module of the visual encoder is used to perform feature fusion on the visual coding features to obtain image features of the sample image.

17. A video encoding device, comprising: A second division module, used to divide the image to be encoded into a plurality of image blocks, wherein the image to be encoded includes an image of any size; A third determination module, configured to determine a fusion feature of each image block according to the plurality of image blocks and position information of each image block in the image to be encoded; The fourth determination module is used to input the fusion features of the respective image blocks into the conversion module of the visual encoder, and the conversion module determines the attention parameters of the multiple image blocks; and the visual encoder predicts the image features of the image to be encoded based on the attention parameters of the multiple image blocks and the fusion features.

18. The device according to claim 17, wherein: The third determining module is used to: Inputting the plurality of image blocks into an embedding module of the visual encoder to obtain image embedding features of the plurality of image blocks; Inputting the position information of each image block in the image to be encoded into the position encoding module of the visual encoder to obtain the coordinate embedding features of the multiple image blocks; Based on the image embedding features and the coordinate embedding features, fusion features of each of the image blocks are obtained.

19. The device according to claim 17 or 18, wherein: The fourth determining module is used to: determine a plurality of windows, each of which contains a plurality of the image blocks; For each of the windows, the fusion features of the image blocks contained in the window are input into the conversion module, and the conversion module determines the local attention parameters corresponding to each of the windows; The local attention parameters corresponding to each of the windows are input into the conversion module, and the conversion module determines the global attention parameters of the multiple image blocks.

20. The device according to claim 19, wherein The fourth determining module is used to: Determine the window size; A plurality of windows are determined based on the window size, each of the windows comprising a plurality of the image blocks, so that the plurality of windows cover the image to be encoded, and the image blocks contained in any two of the windows are different.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

23. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images