A pre-training method, device and electronic device for a large visual model

Abstract

The present invention discloses a pre-training method, device and electronic device for a visual big model, and belongs to the field of artificial intelligence technology. The pre-training method includes: obtaining an unlabeled image and performing segmentation and masking of the segmentation to obtain a masked processed image; based on the masked processed image, pre-training an improved visual big model with total loss as the optimization target; the improved visual big model is obtained by introducing a position prediction module at the output position of the encoder on the basis of a standard visual big model including an encoder and a decoder. The improved visual big model can pay attention to the content and structure of the image at the same time during the pre-training process, thereby improving the generalization ability of the model, the ability to model and understand position information, and the ability to understand image content, and solves the shortcomings of the existing standard visual big model in understanding spatial structure. The present invention provides a more powerful pre-training model for computer vision tasks.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a pre-training method, device and electronic equipment for a large visual model. Background Art

[0002] Currently, Masked Autoencoders (MAE) has become an important solution for pre-training large visual models. MAE is a self-supervised learning model that learns the representation of an image by masking a portion of the image and letting the model reconstruct these masked parts. This pre-training method performs well in the field of computer vision because it can effectively use a large amount of unlabeled image data to pre-train the model.

[0003] However, MAE also has some limitations in "missing spatial position information modeling": the existing MAE model mainly focuses on pixel reconstruction of the image, and directly gives the position information of the unmasked slices in the input of the decoder (docoder) responsible for reconstruction, which reduces the difficulty of pre-training the model to a certain extent, causing the MAE model to ignore the spatial structure information of the image. This means that the MAE model may not be able to fully learn to understand the position and layout of objects in the image, which is very important for certain visual tasks (such as object detection and image segmentation). Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention provides the following technical solutions.

[0005] A first aspect of the present invention provides a pre-training method for a large visual model, comprising:

[0006] Acquire an unlabeled image and perform slice and mask slice processing to obtain a mask-processed image, wherein the mask-processed image includes unmasked image slices and masked image slices;

[0007] Based on the masking process, the total loss To optimize the goal, the improved visual large model is pre-trained;

[0008] The total loss Calculate according to the following formula:

[0009] ;

[0010] ;

[0011] ;

[0012] in, is the total loss, To rebuild the losses, is the position prediction loss, It is used for balance and is a hyperparameter, N is the number of unmasked image slices, are the predicted normalized coordinates of the i-th unmasked image patch, is the true normalized coordinate of the i-th unmasked image patch;

[0013] The improved visual big model is obtained by introducing a position prediction module at the output position of the encoder on the basis of a standard visual big model including an encoder and a decoder; the standard visual big model is used to reconstruct the obscured image slice given the specific position of the unobstructed image slice, and the position prediction module is used to predict the position of the unobstructed image slice in the image.

[0014] Preferably, the position prediction module is composed of a feed-forward layer FFN x and FFN y Composition, feed-forward layer FFN x and FFN y The representation h of the i-th unmasked image slice output by the encoder is i As input, after linear layer transformation, the activation function is used to generate the predicted horizontal and vertical coordinates of the i-th unmasked image segment.

[0015] Preferably, the abscissa and ordinate of the predicted i-th unmasked image slice are calculated according to the following formula:

[0016] ;

[0017] in, is the representation of the i-th unmasked image slice after being encoded by the encoder, FFN is the feed-forward layer x The mapping matrix, FFN is the feed-forward layer y The mapping matrix, is the sigmoid activation function, is the normalized horizontal coordinate of the predicted i-th unmasked image patch, is the normalized ordinate of the predicted i-th unoccluded image patch.

[0018] Preferably, the reconstruction loss is calculated according to the following formula:

[0019] ;

[0020] in, represents the predicted value of the pixel with coordinates (j, k) in the tth masked image slice, for The corresponding actual values ​​are, M is the size of the tth masked image slice, and L is the number of masked image slices.

[0021] Preferably, the standard visual macro model adopts a MAE model.

[0022] A second aspect of the present invention provides a pre-training device for a large visual model, comprising:

[0023] A data acquisition module is used to acquire an unlabeled image and perform slice processing and masked slice processing to obtain a masked processed image, wherein the masked processed image includes unmasked image slices and masked image slices;

[0024] Pre-trained module for processing images based on masking, with a total loss To optimize the goal, the improved visual large model is pre-trained;

[0025] The total loss Calculate according to the following formula:

[0026] ;

[0027] ;

[0028] ;

[0029] in, is the total loss, To rebuild the losses, is the position prediction loss, It is used for balance and is a hyperparameter, N is the number of unmasked image slices, are the predicted normalized coordinates of the i-th unmasked image patch, is the true normalized coordinate of the i-th unmasked image patch;

[0030] The improved visual big model is obtained by introducing a position prediction module at the output position of the encoder on the basis of a standard visual big model including an encoder and a decoder; the standard visual big model is used to reconstruct the obscured image slice given the specific position of the unobstructed image slice, and the position prediction module is used to predict the position of the unobstructed image slice in the image.

[0031] Preferably, the position prediction module is composed of a feed-forward layer FFN x and FFN y Composition, feed-forward layer FFN x and FFN y The representation h of the i-th unmasked image slice output by the encoder isi As input, after linear layer transformation, the activation function is used to generate the predicted horizontal and vertical coordinates of the i-th unmasked image segment.

[0032] Preferably, the standard visual macro model adopts a MAE model.

[0033] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the pre-training method of the large visual model as described in the first aspect.

[0034] The present invention also provides an electronic device, comprising a processor and a memory connected to the processor, wherein the memory stores a plurality of instructions, and the instructions can be loaded and executed by the processor so that the processor can execute the pre-training method of the large visual model as described in the first aspect.

[0035] The beneficial effects of the present invention are as follows: the pre-training method, device and electronic device of the visual big model provided by the present invention introduce a position prediction module at the output position of the encoder of the standard visual big model to predict the position of the unobstructed slice in the image, so that the obtained improved visual big model can pay attention to the content and structure of the image at the same time during the pre-training process, thereby improving the generalization ability of the model, the ability to model and understand position information, and the ability to understand image content, and solves the shortcomings of the existing standard visual big model in understanding spatial structure, so as to provide a more powerful pre-training model for computer vision tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the process of pre-training the large visual model of the present invention;

[0037] Figure 2 This is a block diagram of the pre-training implementation process of the large visual model described in the present invention;

[0038] Figure 3 Schematic diagram of the functional structure of the pre-training device for the large visual model described in the present invention. DETAILED DESCRIPTION

[0039] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0040] The method provided by the present invention can be implemented in the following terminal environment, and the terminal may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method described in the following embodiment.

[0041] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the entire terminal, and executes various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory.

[0042] The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets or instructions.

[0043] The display screen is used to display the user interface of each application.

[0044] In addition, those skilled in the art can understand that the structure of the above terminal does not constitute a limitation on the terminal, and the terminal may include more or fewer components, or combine certain components, or arrange the components differently. For example, the terminal also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, and a power supply, which will not be described in detail here.

[0045] Embodiment 1

[0046] like Figure 1 and Figure 2 As shown, the embodiment of the present invention provides a pre-training method for a large visual model, including: S101, obtaining an unlabeled image and performing segmentation and mask segmentation processing to obtain a masked processed image, wherein the masked processed image includes an unmasked image segment and a masked image segment; S102, based on the masked processed image, using a total loss To optimize the goal, the improved visual big model is pre-trained.

[0047] The total loss Calculate according to the following formula:

[0048] ;

[0049] ;

[0050] ;

[0051] in, is the total loss, To rebuild the losses, is the position prediction loss, It is used for balance and is a hyperparameter, N is the number of unmasked image slices, are the predicted normalized coordinates of the i-th unmasked image patch, are the true normalized coordinates of the i-th unoccluded image patch.

[0052] The improved visual big model is obtained by introducing a position prediction module at the output position of the encoder on the basis of a standard visual big model including an encoder and a decoder; the standard visual big model is used to reconstruct the obscured image slice given the specific position of the unobstructed image slice, and the position prediction module is used to predict the position of the unobstructed image slice in the image.

[0053] In the present invention, based on the standard visual macro model including an encoder and a decoder, a position prediction module is introduced at the output position of the encoder to obtain an improved visual macro model, such as Figure 2 During the pre-training process, the improved visual model not only needs to learn how to reconstruct the masked image slices (from the input image to the target image) given the specific positions of the unmasked slices, but also needs to learn how to predict the positions of the unmasked slices in the image.

[0054] The position prediction module consists of a feed-forward layer FFN x and FFN y Composition, feed-forward layer FFN x and FFN y The representation h of the i-th unmasked image slice output by the encoder is i As input, after linear layer transformation, the activation function is used to generate the predicted horizontal and vertical coordinates of the i-th unmasked image segment.

[0055] In a preferred embodiment of the present invention, the abscissa and ordinate of the predicted i-th unmasked image slice are calculated according to the following formula:

[0056] ;

[0057] in, is the representation of the i-th unmasked image slice after being encoded by the encoder, FFN is the feed-forward layer x The mapping matrix, FFN is the feed-forward layer y The mapping matrix, is the sigmoid activation function, is the normalized horizontal coordinate of the predicted i-th unmasked image patch, is the normalized ordinate of the predicted i-th unobstructed image slice. By introducing a coordinate regression task, the horizontal and vertical coordinates of each unobstructed image slice can be output. In the present invention, this can be achieved by supervised learning, using the actual slice position as a label.

[0058] In the present invention, the reconstruction loss can be calculated according to the following formula:

[0059] ;

[0060] in, represents the predicted value of the pixel with coordinates (j, k) in the tth masked image slice, for The corresponding actual values ​​are, M is the size of the tth masked image slice, and L is the number of masked image slices.

[0061] The reconstruction loss can be used to ensure that the model can accurately reconstruct the occluded image slices, and the position prediction loss can be used to ensure that the model can accurately predict the position of the unoccluded image slices. By adjusting the weights in the total loss, the importance of the two losses can be balanced to ensure that the model can achieve good performance on both the reconstruction task and the position prediction task.

[0062] In the present invention, the standard large visual model may adopt the MAE model.

[0063] The improved large visual model trained by the method of the present invention can be applied to various downstream visual tasks, such as image classification, object detection, and image segmentation, etc. The model is fine-tuned according to the requirements of the specific task to achieve the best performance.

[0064] The beneficial effects of the present invention are mainly reflected in the following aspects:

[0065] 1. Enhanced spatial structure understanding and improved model performance. The improved visual large model provided by the present invention enables the model encoder part to better understand the spatial structure of the image by introducing the position prediction task, thereby showing better performance in tasks such as object detection and image segmentation. Since the position prediction task forces the model to learn the position of each slice in the image, this helps the model capture the position and layout information of objects in the image, which is very important for understanding the spatial relationship between objects and scenes in the image, thereby improving the performance of the model on specific tasks.

[0066] 2. Optimized training efficiency. The introduction of the dynamic masking strategy and the multi-stage training strategy in the present invention enables the model to use data more effectively during the training process, reduces the training time and the consumption of computing resources, and improves the training efficiency. The dynamic masking strategy can ensure that the model is exposed to more diverse image areas during the training process, while the multi-stage training strategy can gradually adjust the training parameters according to the learning progress of the model, thereby improving the training efficiency.

[0067] In general, the beneficial effects of the present invention mainly come from the introduction of the position prediction task, the composite loss function combining the reconstruction loss and the position prediction loss, and the optimization of the training strategy. These improvements enable the model to learn richer image representations during pre-training, improve the generalization ability and task-specific performance of the model, and optimize the training efficiency and the simplicity of downstream task fine-tuning.

[0068] Embodiment 2

[0069] like Figure 3 As shown, another aspect of the present invention also includes a functional module architecture that corresponds to the aforementioned method flow, that is, the embodiment of the present invention also provides a pre-training device for a large visual model, including: a data acquisition module 301, used to acquire an unlabeled image and perform segmentation and masked segmentation processing to obtain a masked processed image, wherein the masked processed image includes an unmasked image segment and a masked image segment; a pre-training module 302, used to perform a total loss on the masked processed image; To optimize the goal, the improved visual big model is pre-trained.

[0070] The total loss Calculate according to the following formula:

[0071] ;

[0072] ;

[0073] ;

[0074] in, is the total loss, To rebuild the losses, is the position prediction loss, It is used for balance and is a hyperparameter, N is the number of unmasked image slices, are the predicted normalized coordinates of the i-th unmasked image patch, is the true normalized coordinate of the i-th unmasked image patch;

[0075] The improved visual big model is obtained by introducing a position prediction module at the output position of the encoder on the basis of a standard visual big model including an encoder and a decoder; the standard visual big model is used to reconstruct the obscured image slice given the specific position of the unobstructed image slice, and the position prediction module is used to predict the position of the unobstructed image slice in the image.

[0076] Furthermore, the position prediction module is composed of a feed-forward layer FFN x and FFN yComposition, feed-forward layer FFN x and FFN y The representation h of the i-th unmasked image slice output by the encoder is i As input, after linear layer transformation, the activation function is used to generate the predicted horizontal and vertical coordinates of the i-th unmasked image segment.

[0077] Furthermore, the standard visual large model adopts the MAE model.

[0078] The device can be implemented through the pre-training method of the large visual model provided in the above-mentioned embodiment 1. The specific implementation method can be found in the description of embodiment 1 and will not be repeated here.

[0079] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the pre-training method of the large visual model as described in the first embodiment.

[0080] The present invention also provides an electronic device, comprising a processor and a memory connected to the processor, wherein the memory stores a plurality of instructions, and the instructions can be loaded and executed by the processor so that the processor can execute the pre-training method of the large visual model as described in Example 1.

[0081] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A pre-training method for a large visual model, characterized in that: include: Acquire an unlabeled image and perform slice and mask slice processing to obtain a mask-processed image, wherein the mask-processed image includes unmasked image slices and masked image slices; Based on the masking process, the total loss To optimize the goal, the improved visual large model is pre-trained; The total loss Calculate according to the following formula: ； ； ； in, is the total loss, To rebuild the losses, is the position prediction loss, It is used for balance and is a hyperparameter, N is the number of unmasked image slices, are the predicted normalized coordinates of the i-th unmasked image patch, is the true normalized coordinate of the i-th unmasked image patch; The improved visual macro model is obtained by introducing a position prediction module at the output position of the encoder on the basis of a standard visual macro model including an encoder and a decoder; the standard visual macro model is used to reconstruct the obscured image slice under the premise of a specific position of the unobstructed image slice, and the position prediction module is used to predict the position of the unobstructed image slice in the image; The standard visual model adopts the MAE model.

2. The method for pre-training a large visual model as claimed in claim 1, characterized in that: The position prediction module consists of a feed-forward layer FFN x and FFN y Composition, feed-forward layer FFN x and FFN y The representation h of the i-th unmasked image slice output by the encoder is i As input, after linear layer transformation, the activation function is used to generate the predicted horizontal and vertical coordinates of the i-th unmasked image segment.

3. The pre-training method of a large visual model as claimed in claim 2, characterized in that: The horizontal coordinate and the vertical coordinate of the predicted i-th unmasked image slice are calculated according to the following formula: ； in, is the representation of the i-th unmasked image slice after being encoded by the encoder, FFN is the feed-forward layer x The mapping matrix, FFN is the feed-forward layer y The mapping matrix, is the sigmoid activation function, is the normalized horizontal coordinate of the predicted i-th unmasked image patch, is the normalized ordinate of the predicted i-th unoccluded image patch.

4. The method for pre-training a large visual model as claimed in claim 1, characterized in that: The reconstruction loss is calculated according to the following formula: ； in, represents the predicted value of the pixel with coordinates (j, k) in the tth masked image slice, for The corresponding actual values ​​are, M is the size of the tth masked image slice, and L is the number of masked image slices.

5. A pre-training device for a large visual model, characterized in that: include: A data acquisition module is used to acquire an unlabeled image and perform slice processing and masked slice processing to obtain a masked processed image, wherein the masked processed image includes unmasked image slices and masked image slices; Pre-trained module for processing images based on masking, with a total loss To optimize the goal, the improved visual large model is pre-trained; The total loss Calculate according to the following formula: ； ； ； in, is the total loss, To rebuild the losses, is the position prediction loss, It is used for balance and is a hyperparameter, N is the number of unmasked image slices, are the predicted normalized coordinates of the i-th unmasked image patch, is the true normalized coordinate of the i-th unmasked image patch; The improved visual macro model is obtained by introducing a position prediction module at the output position of the encoder on the basis of a standard visual macro model including an encoder and a decoder; the standard visual macro model is used to reconstruct the obscured image slices given the specific positions of the unobstructed image slices, and the position prediction module is used to predict the positions of the unobstructed image slices in the image; The standard visual model adopts the MAE model.

6. The pre-training device for a large visual model as claimed in claim 5, characterized in that: The position prediction module consists of a feed-forward layer FFN x and FFN y Composition, feed-forward layer FFN x and FFN y The representation h of the i-th unmasked image slice output by the encoder is i As input, after linear layer transformation, the activation function is used to generate the predicted horizontal and vertical coordinates of the i-th unmasked image segment.

7. A memory, characterized in that: Multiple instructions are stored, and the instructions are used to implement the pre-training method of the large visual model as described in any one of claims 1-4.

8. An electronic device, characterized in that: It includes a processor and a memory connected to the processor, wherein the memory stores a plurality of instructions, and the instructions can be loaded and executed by the processor so that the processor can execute the pre-training method of the large visual model as described in any one of claims 1-4.

Images