Model training method and device, electronic equipment and storage medium

Abstract

Embodiments of the application disclose a model training method and device, electronic equipment and a storage medium. The method comprises: dividing a training image into blocks to obtain a plurality of first image blocks; occluding the plurality of first image blocks to obtain a plurality of second image blocks, wherein the plurality of second image blocks are first image blocks that are not occluded in the plurality of first image blocks; inputting a feature vector of each second image block into the encoding network for encoding to obtain a plurality of encoded features corresponding to the plurality of network blocks, wherein the plurality of network blocks correspond one-to-one to the plurality of encoded features; inputting each encoded feature into a decoding network corresponding to each encoded feature for image reconstruction to obtain a reconstructed image corresponding to each decoding network; and training the model according to the reconstructed image corresponding to each decoding network and supervision information corresponding to each decoding network. The embodiments of the application are beneficial to improving the model training efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a model training method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In recent years, self-supervised visual representation learning has been widely concerned and rapidly developed. The research value lies in that it can learn generalizable representations from almost free massive unlabeled data to improve the representation of various downstream tasks, such as classification, detection and segmentation, etc. Because this learning method does not require high labeling cost, it has broad application prospects.

[0003] With the emergence and rapid development of visual self-attention networks, mask image modeling has attracted more attention. Self-supervised visual representation learning also uses a mask image modeling method based on a "mask-reconstruction" proxy task to train the model. When using this method for self-supervised visual representation learning, first, a part of the input image is randomly occluded, then the image is reconstructed based on the unoccluded part, the occluded part is predicted, and finally, self-supervised visual representation learning is performed based on the predicted occluded part and the real occluded part. However, this "mask-reconstruction" self-supervised visual representation learning method mainly uses the last output feature of the encoder for image reconstruction, and the entire learning process is slow. SUMMARY

[0004] The present application provides a model training method and device, electronic equipment and a storage medium, which uses different scale feature vectors for image reconstruction to guide self-supervised visual representation learning, and improves learning efficiency and model training efficiency.

[0005] In a first aspect, the present application provides a model training method, the model comprising an encoding network and a plurality of decoding networks, wherein the encoding network comprises a plurality of network blocks with different depths, wherein each network block corresponds to one of the plurality of decoding networks, and each network block comprises one or more network layers; the method comprising: dividing a training image into blocks to obtain a plurality of first image blocks; occluding the plurality of first image blocks to obtain a plurality of second image blocks, wherein the plurality of second image blocks are first image blocks in the plurality of first image blocks that are not occluded; inputting a feature vector of each second image block into the encoding network for encoding to obtain a plurality of encoded features corresponding to the plurality of network blocks, wherein the plurality of network blocks correspond one-to-one to the plurality of encoded features; inputting each encoded feature into the decoding network corresponding to each encoded feature for image reconstruction to obtain a reconstructed image corresponding to each decoding network; and training the model according to the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network.

[0006] It can be seen that, in the embodiment of the present application, when self-supervised visual representation learning is performed, a corresponding decoding network is designed at each network layer of different depths of the encoding network, that is, multiple decoding networks are designed; then, image reconstruction is performed using each decoding network to obtain the reconstruction of each decoding network, so that the output features of the network layers of different depths of the decoding network are used for image reconstruction (that is, the low-level features and high-level features output by the encoding network are used for image reconstruction); finally, the model is trained based on the reconstructed image of each decoding network and the supervision information, so that the features of different scales output by the encoding network are used for self-supervised visual representation learning in the present application, not only the top-level features (the features output by the last network layer) are used, so that more detailed features can be used for learning, the learning efficiency and the model training efficiency are improved, and the recognition accuracy of the trained model is high.

[0007] In an embodiment of the present application, the occlusion of the plurality of first image blocks to obtain a plurality of second image blocks comprises: generating an index value corresponding to each first image block, the index value of each first image block being used to indicate whether each first image block is occluded; and occluding the plurality of first image blocks according to the index value of each first image block to obtain a plurality of second image blocks.

[0008] It can be seen that, in the embodiment of the present application, the plurality of first image blocks are occluded by randomly constructing the index value, and the redundancy is largely eliminated by random occlusion, which generates a task that cannot be easily solved by extrapolation from visible adjacent patches, thereby avoiding the model from converging due to special cases and improving the accuracy of model training.

[0009] In an embodiment of the present application, before training the model according to the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network, the method further comprises: obtaining a blocking scale corresponding to each decoding network, wherein the deeper the network block corresponding to the decoding network, the smaller the blocking scale corresponding to the decoding network; blocking the training image according to the blocking scale corresponding to each decoding network to obtain a plurality of third image blocks corresponding to each decoding network; and taking the plurality of third image blocks corresponding to each decoding network as the supervision information of each decoding network.

[0010] It can be seen that, in the embodiment of the present application, the supervision information corresponding to the depth of each decoding network is constructed for each decoding network, that is, the model is trained using multi-scale supervision information, and the shallower the decoding network, the finer the scale of the supervision information corresponding to the decoding network, so that the feature information output by the low-level encoding network layer can be better captured, the lower-level encoding network can be better trained, and the efficiency of model training and the recognition accuracy of the model are further improved.

[0011] In an embodiment of the present application, the reconstructed image corresponding to each decoding network includes a plurality of fourth image blocks; and the training of the model according to the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network includes: determining, according to the plurality of third image blocks in the supervision information corresponding to each decoding network, a third image block corresponding to each fourth image block in the plurality of fourth image blocks corresponding to each decoding network; determining a loss of each decoding network according to each fourth image block corresponding to each decoding network and the third image block corresponding to the fourth image block; and training the model according to the loss of each decoding network.

[0012] It can be seen that, in the embodiment of the present application, after the corresponding supervision information is constructed for each decoding, the loss of each decoding network is determined based on the supervision information of each decoding network, so that the loss of each decoding network is more consistent with the real loss, and the training accuracy of the model is improved.

[0013] In an embodiment of the present application, before determining the loss of each decoding network according to each fourth image block corresponding to each decoding network and the third image block corresponding to the fourth image block, the method further includes: generating a first index value vector according to the index value of each first image block; inputting the first index value vector into the model to obtain a second index value vector corresponding to each decoding network; and determining the loss of each decoding network according to the fourth image block corresponding to each decoding network and the third image block corresponding to the fourth image block includes: determining the index value of each fourth image block corresponding to each decoding network according to the second index value vector corresponding to each decoding network, wherein the index value of each fourth image block is used to indicate whether the real content corresponding to each fourth image block is occluded; and determining the loss corresponding to each decoding network according to each fourth image block corresponding to each decoding network, the index value corresponding to the fourth image block and the third image block corresponding to the fourth image block.

[0014] It can be seen that, in the embodiment of the present application, during the model training process, the index value of the first image block is synchronously transmitted into the model for upsampling and / or downsampling, so that the index value of each image block in the reconstructed image can be determined, that is, which image blocks in the reconstructed image are occluded, so that only the occluded image blocks are used for loss calculation, the efficiency and accuracy of loss calculation are improved, and the model training accuracy and efficiency are further improved.

[0015] In an embodiment of the present application, the loss corresponding to each decoding network is determined based on the feature information of each fourth image block corresponding to each decoding network and the feature information of the third image block corresponding to the fourth image block; wherein the feature information of the image block A is one of the following: the pixel value of each pixel point in the image block A; or the direction gradient histogram of the image block A; or the normalized pixel value of the image block A; the image block A is the fourth image block corresponding to each decoding network or the third image block corresponding to the fourth image block.

[0016] It can be seen that in the embodiments of the present application, various morphological features of image blocks can be used as supervision information to improve the diversity of loss calculation and the diversity of model training.

[0017] In an embodiment of the present application, the method further comprises: after completing the training of the model, using the encoding network as a backbone network of a downstream recognition task; wherein the downstream recognition task comprises one of the following: pedestrian attribute recognition, image segmentation, and image classification.

[0018] It can be seen that since the trained model in the present application has high recognition accuracy, using the encoding network of the present application as the backbone network of the downstream recognition task can improve the recognition accuracy of the downstream recognition task.

[0019] In a second aspect, the embodiments of the present application provide a model training device, the model comprising: an encoding network and a plurality of decoding networks, wherein the encoding network comprises a plurality of network blocks with different depths, wherein each network block corresponds to one of the plurality of decoding networks, and each network block comprises one or more network layers; the model training device comprises: an acquisition unit and a processing unit; the acquisition unit is configured to acquire a training image; the processing unit is configured to divide the training image into a plurality of first image blocks; the plurality of first image blocks are occluded to obtain a plurality of second image blocks, wherein the plurality of second image blocks are the first image blocks in the plurality of first image blocks that are not occluded; a feature vector of each second image block is input to the encoding network for encoding to obtain a plurality of encoding features corresponding to the plurality of network blocks, wherein the plurality of network blocks correspond one-to-one to the plurality of encoding features; each encoding feature is input to the decoding network corresponding to the encoding feature for image reconstruction to obtain a reconstructed image corresponding to each decoding network; and the model is trained according to the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network.

[0020] In an embodiment of the present application, in terms of occluding the plurality of first image blocks to obtain a plurality of second image blocks, the processing unit is specifically configured to: generate an index value corresponding to each first image block, the index value of each first image block being used to indicate whether each first image block is occluded; and occlude the plurality of first image blocks according to the index value of each first image block to obtain the plurality of second image blocks.

[0021] In an embodiment of the present application, before the processing unit trains the model according to the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network, the processing unit is further configured to: obtain a block size corresponding to each decoding network, wherein the deeper the network block corresponding to the decoding network is, the smaller the block size corresponding to the decoding network is; and block the training image according to the block size corresponding to each decoding network to obtain a plurality of third image blocks corresponding to each decoding network; and take the plurality of third image blocks corresponding to each decoding network as the supervision information of each decoding network.

[0022] In an embodiment of the present application, the reconstructed image corresponding to each decoding network includes a plurality of fourth image blocks; and in terms of training the model according to the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network, the processing unit is specifically configured to: determine a third image block corresponding to each fourth image block in the plurality of fourth image blocks corresponding to each decoding network according to the plurality of third image blocks in the supervision information of each decoding network; determine the loss of each decoding network according to each fourth image block corresponding to each decoding network and the third image block corresponding to the fourth image block; and train the model according to the loss of each decoding network.

[0023] In an embodiment of the present application, before the processing unit determines the loss of each decoding network according to each fourth image block corresponding to each decoding network and the third image block corresponding to the fourth image block, the processing unit is further configured to: generate a first index value vector according to the index value of each first image block; input the first index value vector into the model to obtain a second index value vector corresponding to each decoding network; and in terms of determining the loss of each decoding network according to the fourth image block corresponding to each decoding network and the third image block corresponding to the fourth image block, the processing unit is specifically configured to: determine the index value of each fourth image block corresponding to each decoding network according to the second index value vector corresponding to each decoding network, wherein the index value of each fourth image block is used to indicate whether the real content corresponding to each fourth image block is occluded; and determine the loss corresponding to each decoding network according to each fourth image block corresponding to each decoding network, the index value corresponding to the fourth image block and the third image block corresponding to the fourth image block.

[0024] In an embodiment of the present application, the loss corresponding to each decoding network is determined based on the feature information of each fourth image block corresponding to each decoding network and the feature information of the third image block corresponding to the fourth image block; wherein the feature information of the image block A is one of the following: the pixel value of each pixel point in the image block A; or the histogram of the gradient direction of the image block A; or the normalized pixel value of the image block A; the image block A is the fourth image block corresponding to each decoding network or the third image block corresponding to the fourth image block.

[0025] In an embodiment of the present application, the processing unit is further configured to: after completing the training of the model, use the encoding network as a backbone network of a downstream recognition task; wherein the downstream recognition task includes one of the following: pedestrian attribute recognition, image segmentation, and image classification.

[0026] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory configured to store a program; a processor configured to execute the program stored in the memory; and when the program stored in the memory is executed, the processor is configured to implement the method in the first aspect.

[0027] In a fourth aspect, an embodiment of the present application provides a computer readable medium storing program code for execution by a device, the program code including code for implementing the method in the first aspect.

[0028] In a fifth aspect, an embodiment of the present application provides a computer program product containing instructions which, when the computer program product is executed on a computer, cause the computer to implement the method in the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 A schematic diagram of image reconstruction is provided for an embodiment of the present application;

[0030] Figure 2 A structural schematic diagram of a model is provided for an embodiment of the present application;

[0031] Figure 3 A flowchart of a model training method is provided for an embodiment of the present application;

[0032] Figure 4 A schematic diagram of blocking and occlusion of training images is provided for an embodiment of the present application;

[0033] Figure 5 A schematic diagram of constructing supervision information of different scales is provided for an embodiment of the present application;

[0034] Figure 6 A schematic diagram of constructing an index value matrix is provided for an embodiment of the present application;

[0035] Figure 7 A schematic diagram of model training provided for an embodiment of the present application;

[0036] Figure 8 A schematic diagram of a model training device provided for an embodiment of the present application;

[0037] Figure 9 A schematic diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to facilitate the understanding of the present application, first introduce the technical knowledge related to the present application.

[0039] As Figure 1 shown, at present, when self-supervised visual representation learning is carried out, the input image is first blocked to obtain a plurality of image blocks. Then, the plurality of image blocks are randomly occluded, and the feature extraction is carried out on the unoccluded image blocks, and the extracted features are input into the encoder of the model for encoding to obtain the encoding result, i.e. the output of the last layer of the encoder is taken as the encoding result; for the occluded image blocks, a placeholder vector is added in the encoding result to obtain the input data of the decoder, and the input data is input into the decoder for image reconstruction to obtain the predicted image. Finally, the loss is calculated based on the predicted image and the input image, and the model is trained based on the loss, until the model converges, the self-supervised visual representation learning is completed.

[0040] However, the above-mentioned self-supervised visual representation learning method mainly uses the top layer output of the encoder for image reconstruction, i.e. uses the output of the last network layer of the encoder for image reconstruction, so that only the features extracted by the deep network layers of the model are used when self-supervised visual representation learning is carried out, which causes the learning process of the low layer to be relatively slow, and further causes the self-supervised visual representation learning process to be relatively slow and low in efficiency. Therefore, how to improve the efficiency of self-supervised visual representation learning is a technical problem to be solved at present.

[0041] Firstly, it is explained that the encoding network involved in the present application can also be understood as an encoder, and the decoding network can also be understood as a decoder.

[0042] Referring to Figure 2 , Figure 2 A structural schematic diagram of a model provided for an embodiment of the present application. As Figure 2 shown, the model includes an encoding network and a plurality of decoding networks, wherein the encoding network includes a plurality of network blocks, and the depths of the plurality of network blocks are different, such as Figure 2 network block 1, network block 2…network block N, and decoding network 1, decoding network 2…decoding network N as shown.

[0043] Optionally, each network block includes one or more network layers in the decoding network, i.e., one or more network layers in the decoding network are taken as a network block. Further, each network block coding process can be referred to as a stage, i.e., the above-mentioned multiple network blocks split the entire coding process of the coding network into multiple stages (such as stage 1, stage 2, …, stage N as shown). Figure 2 The last network layer of each stage is connected with a decoding network.

[0044] Optionally, the above-mentioned multiple network blocks correspond to one decoding network in the multiple decoding networks, i.e., the last network layer of each network block is connected with the decoding network corresponding to the network block. Figure 2

[0045] For example, the above-mentioned coding network can be various types of coding networks, such as Vit coding network or Swin coding network. It should be noted that if the adopted coding network is already divided into stages, i.e., the coding network itself contains multiple network blocks, such as Swin coding network, the network blocks divided by itself can be used as the above-mentioned multiple network blocks; if the above-mentioned coding network is not divided into stages, such as Vit coding network, the multiple network layers of the coding network can be divided into multiple network blocks first.

[0046] Referring to Figure 3 , Figure 3 A flowchart of a model training method provided by an embodiment of the present application is shown. The model is a model as shown. Figure 2 The method includes but is not limited to the following steps:

[0047] 301: The training image is divided into multiple first image blocks.

[0048] Optionally, the training image is evenly divided into multiple first image blocks without overlapping, as shown. Figure 4

[0049] 302: The multiple first image blocks are occluded to obtain multiple second image blocks, wherein the multiple second image blocks are the first image blocks in the multiple first image blocks that are not occluded.

[0050] For example, the multiple first image blocks are randomly occluded to obtain the multiple second image blocks, as shown. Figure 4 In order to distinguish, the first image blocks in the multiple first image blocks that are not occluded are referred to as second image blocks in the present application.

[0051] ​​Optionally, an index value corresponding to each first image block is generated, wherein the index value of each first image block is used to indicate whether each first image block is occluded. Optionally, the index value of each first image block is randomly generated in 0 and 1, for example, when the index value is 0, it indicates that the first image block is not occluded, and when the index value is 1, it indicates that the first image block is occluded. Based on the index value corresponding to each first image block, the plurality of first image blocks are occluded to obtain a plurality of second image blocks.

[0052] 303: input the feature vector of each second image block into the encoding network for encoding to obtain a plurality of encoding features corresponding to the plurality of network blocks, wherein the plurality of network blocks correspond one-to-one to the plurality of encoding features.

[0053] Optionally, after the training image is blocked, the embedding of each first image block is performed to obtain an embedding vector of each first image block; then, the position encoding is added to the embedding vector of each first image block to obtain a feature vector of each first image block, so that after the plurality of first image blocks are occluded, the feature vector of each second image block can be directly input into the encoding network for encoding, that is, only the feature vector of the first image block which is not occluded is input.

[0054] Optionally, after the training image is blocked, instead of directly embedding each first image block, the plurality of first image blocks are first occluded; then, the embedding of each second image block is performed to obtain an embedding vector of each second image block, and the position encoding is added to the embedding vector of each second image block to obtain a feature vector of each second image block; finally, the feature vector of each second image block is input into the encoding network for encoding.

[0055] Optionally, the feature vector of each second image block is input into the encoding network for encoding, and the feature vector is encoded by a plurality of network blocks in the encoding network to obtain a plurality of encoding features corresponding to the plurality of network blocks, that is, the output of the last network layer in each network block is taken as the encoding feature corresponding to each network block.

[0056] 304: input each encoding feature into a decoding network corresponding to each encoding feature for image reconstruction to obtain a reconstructed image corresponding to each decoding network.

[0057] Optionally, the encoding feature output by each network block is input into a decoding network corresponding to the network block for image reconstruction to obtain a reconstructed image corresponding to each decoding network.

[0058] Specifically, a placeholder vector is added to the encoding feature corresponding to each decoding network (i.e., the sequence of feature vectors), to generate a new encoding feature (i.e., a new feature vector), wherein the placeholder vector is pre-learned and is a shared learning vector, used to represent that the position corresponding to the placeholder vector has an occluded image block to be predicted. Then, a position encoding is added to the new encoding feature corresponding to each decoding network to obtain input data corresponding to each decoding network; finally, the input data corresponding to each decoding network is input into each decoding network for image reconstruction to obtain a reconstructed image corresponding to each decoding network.

[0059] 305: training the model according to the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network.

[0060] For example, based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network, the loss corresponding to each decoding network is determined; and the model is trained according to the loss corresponding to each decoding network.

[0061] For example, since the decoding network reconstructs the training image, and the training image is segmented into a plurality of first image blocks and input into the model for image reconstruction, the reconstruction of the training image can also be understood as the reconstruction of the image block, and the reconstructed image corresponding to each decoding network includes a plurality of fourth image blocks. Correspondingly, the supervision information corresponding to each decoding network is the supervision information corresponding to the plurality of fourth image blocks of each decoding network. Therefore, according to each fourth image block in the reconstructed image corresponding to each decoding network and the supervision information corresponding to each fourth image block, the loss corresponding to each decoding network is determined.

[0062] The process of obtaining the supervision information corresponding to each decoding network is described below.

[0063] Scenario 1: The scales of the supervision information of the decoding networks are different.

[0064] For example, when the scales of the supervision information of the decoding networks are different, the block size corresponding to each decoding network is obtained, wherein the deeper the network block corresponding to the decoding network, the smaller the block size corresponding to the decoding network. For example, as shown in FIG. 3, the block size of decoding network 1 is smaller than the block size of decoding network 2. Figure 2 According to the block size corresponding to each decoding network, the training image is segmented to obtain a plurality of third image blocks corresponding to each decoding network, i.e., using the block size corresponding to each decoding network, the training image is uniformly and non-overlappingly segmented to obtain a plurality of third image blocks corresponding to each decoding network, and the plurality of third image blocks corresponding to each decoding network are used as the supervision information of each decoding network. For example, as shown in FIG. 4, the training image is segmented into a plurality of third image blocks corresponding to decoding network 1 and a plurality of third image blocks corresponding to decoding network 2. Figure 5As shown, the training image is divided into third image blocks corresponding to decoding network 1, decoding network 2, …, decoding network N through block scale 1, block scale 2, …, block scale N corresponding to decoding network 1, decoding network 2, …, decoding network N, to obtain the supervision information corresponding to decoding network 1, decoding network 2, …, decoding network N. After the training image is divided into blocks through the block scale of each decoding network, the deeper the decoding network, the larger the size of the third image block corresponding to the decoding network, and the fewer the number of third image blocks.

[0065] In addition, the plurality of fourth image blocks in the reconstructed image of each decoding network one-to-one correspond to the plurality of third image blocks corresponding to each decoding network, so that each fourth image block corresponding to the plurality of third image blocks in the plurality of fourth image blocks corresponding to each decoding network can be determined according to the supervision information of each decoding network (i.e., the supervision information of each fourth image block is determined); then according to each fourth image block corresponding to each decoding network and the third image block corresponding to the fourth image block, the loss of each decoding network is determined, that is, the loss between each fourth image block corresponding to each decoding network and the third image block corresponding to the fourth image block is integrated to obtain the loss of each decoding network, wherein the loss between each fourth image block and the third image block corresponding to the fourth image block can be the cross-entropy loss between the two image blocks.

[0066] It can be seen that different scales of supervision information are designed for each decoding network, and in particular, the shallower the decoding network, the finer the granularity of the supervision information, so that finer scale feature information can be used for loss calculation, thereby making the accuracy of the calculated loss of each decoding network relatively high, and further improving the training efficiency and accuracy of the model.

[0067] It should be noted that when the scales of the supervision information of the decoding networks are designed to be different, the sizes of the plurality of fourth image blocks in the reconstructed image output by each decoding network are designed to be the same as the sizes of the plurality of third image blocks. It should be noted that since the input data for image reconstruction by each decoding network comes from the decoding network, that is, the number of blocks of the reconstructed image depends on the number of blocks of the training image, in order to make the size of the fourth image block the same as the size of the third image block, the decoding network needs to perform corresponding up-sampling or down-sampling when performing image reconstruction according to the size of the third image block corresponding to the decoding network, so that the size of the reconstructed fourth image block is the same as the size of the third image block. For example, the size of the training image is 20*20, the size of the first image block is 10*10, if the decoding network is not designed to have an up-sampling or down-sampling process, then the block size of the reconstructed image is also 10*10, but if the size of the third image block corresponding to the decoding network is 5*5, then up-sampling needs to be performed when performing image reconstruction to make the block size of the reconstructed image 5*5.

[0068] In practical applications, the model can be trained by calculating only the loss between the occluded content and the predicted content in the training image. Therefore, for each decoding network, a target fourth image patch can be identified among multiple fourth image patches corresponding to each decoding network. The content corresponding to this target fourth image patch is occluded, meaning the actual content of the target fourth image patch in the training image is occluded. The loss between the target fourth image patch corresponding to each decoding network and the corresponding third image patch is determined, resulting in the loss for each decoding network. Finally, the model is trained based on the loss of each decoding network.

[0069] In one embodiment of this application, a target fourth image block among a plurality of fourth image blocks corresponding to each decoding network can be determined based on an index value. For example, the index value of each fourth image block corresponding to each decoding network is obtained. The index value of each fourth image block indicates whether the actual content corresponding to that fourth image block is occluded. For instance, an index value of 0 indicates that the actual content corresponding to that fourth image block is not occluded, while an index value of 1 indicates that the actual content corresponding to that fourth image block is occluded. The target fourth image block among a plurality of fourth image blocks corresponding to each decoding network is determined based on the index value of each fourth image block.

[0070] Specifically, such as Figure 6As shown, after generating the index value corresponding to each first image patch, a first index value matrix can be generated based on the index value corresponding to each first image patch. This first index value matrix is ​​a square matrix, and the product of its length and width is the number of first image patches. Therefore, the index value in the i-th row and j-th column of the first index value matrix is ​​the index value of the first image patch in the i-th row and j-th column among the multiple first image patches. Then, the first index value matrix is ​​tiled to obtain a first index value vector. Then, when inputting the feature vector of the second image patch, this first index value vector is simultaneously input into the model, i.e., encoded (downsampled) by the decoding network and decoded (downsampled) by the decoding network to obtain a second index value vector corresponding to each decoding network. Then, the second index value vector is reshaped to obtain a second index value matrix corresponding to each decoding network, where the second index value matrix is ​​a square matrix, and the product of its length and width is the number of fourth image patches. Accordingly, the element in the i-th row and j-th column of the second index value matrix corresponding to each decoding network is the index value of the i-th row and j-th column fourth image block among the multiple fourth image blocks corresponding to each decoding network. Then, based on the second index value matrix, the index value corresponding to each fourth image block corresponding to each decoding network is determined. Based on the index value corresponding to each fourth image block corresponding to each decoding network, the target fourth image block among the multiple fourth image blocks corresponding to each decoding network can be determined.

[0071] In one embodiment of this application, after determining the index value of each fourth image block, the loss corresponding to each decoding network can also be determined based on each fourth image block corresponding to each decoding network, the index value corresponding to the fourth image block, and the third image block corresponding to the fourth image block.

[0072] For example, when using index values, the loss for each decoding network can be expressed by formula (1):

[0073]

[0074] in, Let be the loss of the l-th decoding network, and let i represent the i-th fourth image block among the multiple fourth image blocks corresponding to the l-th decoding network. This is the index value corresponding to the i-th fourth image patch. This refers to the supervision information corresponding to the i-th fourth image block, which is the third image block corresponding to the i-th fourth image block. For the i-th fourth image patch, Let represent the cross-entropy loss between the i-th fourth image block and the supervision information corresponding to the i-th fourth image block.

[0075] As can be seen, by using the index value of each fourth image block, only the fourth image block with an index value of 1 (i.e., the target fourth image block mentioned above) participates in the loss calculation. This means that only the loss between the occluded content and the predicted content is calculated, thereby speeding up the loss calculation and improving the computational efficiency.

[0076] In one embodiment of this application, the loss of each decoding network can be determined based on the feature information of each fourth image block corresponding to each decoding network and the loss between the feature information of the third image block corresponding to the fourth image block. The feature information of each fourth image block can be the pixel value of each pixel in the fourth image block, and correspondingly, the feature information of the third image block corresponding to the fourth image block is also the pixel value of each pixel in the third image block. That is, calculating the cross-entropy loss between the pixel values ​​of the fourth and third image blocks yields the loss of each decoding network. Alternatively, the feature information of the fourth image block can be its directional gradient histogram, and correspondingly, the feature information of the third image block corresponding to the fourth image block is also its directional gradient histogram. Calculating the cross-entropy loss between the directional gradient histograms of the fourth and third image blocks yields the loss of each decoding network. Alternatively, the feature information of the fourth image block can be its normalized pixel value, and correspondingly, the feature information of the third image block is also its normalized pixel value. This application does not limit the feature information of the image blocks.

[0077] Scenario 2: The scale of the supervision information of each decoding network is the same.

[0078] Optionally, the scale of the supervision information of each decoding network is the same, that is, the size of the supervision information corresponding to the fourth image block reconstructed by each decoding network is the same.

[0079] For example, in this application, the segmented training images can be used as supervision information for each decoding network, that is, multiple first image blocks can be used as supervision information for each decoding network. Of course, in practical applications, a new segmentation scale can also be used to segment the training images into multiple fifth image blocks, and then these multiple fifth image blocks can be used as supervision information for each decoding network. In this application, the use of multiple first image blocks as supervision information for each decoding network is used as an example for illustration.

[0080] Then, obtain the first image block (supervision information) corresponding to each fourth image block of each decoding network; based on each fourth image block corresponding to each decoding network and the first image block corresponding to that fourth image block, determine the loss of each decoding network, that is, determine the loss between each fourth image block and the first image block of each decoding network, and obtain the loss of each decoding network. Similarly, the loss of each decoding network can be determined based on the index value of each fourth image block of each decoding network, except that the supervision information of each fourth image block is changed from the third image block to the first image block, as shown in the above formula (1), which will not be described again.

[0081] It should be noted that if the encoding network changes the size of the input data, for example, by downsampling the input data, so that the size of the input data of each decoding network is different, then in order to make the size of the fourth image block reconstructed by each decoding network the same, the decoding network will perform corresponding upsampling so that the size of the fourth image block reconstructed by each decoding network is the same and the same as the size of the first image block.

[0082] In one embodiment of this application, after obtaining the loss of each decoding network, the model can be trained as a whole based on the loss of each decoding network. For example, after obtaining the loss of each decoding network, the losses of multiple decoding networks are weighted according to the weight of each decoding network to obtain the target loss; the model is trained based on the target loss until the model converges, thus completing the training of the model.

[0083] For example, the target loss can be expressed by formula (2):

[0084]

[0085] Among them, L LaMSR Let w be the target loss, l be the l-th network layer of the model, and w be the loss function. l Let the weights be those of the l-th network layer. This refers to the set of network layers consisting of the last network layer among the aforementioned network blocks, i.e. This represents a weighted summation of the losses from multiple decoding networks.

[0086] In one embodiment of this application, after obtaining the loss of each decoding network, the above model can be trained layer by layer based on the loss of each decoding network, i.e., greedy training.

[0087] The following explains why this application can perform layer-by-layer training.

[0088] First, with either the ViT-B or Swin-B decoding network, the model of this application (i.e., the Local Multi-Scale Reconfiguration (LaSMR) model) was trained using gradient truncation training, and the LaSMR model of this application was trained without gradient truncation training, i.e., the LaSMR model was trained as a whole. Then, the training time (i.e., the average time consumed by the image processor) and the model accuracy (acc) were obtained for both training modes. Table 1 shows the training time and accuracy for the two training modes.

[0089] Table 1:

[0090]

[0091] As shown in Table 1, when the encoding network is ViT-B, the training time required for training the LaSMR model using gradient truncation is the same as that without gradient truncation, and the achieved accuracy is also the same. When the encoding network is Swin-B, the training time required for training the LaSMR model using gradient truncation is the same as that without gradient truncation, and the accuracy is also very close. Table 1 shows that gradient truncation training of the model in this application does not increase the training time or decrease the accuracy. In other words, the model in this application can be trained using gradient truncation, therefore, the model in this application supports layer training.

[0092] See Table 2, which shows the training results when training the model of this application layer by layer.

[0093] Table 2:

[0094]

[0095] Based on the content shown in Table 2, it can be seen that by training the model of this application layer by layer, memory overhead and pre-training time can be significantly reduced. Layer-by-layer training brings a speedup of 30-40% and saves 40-55% of memory usage without reducing the accuracy of the model.

[0096] The training process of the model in this application is illustrated below using the Swin network (which consists of four stages) as an example.

[0097] like Figure 7As shown, the training image is first divided into image blocks, and these blocks are then occluded to obtain multiple unoccluded image blocks (i.e., the second image block of this application). Then, the features of the unoccluded images are input into the encoding network for encoding, resulting in encoded features corresponding to each stage of the encoding network, i.e., encoded features corresponding to each decoding network. Next, placeholder vectors of the occluded image blocks are added to the encoded features of each decoding network to obtain new encoded features corresponding to each decoding network, and positional encoding is added to these new encoded features. Figure 7 (Not shown in the image), the input data of each decoding network can be obtained; the input data of each decoding network is input into each decoding network to reconstruct the image, and the reconstructed image (including multiple reconstructed image blocks) corresponding to each decoding network is obtained.

[0098] Furthermore, this application constructs corresponding supervision information for each decoding network; that is, the scale of the supervision information for decoding networks of different depths in this application is different. Then, based on the supervision information of each decoding network and the image patches reconstructed by each decoding network, the loss corresponding to each decoding network is determined; finally, the model is trained based on the loss corresponding to each decoding network.

[0099] In one embodiment of this application, after training the model is completed, the encoding network is used as the backbone network for a downstream recognition task; wherein, the downstream recognition task includes one of the following: pedestrian attribute recognition, image segmentation, and image classification. Specifically, the trained encoding network is used as the backbone network for the downstream recognition task to extract features, so as to perform pedestrian attribute recognition, image segmentation, and image classification.

[0100] Specifically, in autonomous driving scenarios, it is necessary to segment the road images captured by cameras to separate different objects such as road surface, roadbed, vehicles, and pedestrians, thereby ensuring that the vehicle travels within the correct area. Using the coding network of this application as the backbone network for semantic feature extraction of the image can improve the accuracy of image segmentation. In intelligent monitoring systems, it is necessary to identify pedestrian attributes in the images captured by cameras, such as the pedestrian's gender, age, hair, clothing, and backpack. Using the coding network of this application as the backbone network for semantic feature extraction of the image can improve the accuracy of pedestrian attribute identification.

[0101] See Figure 8 , Figure 8 This is a schematic diagram of a model training device provided in an embodiment of this application. Figure 8 As shown, the model training device includes an acquisition unit 801 and a processing unit 802;

[0102] The acquisition unit 801 is used to acquire training images;

[0103] The processing unit 802 is used to divide the training image into blocks to obtain multiple first image blocks;

[0104] The plurality of first image blocks are occluded to obtain a plurality of second image blocks, wherein the plurality of second image blocks are the first image blocks among the plurality of first image blocks that are not occluded;

[0105] The feature vector of each second image block is input into the coding network for encoding to obtain multiple coding features corresponding to the multiple network blocks, wherein the multiple network blocks and the multiple coding features correspond one-to-one;

[0106] Each encoded feature is input into the corresponding decoding network to reconstruct the image, resulting in a reconstructed image for each decoding network.

[0107] The model is trained based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network.

[0108] For a more detailed description of the acquisition unit 801 and the processing unit 802, please refer to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0109] See Figure 9 , Figure 9 This is a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device 900 can be the model training device 800 described above. The electronic device 900 includes a memory 901, a processor 902, a communication interface 903, and a bus 904. The memory 901, processor 902, and communication interface 903 are interconnected via the bus 904.

[0110] The memory 901 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 901 can store programs; when the electronic device 900 is the model training device 800 described above, when the program stored in the memory 901 is executed by the processor 902, the processor 902 and the communication interface 903 are used to execute the various steps in the model training method of the embodiments of this application.

[0111] The processor 902 may be a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to implement the model training method of the embodiments of this application.

[0112] The processor 902 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the data stream transmission method of this application can be completed by the integrated logic circuits in the hardware of the processor 902 or by instructions in software form. The processor 902 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 901, and processor 902 reads the information in memory 901 to execute the various steps in the model training method.

[0113] The communication interface 903 can be a transceiver device such as a transceiver to enable communication between the electronic device 900 and other devices or communication networks. The communication interface 903 can also be an input-output interface to enable data transmission between the electronic device 900 and input-output devices, including but not limited to keyboards, mice, displays, USB flash drives, and hard drives. For example, the processor 902 can acquire voice samples through the communication interface 903.

[0114] Bus 904 may include a pathway for transmitting information between various components of device electronics 900 (e.g., memory 901, processor 902, communication interface 903).

[0115] It should be noted that, although Figure 9The illustrated electronic device 900 only shows the memory, processor, and communication interface. However, those skilled in the art should understand that in specific implementations, the electronic device 900 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the electronic device 900 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the electronic device 900 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 9 All the devices shown.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0117] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0118] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0119] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates an "or" relationship between the preceding and following related objects; in the formulas of this application, the character " / " indicates a "division" relationship between the preceding and following related objects.

[0120] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The order of the process numbers described above does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

[0121] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0122] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model training method, characterized in that, The model includes an encoding network and multiple decoding networks, wherein the encoding network comprises multiple network blocks of different depths, each network block corresponds to one of the multiple decoding networks, and each network block includes one or more network layers; the method includes: The training image is divided into blocks to obtain multiple first image blocks; The plurality of first image blocks are occluded to obtain a plurality of second image blocks, wherein the plurality of second image blocks are the first image blocks among the plurality of first image blocks that are not occluded; The feature vector of each second image block is input into the coding network for encoding to obtain multiple coding features corresponding to the multiple network blocks, wherein the multiple network blocks and the multiple coding features correspond one-to-one; Each encoded feature is input into the corresponding decoding network to reconstruct the image, resulting in a reconstructed image for each decoding network. The model is trained based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network.

2. The method according to claim 1, characterized in that, The process of occluding the plurality of first image blocks to obtain a plurality of second image blocks includes: Generate an index value corresponding to each first image block. The index value of each first image block is used to indicate whether each first image block is occluded. Based on the index value of each first image block, the plurality of first image blocks are occluded to obtain the plurality of second image blocks.

3. The method according to claim 2, characterized in that, Before training the model, based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network, the method further includes: Obtain the block scale corresponding to each decoding network, wherein the shallower the depth of the network block corresponding to the decoding network, the smaller the block scale corresponding to the decoding network; The training image is divided into blocks according to the block scale corresponding to each decoding network to obtain multiple third image blocks corresponding to each decoding network. Multiple third image blocks corresponding to each decoding network are used as supervision information for each decoding network.

4. The method according to claim 3, characterized in that, The reconstructed image corresponding to each decoding network includes multiple fourth image patches; the step of training the model based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network includes: Based on the multiple third image blocks in the supervision information of each decoding network, determine the third image block corresponding to each of the multiple fourth image blocks corresponding to each decoding network; The loss of each decoding network is determined based on each fourth image block corresponding to each decoding network and the third image block corresponding to that fourth image block; The model is trained based on the loss of each decoding network.

5. The method according to claim 4, characterized in that, Before determining the loss of each decoding network based on each fourth image block corresponding to each decoding network and the third image block corresponding to that fourth image block, the method further includes: Generate a first index value vector based on the index value of each first image patch; The first index value vector is input into the model to obtain the second index value vector corresponding to each decoding network; The step of determining the loss of each decoding network based on a fourth image block corresponding to each decoding network and a third image block corresponding to that fourth image block includes: Based on the second index value vector corresponding to each decoding network, determine the index value of each fourth image block corresponding to each decoding network, wherein the index value of each fourth image block is used to indicate whether the real content corresponding to each fourth image block is occluded; The loss for each decoding network is determined based on each fourth image block corresponding to each decoding network, the index value corresponding to that fourth image block, and the third image block corresponding to that fourth image block.

6. The method according to claim 4 or 5, characterized in that, The loss corresponding to each decoding network is determined based on the feature information of each fourth image block corresponding to each decoding network, and the feature information of the third image block corresponding to that fourth image block; The feature information of image block A is one of the following: The pixel value of each pixel in image block A; or, the directional gradient histogram of image block A; or, the normalized pixel value of image block A; image block A is the fourth image block corresponding to each decoding network or the third image block corresponding to the fourth image block.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: After training the model is completed, the encoding network is used as the backbone network for downstream recognition tasks. The downstream identification tasks include one of the following: pedestrian attribute recognition, image segmentation, and image classification.

8. A model training device, characterized in that, The model includes an encoding network and multiple decoding networks, wherein the encoding network includes multiple network blocks of different depths, each network block corresponds to one of the multiple decoding networks, and each network block includes one or more network layers; the model training device includes an acquisition unit and a processing unit. The acquisition unit is used to acquire training images; The processing unit is used to divide the training image into blocks to obtain multiple first image blocks; The plurality of first image blocks are occluded to obtain a plurality of second image blocks, wherein the plurality of second image blocks are the first image blocks among the plurality of first image blocks that are not occluded; The feature vector of each second image block is input into the coding network for encoding to obtain multiple coding features corresponding to the multiple network blocks, wherein the multiple network blocks and the multiple coding features correspond one-to-one; Each encoded feature is input into the corresponding decoding network to reconstruct the image, resulting in a reconstructed image for each decoding network. The model is trained based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network.

9. The apparatus according to claim 8, characterized in that, In occluding the plurality of first image blocks to obtain a plurality of second image blocks, the processing unit is specifically configured to: Generate an index value corresponding to each first image block. The index value of each first image block is used to indicate whether each first image block is occluded. Based on the index value of each first image block, the plurality of first image blocks are occluded to obtain the plurality of second image blocks.

10. The apparatus according to claim 9, characterized in that, Before training the model based on the reconstructed image and supervision information corresponding to each decoding network, the processing unit is further configured to: Obtain the block scale corresponding to each decoding network, wherein the shallower the depth of the network block corresponding to the decoding network, the smaller the block scale corresponding to the decoding network; The training image is divided into blocks according to the block scale corresponding to each decoding network to obtain multiple third image blocks corresponding to each decoding network. Multiple third image blocks corresponding to each decoding network are used as supervision information for each decoding network.

11. The apparatus according to claim 10, characterized in that, The reconstructed image corresponding to each decoding network includes multiple fourth image patches; in training the model based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network, the processing unit is specifically used for: Based on the multiple third image blocks in the supervision information of each decoding network, determine the third image block corresponding to each of the multiple fourth image blocks corresponding to each decoding network; The loss of each decoding network is determined based on each fourth image block corresponding to each decoding network and the third image block corresponding to that fourth image block; The model is trained based on the loss of each decoding network.

12. The apparatus according to claim 11, characterized in that, Before the processing unit determines the loss of each decoding network based on each fourth image block corresponding to each decoding network and the third image block corresponding to that fourth image block, the processing unit is further configured to: Generate a first index value vector based on the index value of each first image patch; The first index value vector is input into the model to obtain the second index value vector corresponding to each decoding network; In determining the loss of each decoding network based on a fourth image block corresponding to each decoding network and a third image block corresponding to that fourth image block, the processing unit is specifically configured to: Based on the second index value vector corresponding to each decoding network, determine the index value of each fourth image block corresponding to each decoding network, wherein the index value of each fourth image block is used to indicate whether the real content corresponding to each fourth image block is occluded; The loss for each decoding network is determined based on each fourth image block corresponding to each decoding network, the index value corresponding to that fourth image block, and the third image block corresponding to that fourth image block.

13. The apparatus according to claim 11 or 12, characterized in that, The loss corresponding to each decoding network is determined based on the feature information of each fourth image block corresponding to each decoding network, and the feature information of the third image block corresponding to that fourth image block; The feature information of image block A is one of the following: The pixel value of each pixel in image block A; or, the directional gradient histogram of image block A; or, the normalized pixel value of image block A; image block A is the fourth image block corresponding to each decoding network or the third image block corresponding to the fourth image block.

14. The apparatus according to any one of claims 8-13, characterized in that, The processing unit is also used for: After training the model is completed, the encoding network is used as the backbone network for downstream recognition tasks. The downstream identification tasks include one of the following: pedestrian attribute recognition, image segmentation, and image classification.

15. An electronic device, characterized in that it comprises: Memory, used to store programs; A processor is used to execute programs stored in memory; When the program stored in the memory is executed, the processor is used to implement the method of any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code for execution by the device, the program code including methods for implementing any one of claims 1-7.

Images