Model training method and device, electronic equipment and storage medium
By combining multi-scale feature vectors and multi-scale supervision information, the problem of insufficient utilization of low-level features in self-supervised visual representation learning is solved, which improves learning and training efficiency and enhances the model's recognition accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-22
- Publication Date
- 2026-04-03
AI Technical Summary
Existing self-supervised visual representation learning methods mainly utilize the top-level output of the encoder for image reconstruction, resulting in slow and inefficient low-level learning processes.
Image reconstruction is performed using multi-scale feature vectors. By combining an encoding network and multiple decoding networks, features from different network layers of different depths are used to reconstruct images. Multi-scale supervision information and index values are designed for model training.
It improves the efficiency of self-supervised visual representation learning and model training, enhances the model's recognition accuracy, and enables the trained model to perform well in downstream recognition tasks.
Smart Images
Figure CN121788971A_ABST
Abstract
Description
[0001] This application is a divisional application. The original application has the application number 202310183197.6 and the original application date is February 22, 2023. The entire contents of the original application are incorporated herein by reference. Technical Field
[0002] This invention relates to the field of artificial intelligence technology, specifically to a model training method, apparatus, electronic device, and storage medium. Background Technology
[0003] In recent years, self-supervised visual representation learning has received widespread attention and rapid development. Its research value lies in its ability to learn generalized representations from massive amounts of virtually free, unlabeled data, thereby improving representations for various downstream tasks, such as classification, detection, and segmentation. Because this learning method does not require expensive labeling costs, it has broad application prospects.
[0004] With the emergence and rapid development of visual self-attention networks, masked image modeling has received increasing attention. Self-supervised visual representation learning also employs a masked image modeling method based on a "mask-reconstruction" proxy task to train the model. In this method, a portion of the input image is first randomly occluded. Then, image reconstruction is performed based on the unoccluded portion to predict the occluded portion. Finally, self-supervised visual representation learning is performed based on the predicted occluded portion and the actual occluded portion. However, this "mask-reconstruction" self-supervised visual representation learning method primarily utilizes the encoder's final output features for image reconstruction, making the entire learning process slow. Summary of the Invention
[0005] This application provides a model training method, apparatus, electronic device, and storage medium that utilizes feature vectors of different scales to reconstruct images, guides self-supervised visual representation learning, and improves learning efficiency and model training efficiency.
[0006] In a first aspect, embodiments of this application provide a model training method. The model includes an encoding network and multiple decoding networks. The encoding network includes multiple network blocks of different depths, each network block corresponding to one of the multiple decoding networks, and each network block including one or more network layers. The method includes: dividing a training image into blocks to obtain multiple first image blocks; occluding the multiple first image blocks to obtain multiple second image blocks, wherein the multiple second image blocks are the unoccluded first image blocks among the multiple first image blocks; inputting the feature vector of each second image block into the encoding network for encoding to obtain multiple encoded features corresponding to the multiple network blocks, wherein the multiple network blocks correspond one-to-one with the multiple encoded features; inputting each encoded feature into the decoding network corresponding to each encoded feature for image reconstruction to obtain a reconstructed image corresponding to each decoding network; and training the model based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network.
[0007] As can be seen in this embodiment, during self-supervised visual representation learning, corresponding decoding networks are designed at different depths of the encoding network, i.e., multiple decoding networks are designed. Then, each decoding network is used to reconstruct the image, resulting in the reconstruction of each decoding network. This means that the output features of different depths of the decoding network are used for image reconstruction (i.e., low-level features and high-level features output by the encoding network are used for image reconstruction). Finally, the model is trained based on the reconstructed image of each decoding network and the supervision information. Therefore, this application uses features of different scales output by the encoding network for self-supervised visual representation learning, not just the top-level features (features output by the last network layer), thus enabling the use of more detailed features for learning, improving learning efficiency and model training efficiency, and resulting in a higher recognition accuracy of the trained model.
[0008] In one embodiment of this application, multiple first image blocks are occluded to obtain multiple second image blocks, including: generating an index value corresponding to each first image block, wherein the index value of each first image block is used to indicate whether each first image block is occluded; and occluding multiple first image blocks according to the index value of each first image block to obtain multiple second image blocks.
[0009] As can be seen, in the embodiments of this application, multiple first image blocks are occluded by randomly constructing index values. This random occlusion largely eliminates redundancy and generates a task that cannot be easily solved by extrapolating from visible adjacent patches. This avoids special cases that could lead to model convergence and improves the accuracy of model training.
[0010] In one embodiment of this application, before training the model based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network, the method further includes: obtaining the block scale corresponding to each decoding network, wherein the shallower the network block corresponding to the decoding network, the smaller the block scale corresponding to the decoding network; dividing the training image into blocks according to the block scale corresponding to each decoding network to obtain multiple third image blocks corresponding to each decoding network; and using the multiple third image blocks corresponding to each decoding network as the supervision information of each decoding network.
[0011] As can be seen, in this embodiment, supervision information corresponding to its own depth is constructed for each decoding network, that is, multi-scale supervision information is used for model training. The shallower the decoding network, the finer the scale of the supervision information. This makes it easier to extract the feature information of the output of the lower-level encoding network, better train the lower-level encoding network, and further improve the efficiency of model training and the recognition accuracy of the model.
[0012] In one embodiment of this application, the reconstructed image corresponding to each decoding network includes multiple fourth image blocks; training a model based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network includes: determining a third image block corresponding to each of the multiple fourth image blocks corresponding to each decoding network based on multiple third image blocks in the supervision information of each decoding network; determining the loss of each decoding network based on each fourth image block corresponding to each decoding network and the third image block corresponding to that fourth image block; and training a model based on the loss of each decoding network.
[0013] As can be seen in the embodiments of this application, after constructing corresponding supervision information for each decoder, the loss of each decoder network is determined based on the supervision information of each decoder network, so that the loss of each decoder network is more consistent with the real loss and the training accuracy of the model is improved.
[0014] In one embodiment of this application, before determining the loss of each decoding network based on each fourth image block corresponding to each decoding network and the third image block corresponding to the fourth image block, the method further includes: generating a first index value vector based on the index value of each first image block; inputting the first index value vector into the model to obtain a second index value vector corresponding to each decoding network; determining the loss of each decoding network based on the fourth image block corresponding to each decoding network and the third image block corresponding to the fourth image block, including: determining the index value of each fourth image block corresponding to each decoding network based on the second index value vector corresponding to each decoding network, wherein the index value of each fourth image block is used to indicate whether the real content corresponding to each fourth image block is occluded; and determining the loss corresponding to each decoding network based on each fourth image block corresponding to each decoding network, the index value corresponding to the fourth image block, and the third image block corresponding to the fourth image block.
[0015] As can be seen in this embodiment, during model training, the index value of the first image block is synchronously transmitted to the model for upsampling and / or downsampling. This allows the index value of each image block in the reconstructed image to be determined, i.e., to determine which image blocks in the reconstructed image are occluded. As a result, only the occluded image blocks can be used for loss calculation, improving the efficiency and accuracy of loss calculation, and further improving the accuracy and efficiency of model training.
[0016] In one embodiment of this application, the loss corresponding to each decoding network is determined based on the feature information of each fourth image block corresponding to each decoding network, and the feature information of the third image block corresponding to the fourth image block; wherein, the feature information of image block A is one of the following: the pixel value of each pixel in image block A; or, the directional gradient histogram of image block A; or, the normalized pixel value of image block A; image block A is the fourth image block corresponding to each decoding network or the third image block corresponding to the fourth image block.
[0017] As can be seen, in the embodiments of this application, various morphological features of image blocks can be used as supervision information to improve the diversity of loss calculation and the diversity of model training.
[0018] In one embodiment of this application, the method further includes: after training the model, using the encoding network as the backbone network for downstream recognition tasks; wherein the downstream recognition tasks include one of the following: pedestrian attribute recognition, image segmentation, and image classification.
[0019] As can be seen, since the recognition accuracy of the model trained in this application is high, the encoding network of this application is used as the backbone network of the downstream recognition task, thereby improving the recognition accuracy of the downstream recognition task.
[0020] Secondly, embodiments of this application provide a model training apparatus. The model includes: an encoding network and multiple decoding networks. The encoding network includes multiple network blocks of different depths, each network block corresponds to one of the multiple decoding networks, and each network block includes one or more network layers. The model training apparatus includes: an acquisition unit and a processing unit. The acquisition unit is used to acquire training images. The processing unit is used to divide the training images into blocks to obtain multiple first image blocks; to occlude the multiple first image blocks to obtain multiple second image blocks, wherein the multiple second image blocks are the unoccluded first image blocks among the multiple first image blocks; to input the feature vector of each second image block into the encoding network for encoding to obtain multiple encoded features corresponding to the multiple network blocks, wherein the multiple network blocks correspond one-to-one with the multiple encoded features; to input each encoded feature into the decoding network corresponding to each encoded feature for image reconstruction to obtain a reconstructed image corresponding to each decoding network; and to train the model based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network.
[0021] In one embodiment of this application, in order to occlude multiple first image blocks to obtain multiple second image blocks, the processing unit is specifically configured to: generate an index value corresponding to each first image block, wherein the index value of each first image block is used to indicate whether each first image block is occluded; and occlude multiple first image blocks according to the index value of each first image block to obtain multiple second image blocks.
[0022] In one embodiment of this application, before training the model based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network, the processing unit is further configured to: obtain the block scale corresponding to each decoding network, wherein the shallower the network block corresponding to the decoding network, the smaller the block scale corresponding to the decoding network; divide the training image into blocks according to the block scale corresponding to each decoding network to obtain multiple third image blocks corresponding to each decoding network; and use the multiple third image blocks corresponding to each decoding network as the supervision information of each decoding network.
[0023] In one embodiment of this application, the reconstructed image corresponding to each decoding network includes multiple fourth image blocks; in terms of training the model based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network, the processing unit is specifically configured to: determine a third image block corresponding to each of the multiple fourth image blocks corresponding to each decoding network based on the multiple third image blocks in the supervision information of each decoding network; determine the loss of each decoding network based on each fourth image block corresponding to each decoding network and the third image block corresponding to that fourth image block; and train the model based on the loss of each decoding network.
[0024] In one embodiment of this application, before the processing unit determines the loss of each decoding network based on each fourth image block corresponding to each decoding network and the third image block corresponding to the fourth image block, the processing unit is further configured to: generate a first index value vector based on the index value of each first image block; input the first index value vector into the model to obtain a second index value vector corresponding to each decoding network; in determining the loss of each decoding network based on the fourth image block corresponding to each decoding network and the third image block corresponding to the fourth image block, the processing unit is specifically configured to: determine the index value of each fourth image block corresponding to each decoding network based on the second index value vector corresponding to each decoding network, wherein the index value of each fourth image block is used to indicate whether the real content corresponding to each fourth image block is occluded; and determine the loss corresponding to each decoding network based on each fourth image block corresponding to each decoding network, the index value corresponding to the fourth image block, and the third image block corresponding to the fourth image block.
[0025] In one embodiment of this application, the loss corresponding to each decoding network is determined based on the feature information of each fourth image block corresponding to each decoding network, and the feature information of the third image block corresponding to the fourth image block; wherein, the feature information of image block A is one of the following: the pixel value of each pixel in image block A; or, the histogram of the gradient orientation of image block A; or, the normalized pixel value of image block A; image block A is the fourth image block corresponding to each decoding network or the third image block corresponding to the fourth image block.
[0026] In one embodiment of this application, the processing unit is further configured to: after completing the training of the model, use the encoding network as the backbone network for downstream recognition tasks; wherein, the downstream recognition tasks include one of the following: pedestrian attribute recognition, image segmentation, and image classification.
[0027] Thirdly, embodiments of this application provide an electronic device, including: a memory for storing a program; and a processor for executing the program stored in the memory; when the program stored in the memory is executed, the processor is used to implement the method in the first aspect described above.
[0028] Fourthly, embodiments of this application provide a computer-readable medium storing program code for execution by a device, the program code including methods for implementing the methods in the first aspect described above.
[0029] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to implement the method described in the first aspect. Attached Figure Description
[0030] Figure 1 A schematic diagram illustrating an image reconstruction provided in an embodiment of this application; Figure 2 A schematic diagram of the structure of a model provided in an embodiment of this application; Figure 3 A schematic flowchart illustrating a model training method provided in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the segmentation and occlusion of a training image as provided in an embodiment of this application; Figure 5 This is a schematic diagram illustrating the construction of supervisory information at different scales, provided as an embodiment of this application. Figure 6 A schematic diagram illustrating the construction of an index value matrix provided in an embodiment of this application; Figure 7 A schematic diagram illustrating model training as provided in an embodiment of this application; Figure 8 A schematic diagram of a model training device provided in an embodiment of this application; Figure 9 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0031] To facilitate understanding of this application, the technical knowledge related to this application will be introduced first.
[0032] like Figure 1 As shown, in the current self-supervised visual representation learning process, the input image is first divided into blocks to obtain multiple image patches. Then, these image patches are randomly occluded, and features are extracted from the unoccluded patches. The extracted features are then input into the encoder of the model for encoding, obtaining the encoding result, which is the output of the last layer of the encoder. For occluded image patches, placeholder vectors are added to the encoding result to obtain the input data for the decoder. This input data is then input into the decoder for image reconstruction to obtain the predicted image. Finally, the loss is calculated based on the predicted image and the input image, and the model is trained based on this loss until the model converges, completing the self-supervised visual representation learning process.
[0033] However, the aforementioned self-supervised visual representation learning methods primarily utilize the top-level output of the encoder for image reconstruction, specifically the output of the last network layer. This means that self-supervised visual representation learning only leverages features extracted from the deeper layers of the model, resulting in a slow learning process at lower levels and consequently, a slow and inefficient self-supervised visual representation learning process. Therefore, improving the efficiency of self-supervised visual representation learning is a pressing technical problem that needs to be addressed.
[0034] First, it should be noted that the encoding network involved in this application can also be understood as an encoder, and the decoding network can also be understood as a decoder.
[0035] See Figure 2 , Figure 2 This is a schematic diagram of the structure of a model provided in an embodiment of this application. Figure 2 As shown, the model includes an encoding network and multiple decoding networks. The encoding network comprises multiple network blocks of varying depths, such as... Figure 2 The network block 1, network block 2... network block N are shown, as well as decoding network 1, decoding network 2... decoding network N.
[0036] Optionally, each network block includes one or more network layers in the decoding network, that is, one or more network layers in the decoding network are considered as a network block. Further, the encoding process of each network block can be referred to as a stage; that is, the above-mentioned multiple network blocks divide the entire encoding process of the encoding network into multiple stages (e.g., ...). Figure 2 The diagram shows stages 1, 2, ..., N), and then a decoding network is connected to the last network layer of each stage.
[0037] Optionally, the aforementioned multiple network blocks correspond to one of the multiple decoding networks, such as... Figure 2 As shown, the last network layer of each network block is connected to the decoding network corresponding to that network block.
[0038] For example, the above-mentioned encoding network can be various types of encoding networks, such as the Vit encoding network or the Swin encoding network. It should be noted that if the encoding network itself is pre-divided into stages, that is, the encoding network itself contains multiple network blocks, such as the Swin encoding network, then the pre-divided network blocks can be used as the multiple network blocks mentioned above; if the encoding network is not pre-divided into stages, such as the Vit encoding network, the multiple network layers of the encoding network can be divided into blocks to obtain multiple network blocks.
[0039] See Figure 3 , Figure 3 This is a flowchart illustrating a model training method provided in an embodiment of this application. The model is... Figure 2 The model shown. This method includes, but is not limited to, the following steps: 301: Divide the training image into blocks to obtain multiple first image blocks.
[0040] Optionally, such as Figure 4 As shown, the training image is uniformly divided into non-overlapping blocks to obtain multiple first image blocks.
[0041] 302: Occlude multiple first image blocks to obtain multiple second image blocks, wherein the multiple second image blocks are the first image blocks that are not occluded among the multiple first image blocks.
[0042] For example, such as Figure 4 As shown, multiple first image blocks are randomly occluded to obtain multiple second image blocks. For ease of distinction, in this application, the first image block that is not occluded among the multiple first image blocks is referred to as the second image block.
[0043] For example, an index value is generated for each first image block, where the index value of each first image block indicates whether each first image block is occluded. Optionally, the index value of each first image block is randomly generated between 0 and 1; for example, an index value of 0 indicates that the first image block is not occluded, and an index value of 1 indicates that the first image block is occluded. Based on the index value corresponding to each first image block, multiple first image blocks are occluded to obtain multiple second image blocks.
[0044] 303: Input the feature vector of each second image block into the coding network for encoding to obtain multiple coding features corresponding to the multiple network blocks, wherein the multiple network blocks correspond one-to-one with the multiple coding features.
[0045] Optionally, after dividing the training image into blocks, each first image block is embedded to obtain an embedding vector for each first image block; then, position encoding is added to the embedding vector of each first image block to obtain a feature vector for each first image block. In this way, after occluding multiple first image blocks, the feature vector of each second image block can be directly input into the encoding network for encoding, that is, only the feature vector of the unoccluded first image block is input.
[0046] Optionally, after dividing the training image into blocks, instead of directly embedding each first image block, multiple first image blocks are first occluded; then, each second image block is embedded to obtain the embedding vector of each second image block; then, position encoding is added to the embedding vector of each second image block to obtain the feature vector of each second image block; finally, the feature vector of each second image block is input into the encoding network for encoding.
[0047] For example, the feature vector of each second image block is input into the encoding network for encoding. The feature vector is encoded by multiple network blocks in the encoding network to obtain multiple encoded features corresponding to multiple network blocks. That is, the output of the last network layer in each network block is used as the encoded feature corresponding to each network block.
[0048] 304: Input each encoded feature into the corresponding decoding network to reconstruct the image, and obtain the reconstructed image corresponding to each decoding network.
[0049] For example, the encoded features output by each network block are input into the decoding network corresponding to that network block to reconstruct the image, thereby obtaining the reconstructed image corresponding to each decoding network.
[0050] Specifically, placeholder vectors are added to the encoded features (i.e., feature vector sequences) corresponding to each decoding network to generate new encoded features (i.e., new feature vectors). These placeholder vectors are pre-learned and are shared learning vectors used to indicate that there is an occluded image patch to be predicted at the location corresponding to the placeholder vector. Then, positional encoding is added to the new encoded features corresponding to each decoding network to obtain the input data corresponding to each decoding network. Finally, the input data corresponding to each decoding network is input into each decoding network for image reconstruction, resulting in the reconstructed image corresponding to each decoding network.
[0051] 305: Train the model based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network.
[0052] For example, based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network, the loss corresponding to each decoding network is determined; and the model is trained according to the loss corresponding to each decoding network.
[0053] For example, since the decoding network reconstructs the training image, and this application segments the training image into multiple first image patches and inputs them into the model for image reconstruction, the reconstruction of the training image can also be understood as the reconstruction of image patches. Therefore, the reconstructed image corresponding to each decoding network includes multiple fourth image patches. Accordingly, the supervision information corresponding to each decoding network is the supervision information corresponding to the multiple fourth image patches of each decoding network. Therefore, the loss corresponding to each decoding network is determined based on each fourth image patch in the reconstructed image corresponding to each decoding network and the supervision information corresponding to each fourth image patch.
[0054] The process of obtaining the supervision information corresponding to each decoding network is described below.
[0055] Scenario 1: The scale of the supervisory information differs among the various decoding networks.
[0056] For example, when the scale of the supervision information of each decoding network is different, the block scale corresponding to each decoding network is obtained, wherein the shallower the depth of the network block corresponding to the decoding network, the smaller the block scale corresponding to that decoding network. Figure 2As shown, the block scale of decoding network 1 is smaller than that of decoding network 2. Based on the block scale corresponding to each decoding network, the training image is divided into blocks, resulting in multiple third image blocks corresponding to each decoding network. That is, using the block scale corresponding to each decoding network, the training image is divided into uniform and non-overlapping blocks to obtain multiple third image blocks corresponding to each decoding network. These multiple third image blocks corresponding to each decoding network are used as the supervision information for each decoding network. For example, as... Figure 5 As shown, the training image is divided into blocks using block scales 1, 2, ..., N corresponding to decoding networks 1, 2, ..., N, respectively, to obtain supervision information corresponding to each decoding network. After dividing the training image into blocks using the block scale of each decoding network, the deeper the decoding network, the larger the size of its corresponding third image block, and the fewer the number of third image blocks.
[0057] Furthermore, each of the multiple fourth image blocks in the reconstructed image of each decoding network corresponds one-to-one with the multiple third image blocks corresponding to each decoding network. Therefore, based on the multiple third image blocks in the supervision information of each decoding network, the third image block corresponding to each of the multiple fourth image blocks corresponding to each decoding network can be determined (i.e., the supervision information of each fourth image block can be determined). Then, based on each fourth image block corresponding to each decoding network and the third image block corresponding to that fourth image block, the loss of each decoding network can be determined. That is, by combining the losses between each fourth image block corresponding to each decoding network and the third image block corresponding to that fourth image block, the loss of each decoding network can be obtained. The loss between each fourth image block and the third image block corresponding to that fourth image block can be the cross-entropy loss between the two image blocks.
[0058] It can be seen that different scales of supervision information are designed for each decoding network. Specifically, the shallower the decoding network, the finer the granularity of the supervision information. This allows for the use of finer-scale feature information for loss calculation, resulting in higher accuracy of the calculated loss for each decoding network, thereby improving the training efficiency and accuracy of the model.
[0059] It should be noted that since the scale of the supervision information for each decoding network is different, the size of multiple fourth image patches in the reconstructed image output by each decoding network is designed to be the same as the size of multiple third image patches. It should also be noted that since the input data for image reconstruction by each decoding network comes from the decoding network itself, the number of patches in the reconstructed image depends on the number of patches in the training image. Therefore, to ensure that the size of the fourth image patch is the same as the size of the third image patch, the decoding network needs to perform appropriate upsampling or downsampling based on the size of the third image patch corresponding to that decoding network during image reconstruction. For example, the size of the training image is 20... 20, the size of the first image block is 10. 10. If no upsampling or downsampling process is designed for the decoding network, the block size in the reconstructed image will also be 10. 10, but if the size of the third image block corresponding to the decoding network is 5 If the value is 5, then an upsampling process is required during image reconstruction to ensure that the block size of the reconstructed image is 5. 5.
[0060] In practical applications, the model can be trained by calculating only the loss between the occluded content and the predicted content in the training image. Therefore, for each decoding network, a target fourth image patch can be identified among multiple fourth image patches corresponding to each decoding network. The content corresponding to this target fourth image patch is occluded, meaning the actual content of the target fourth image patch in the training image is occluded. The loss between the target fourth image patch corresponding to each decoding network and the corresponding third image patch is determined, resulting in the loss for each decoding network. Finally, the model is trained based on the loss of each decoding network.
[0061] In one embodiment of this application, a target fourth image block among a plurality of fourth image blocks corresponding to each decoding network can be determined based on an index value. For example, the index value of each fourth image block corresponding to each decoding network is obtained. The index value of each fourth image block indicates whether the actual content corresponding to that fourth image block is occluded. For instance, an index value of 0 indicates that the actual content corresponding to that fourth image block is not occluded, while an index value of 1 indicates that the actual content corresponding to that fourth image block is occluded. The target fourth image block among a plurality of fourth image blocks corresponding to each decoding network is determined based on the index value of each fourth image block.
[0062] Specifically, such as Figure 6As shown, after generating the index value corresponding to each first image patch, a first index value matrix can be generated based on the index value corresponding to each first image patch. This first index value matrix is a square matrix, and the product of its length and width is the number of first image patches. Therefore, the index value in the i-th row and j-th column of the first index value matrix is the index value of the first image patch in the i-th row and j-th column among the multiple first image patches. Then, the first index value matrix is tiled to obtain a first index value vector. Then, when inputting the feature vector of the second image patch, this first index value vector is simultaneously input into the model, i.e., encoded (downsampled) by the decoding network and decoded (downsampled) by the decoding network to obtain a second index value vector corresponding to each decoding network. Then, the second index value vector is reshaped to obtain a second index value matrix corresponding to each decoding network, where the second index value matrix is a square matrix, and the product of its length and width is the number of fourth image patches. Accordingly, the element in the i-th row and j-th column of the second index value matrix corresponding to each decoding network is the index value of the i-th row and j-th column fourth image block among the multiple fourth image blocks corresponding to each decoding network. Then, based on the second index value matrix, the index value corresponding to each fourth image block corresponding to each decoding network is determined. Based on the index value corresponding to each fourth image block corresponding to each decoding network, the target fourth image block among the multiple fourth image blocks corresponding to each decoding network can be determined.
[0063] In one embodiment of this application, after determining the index value of each fourth image block, the loss corresponding to each decoding network can also be determined based on each fourth image block corresponding to each decoding network, the index value corresponding to the fourth image block, and the third image block corresponding to the fourth image block.
[0064] For example, when using index values, the loss for each decoding network can be expressed by formula (1): Formula (1); in, For the first l The loss of the decoding network, i Indicates the first l The first of the multiple fourth image blocks corresponding to the decoding network i The fourth image block, For the first i The index value corresponding to the fourth image patch For the first i The supervision information corresponding to the i-th fourth image block is the third image block corresponding to the i-th fourth image block. For the first i The fourth image block, Indicates the first i The fourth image patch and the i Cross-entropy loss between the supervision information corresponding to the fourth image patch.
[0065] As can be seen, by using the index value of each fourth image block, only the fourth image block with an index value of 1 (i.e., the target fourth image block mentioned above) participates in the loss calculation. This means that only the loss between the occluded content and the predicted content is calculated, thereby speeding up the loss calculation and improving the computational efficiency.
[0066] In one embodiment of this application, the loss of each decoding network can be determined based on the feature information of each fourth image block corresponding to each decoding network and the loss between the feature information of the third image block corresponding to the fourth image block. The feature information of each fourth image block can be the pixel value of each pixel in the fourth image block, and correspondingly, the feature information of the third image block corresponding to the fourth image block is also the pixel value of each pixel in the third image block. That is, calculating the cross-entropy loss between the pixel values of the fourth and third image blocks yields the loss of each decoding network. Alternatively, the feature information of the fourth image block can be its directional gradient histogram, and correspondingly, the feature information of the third image block corresponding to the fourth image block is also its directional gradient histogram. Calculating the cross-entropy loss between the directional gradient histograms of the fourth and third image blocks yields the loss of each decoding network. Alternatively, the feature information of the fourth image block can be its normalized pixel value, and correspondingly, the feature information of the third image block is also its normalized pixel value. This application does not limit the feature information of the image blocks.
[0067] Scenario 2: The scale of the supervision information of each decoding network is the same.
[0068] Optionally, the scale of the supervision information of each decoding network is the same, that is, the size of the supervision information corresponding to the fourth image block reconstructed by each decoding network is the same.
[0069] For example, in this application, the segmented training images can be used as supervision information for each decoding network, that is, multiple first image blocks can be used as supervision information for each decoding network. Of course, in practical applications, a new segmentation scale can also be used to segment the training images into multiple fifth image blocks, and then these multiple fifth image blocks can be used as supervision information for each decoding network. In this application, the use of multiple first image blocks as supervision information for each decoding network is used as an example for illustration.
[0070] Then, obtain the first image block (supervision information) corresponding to each fourth image block of each decoding network; based on each fourth image block corresponding to each decoding network and the first image block corresponding to that fourth image block, determine the loss of each decoding network, that is, determine the loss between each fourth image block and the first image block of each decoding network, and obtain the loss of each decoding network. Similarly, the loss of each decoding network can be determined based on the index value of each fourth image block of each decoding network, except that the supervision information of each fourth image block is changed from the third image block to the first image block, as shown in the above formula (1), which will not be described again.
[0071] It should be noted that if the encoding network changes the size of the input data, for example, by downsampling the input data, so that the size of the input data of each decoding network is different, then in order to make the size of the fourth image block reconstructed by each decoding network the same, the decoding network will perform corresponding upsampling so that the size of the fourth image block reconstructed by each decoding network is the same and the same as the size of the first image block.
[0072] In one embodiment of this application, after obtaining the loss of each decoding network, the model can be trained as a whole based on the loss of each decoding network. For example, after obtaining the loss of each decoding network, the losses of multiple decoding networks are weighted according to the weight of each decoding network to obtain the target loss; the model is trained based on the target loss until the model converges, thus completing the training of the model.
[0073] For example, the target loss can be expressed by formula (2): Formula (2); in, Loss to the target For the model's first Each network layer, Let the weights be those of the l-th network layer. This refers to the set of network layers consisting of the last network layer among the aforementioned network blocks, i.e. This represents a weighted summation of the losses from multiple decoding networks.
[0074] In one embodiment of this application, after obtaining the loss of each decoding network, the above model can be trained layer by layer based on the loss of each decoding network, i.e., greedy training.
[0075] The following explains why this application can perform layer-by-layer training.
[0076] First, with either the ViT-B or Swin-B decoding network, the model of this application (i.e., the Local Multi-Scale Reconfiguration (LaSMR) model) was trained using gradient truncation training, and the LaSMR model of this application was trained without gradient truncation training, i.e., the LaSMR model was trained as a whole. Then, the training time (i.e., the average time consumed by the image processor) and the model accuracy (acc) were obtained for both training modes. Table 1 shows the training time and accuracy for the two training modes.
[0077] Table 1:
[0078] As shown in Table 1, when the encoding network is ViT-B, the training time required for training the LaSMR model using gradient truncation is the same as that without gradient truncation, and the achieved accuracy is also the same. When the encoding network is Swin-B, the training time required for training the LaSMR model using gradient truncation is the same as that without gradient truncation, and the accuracy is also very close. Table 1 shows that gradient truncation training of the model in this application does not increase the training time or decrease the accuracy. In other words, the model in this application can be trained using gradient truncation, therefore, the model in this application supports layer training.
[0079] See Table 2, which shows the training results when training the model of this application layer by layer.
[0080] Table 2:
[0081] Based on the content shown in Table 2, it can be seen that by training the model of this application layer by layer, memory overhead and pre-training time can be significantly reduced. Layer-by-layer training brings a speedup of 30-40% and saves 40-55% of memory usage without reducing the accuracy of the model.
[0082] The training process of the model in this application is illustrated below using the Swin network (which consists of four stages) as an example.
[0083] like Figure 7As shown, the training image is first divided into image blocks, and these blocks are then occluded to obtain multiple unoccluded image blocks (i.e., the second image block of this application). Then, the features of the unoccluded images are input into the encoding network for encoding, resulting in encoded features corresponding to each stage of the encoding network, i.e., encoded features corresponding to each decoding network. Next, placeholder vectors of the occluded image blocks are added to the encoded features of each decoding network to obtain new encoded features corresponding to each decoding network, and positional encoding is added to these new encoded features. Figure 7 (Not shown in the image), the input data of each decoding network can be obtained; the input data of each decoding network is input into each decoding network to reconstruct the image, and the reconstructed image (including multiple reconstructed image blocks) corresponding to each decoding network is obtained.
[0084] Furthermore, this application constructs corresponding supervision information for each decoding network; that is, the scale of the supervision information for decoding networks of different depths in this application is different. Then, based on the supervision information of each decoding network and the image patches reconstructed by each decoding network, the loss corresponding to each decoding network is determined; finally, the model is trained based on the loss corresponding to each decoding network.
[0085] In one embodiment of this application, after training the model is completed, the encoding network is used as the backbone network for a downstream recognition task; wherein, the downstream recognition task includes one of the following: pedestrian attribute recognition, image segmentation, and image classification. Specifically, the trained encoding network is used as the backbone network for the downstream recognition task to extract features, so as to perform pedestrian attribute recognition, image segmentation, and image classification.
[0086] Specifically, in autonomous driving scenarios, it is necessary to segment the road images captured by cameras to separate different objects such as road surface, roadbed, vehicles, and pedestrians, thereby ensuring that the vehicle travels within the correct area. Using the coding network of this application as the backbone network for semantic feature extraction of the image can improve the accuracy of image segmentation. In intelligent monitoring systems, it is necessary to identify pedestrian attributes in the images captured by cameras, such as the pedestrian's gender, age, hair, clothing, and backpack. Using the coding network of this application as the backbone network for semantic feature extraction of the image can improve the accuracy of pedestrian attribute identification.
[0087] See Figure 8 , Figure 8 This is a schematic diagram of a model training device provided in an embodiment of this application. Figure 8 As shown, the model training device includes an acquisition unit 801 and a processing unit 802; The acquisition unit 801 is used to acquire training images; The processing unit 802 is used to divide the training image into blocks to obtain multiple first image blocks; The plurality of first image blocks are occluded to obtain a plurality of second image blocks, wherein the plurality of second image blocks are the first image blocks among the plurality of first image blocks that are not occluded; The feature vector of each second image block is input into the coding network for encoding to obtain multiple coding features corresponding to the multiple network blocks, wherein the multiple network blocks and the multiple coding features correspond one-to-one; Each encoded feature is input into the corresponding decoding network to reconstruct the image, resulting in a reconstructed image for each decoding network. The model is trained based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network.
[0088] For a more detailed description of the acquisition unit 801 and the processing unit 802, please refer to the relevant descriptions in the above method embodiments, which will not be repeated here.
[0089] See Figure 9 , Figure 9 This is a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device 900 can be the model training device 800 described above. The electronic device 900 includes a memory 901, a processor 902, a communication interface 903, and a bus 904. The memory 901, processor 902, and communication interface 903 are interconnected via the bus 904.
[0090] The memory 901 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 901 can store programs; when the electronic device 900 is the model training device 800 described above, when the program stored in the memory 901 is executed by the processor 902, the processor 902 and the communication interface 903 are used to execute the various steps in the model training method of the embodiments of this application.
[0091] The processor 902 may be a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to implement the model training method of the embodiments of this application.
[0092] The processor 902 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the data stream transmission method of this application can be completed by the integrated logic circuits in the hardware of the processor 902 or by instructions in software form. The processor 902 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 901, and processor 902 reads the information in memory 901 to execute the various steps in the model training method.
[0093] The communication interface 903 can be a transceiver device such as a transceiver to enable communication between the electronic device 900 and other devices or communication networks. The communication interface 903 can also be an input-output interface to enable data transmission between the electronic device 900 and input-output devices, including but not limited to keyboards, mice, displays, USB flash drives, and hard drives. For example, the processor 902 can acquire voice samples through the communication interface 903.
[0094] Bus 904 may include a pathway for transmitting information between various components of device electronics 900 (e.g., memory 901, processor 902, communication interface 903).
[0095] It should be noted that, although Figure 9 The illustrated electronic device 900 only shows the memory, processor, and communication interface. However, those skilled in the art should understand that in specific implementations, the electronic device 900 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the electronic device 900 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the electronic device 900 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 9 All the devices shown.
[0096] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0097] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0098] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0099] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates an "or" relationship between the preceding and following related objects; in the formulas of this application, the character " / " indicates a "division" relationship between the preceding and following related objects.
[0100] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The order of the process numbers described above does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.
[0101] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0102] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A model training method, characterized in that, The model includes an encoding network and multiple decoding networks, wherein the encoding network comprises multiple network blocks of different depths, and each network block corresponds to one of the multiple decoding networks; the method includes: The training image is divided into blocks to obtain multiple first image blocks; The plurality of first image blocks are occluded to obtain a plurality of second image blocks, wherein the plurality of second image blocks are the first image blocks among the plurality of first image blocks that are not occluded; The feature vector of each second image block is input into the coding network for encoding to obtain multiple coding features corresponding to the multiple network blocks, wherein the multiple network blocks and the multiple coding features correspond one-to-one; Each encoded feature is input into the corresponding decoding network to reconstruct the image, resulting in a reconstructed image for each decoding network. The model is trained based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network.
2. The method according to claim 1, characterized in that, The process of occluding the plurality of first image blocks to obtain a plurality of second image blocks includes: Generate an index value corresponding to each first image block. The index value of each first image block is used to indicate whether each first image block is occluded. Based on the index value of each first image block, the plurality of first image blocks are occluded to obtain the plurality of second image blocks.
3. The method according to claim 2, characterized in that, Before training the model, based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network, the method further includes: Obtain the block scale corresponding to each decoding network, wherein the shallower the depth of the network block corresponding to the decoding network, the smaller the block scale corresponding to the decoding network; The training image is divided into blocks according to the block scale corresponding to each decoding network to obtain multiple third image blocks corresponding to each decoding network. Multiple third image blocks corresponding to each decoding network are used as supervision information for each decoding network.
4. The method according to claim 3, characterized in that, The reconstructed image corresponding to each decoding network includes multiple fourth image patches; the step of training the model based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network includes: Based on the multiple third image blocks in the supervision information of each decoding network, determine the third image block corresponding to each of the multiple fourth image blocks corresponding to each decoding network; The loss of each decoding network is determined based on each fourth image block corresponding to each decoding network and the third image block corresponding to that fourth image block; The model is trained based on the loss of each decoding network.
5. The method according to claim 4, characterized in that, Before determining the loss of each decoding network based on each fourth image block corresponding to each decoding network and the third image block corresponding to that fourth image block, the method further includes: Generate a first index value vector based on the index value of each first image patch; The first index value vector is input into the model to obtain the second index value vector corresponding to each decoding network; The step of determining the loss of each decoding network based on a fourth image block corresponding to each decoding network and a third image block corresponding to that fourth image block includes: Based on the second index value vector corresponding to each decoding network, determine the index value of each fourth image block corresponding to each decoding network, wherein the index value of each fourth image block is used to indicate whether the real content corresponding to each fourth image block is occluded; The loss for each decoding network is determined based on each fourth image block corresponding to each decoding network, the index value corresponding to that fourth image block, and the third image block corresponding to that fourth image block.
6. The method according to claim 4 or 5, characterized in that, The loss corresponding to each decoding network is determined based on the feature information of each fourth image block corresponding to each decoding network, and the feature information of the third image block corresponding to that fourth image block; The feature information of image block A is one of the following: The pixel value of each pixel in image block A; or, the directional gradient histogram of image block A; or, the normalized pixel value of image block A; image block A is the fourth image block corresponding to each decoding network or the third image block corresponding to the fourth image block.
7. The method according to any one of claims 1-6, characterized in that, The method further includes: After training the model is completed, the encoding network is used as the backbone network for downstream recognition tasks. The downstream identification tasks include one of the following: pedestrian attribute recognition, image segmentation, and image classification.
8. A model training device, characterized in that, The model includes an encoding network and multiple decoding networks, wherein the encoding network includes multiple network blocks of different depths, and each network block corresponds to one of the multiple decoding networks; the model training device includes an acquisition unit and a processing unit. The acquisition unit is used to acquire training images; The processing unit is used to divide the training image into blocks to obtain multiple first image blocks; The plurality of first image blocks are occluded to obtain a plurality of second image blocks, wherein the plurality of second image blocks are the first image blocks among the plurality of first image blocks that are not occluded; The feature vector of each second image block is input into the coding network for encoding to obtain multiple coding features corresponding to the multiple network blocks, wherein the multiple network blocks and the multiple coding features correspond one-to-one; Each encoded feature is input into the corresponding decoding network to reconstruct the image, resulting in a reconstructed image for each decoding network. The model is trained based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network.
9. The apparatus according to claim 8, characterized in that, In occluding the plurality of first image blocks to obtain a plurality of second image blocks, the processing unit is specifically configured to: Generate an index value corresponding to each first image block. The index value of each first image block is used to indicate whether each first image block is occluded. Based on the index value of each first image block, the plurality of first image blocks are occluded to obtain the plurality of second image blocks.
10. The apparatus according to claim 9, characterized in that, Before training the model based on the reconstructed image and supervision information corresponding to each decoding network, the processing unit is further configured to: Obtain the block scale corresponding to each decoding network, wherein the shallower the depth of the network block corresponding to the decoding network, the smaller the block scale corresponding to the decoding network; The training image is divided into blocks according to the block scale corresponding to each decoding network to obtain multiple third image blocks corresponding to each decoding network. Multiple third image blocks corresponding to each decoding network are used as supervision information for each decoding network.
11. The apparatus according to claim 10, characterized in that, The reconstructed image corresponding to each decoding network includes multiple fourth image patches; in training the model based on the reconstructed image corresponding to each decoding network and the supervision information corresponding to each decoding network, the processing unit is specifically used for: Based on the multiple third image blocks in the supervision information of each decoding network, determine the third image block corresponding to each of the multiple fourth image blocks corresponding to each decoding network; The loss of each decoding network is determined based on each fourth image block corresponding to each decoding network and the third image block corresponding to that fourth image block; The model is trained based on the loss of each decoding network.
12. The apparatus according to claim 11, characterized in that, Before the processing unit determines the loss of each decoding network based on each fourth image block corresponding to each decoding network and the third image block corresponding to that fourth image block, the processing unit is further configured to: Generate a first index value vector based on the index value of each first image patch; The first index value vector is input into the model to obtain the second index value vector corresponding to each decoding network; In determining the loss of each decoding network based on a fourth image block corresponding to each decoding network and a third image block corresponding to that fourth image block, the processing unit is specifically configured to: Based on the second index value vector corresponding to each decoding network, determine the index value of each fourth image block corresponding to each decoding network, wherein the index value of each fourth image block is used to indicate whether the real content corresponding to each fourth image block is occluded; The loss for each decoding network is determined based on each fourth image block corresponding to each decoding network, the index value corresponding to that fourth image block, and the third image block corresponding to that fourth image block.
13. The apparatus according to claim 11 or 12, characterized in that, The loss corresponding to each decoding network is determined based on the feature information of each fourth image block corresponding to each decoding network, and the feature information of the third image block corresponding to that fourth image block; The feature information of image block A is one of the following: The pixel value of each pixel in image block A; or, the directional gradient histogram of image block A; or, the normalized pixel value of image block A; image block A is the fourth image block corresponding to each decoding network or the third image block corresponding to the fourth image block.
14. The apparatus according to any one of claims 8-13, characterized in that, The processing unit is also used for: After training the model is completed, the encoding network is used as the backbone network for downstream recognition tasks. The downstream identification tasks include one of the following: pedestrian attribute recognition, image segmentation, and image classification.
15. An electronic device, characterized in that it comprises: Memory, used to store programs; A processor is used to execute programs stored in memory; When the program stored in the memory is executed, the processor is used to implement the method of any one of claims 1-7.
16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code for execution by the device, the program code including methods for implementing any one of claims 1-7.