Model training method and device, computer equipment and storage medium

Abstract

The invention relates to a model training method and device, computer equipment and a storage medium. The method comprises the steps that a to-be-annotated image is processed based on prompt information and an image segmentation model, a target mask image is obtained, the prompt information is used for specifying a graphic code in the to-be-annotated image, and the target mask image is used for segmenting the graphic code from the to-be-annotated image; and based on the to-be-labeled image and the target mask image, performing hierarchical training on each network layer of an image decoding network of the image segmentation model to obtain the graphic code labeling model, the graphic code labeling model being used for labeling graphic codes in the image. By adopting the method and the device, the graphic code labeling efficiency can be improved based on a large model, and the data volume of the training data can be efficiently expanded.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a training method, device, computer equipment and computer-readable storage medium for a graphic code annotation model. Background Art

[0002] The development of computer technology has brought a lot of convenience to people's lives. For example, people transfer money to merchants by shooting QR codes with the camera of their mobile phones. In this process, the QR code in the captured image needs to be recognized. In order to accurately recognize the QR code in the image, it is usually necessary to use a large amount of training data to train the recognition model that can recognize the QR code.

[0003] However, in the current process of preparing training data, it is usually necessary to manually annotate the QR codes or graphic codes in the images, which results in low annotation efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a training method, device, computer equipment, computer-readable storage medium and computer program product for a graphic code annotation model to address the above-mentioned technical problems, which can improve the efficiency of graphic code annotation and efficiently expand the amount of training data.

[0005] In a first aspect, the present application provides a method for training a graphic code annotation model, comprising:

[0006] Processing the image to be annotated based on the prompt information and the image segmentation model to obtain a target mask image, wherein the prompt information is used to specify the graphic code in the image to be annotated, and the target mask image is used to segment the graphic code from the image to be annotated;

[0007] Based on the image to be annotated and the target mask image, each network layer of the image decoding network of the image segmentation model is hierarchically trained to obtain the graphic code annotation model, and the graphic code annotation model is used to annotate the graphic code in the image.

[0008] In a second aspect, the present application provides a training device for a graphic code annotation model, comprising:

[0009] A mask image acquisition module, used for processing the image to be annotated based on the prompt information and the image segmentation model to obtain a target mask image, wherein the prompt information is used to specify the graphic code in the image to be annotated, and the target mask image is used to segment the graphic code from the image to be annotated;

[0010] A training module is used to hierarchically adjust the various network layers of the image decoding network of the image segmentation model based on the image to be annotated and the target mask image to obtain the graphic code annotation model, and the graphic code annotation model is used to annotate the graphic code in the image.

[0011] In a third aspect, the present application provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method when executing the computer program.

[0012] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above method when executed by a processor.

[0013] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps in the above method.

[0014] The training method, device, computer equipment, computer-readable storage medium and computer program product of the above-mentioned graphic code annotation model obtain a target mask image by processing the image to be annotated based on the prompt information and the image segmentation model, thereby improving the generation efficiency of training samples based on the large model and increasing the data volume of the training samples. Then, based on the annotated image and the target mask image, the various network layers of the image decoding network of the image segmentation model are layered trained to obtain a graphic code annotation model, which can improve the accuracy of training and reduce the influence of other network layers on the network layer during training, thereby affecting the accuracy and convergence rate of the network layer parameters. Moreover, compared with manually annotating graphic codes, the generated graphic code annotation model can directly annotate the graphic codes in the image, thereby improving the annotation efficiency of the graphic codes, and further improving the data volume and generation efficiency of the training samples required for the graphic code recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG1 is a flow chart of a method for training a graphic code annotation model provided in an embodiment of the present application;

[0016] FIG2 is a schematic diagram of a process for obtaining a graphic code annotation model provided in an embodiment of the present application;

[0017] Figure 3 A schematic diagram of a process of marking a graphic code provided in an embodiment of the present application;

[0018] Figure 4 A schematic diagram of a process for training a lightweight coding network provided in an embodiment of the present application;

[0019] Figure 5A schematic diagram of another process of marking a graphic code provided in an embodiment of the present application;

[0020] Figure 6 A structural block diagram of a training device for a graphic code annotation model provided in an embodiment of the present application;

[0021] FIG7 is an internal structure diagram of a computer device provided in an embodiment of the present application;

[0022] Figure 8 An internal structure diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0024] As shown in FIG1 , the embodiment of the present application provides a training method for a graphic code annotation model, and the method is applied to a server as an example for explanation. It is understandable that the computer device may include at least one of a terminal and a server. The method includes the following steps:

[0025] S102, processing the image to be annotated based on the prompt information and the image segmentation model to obtain a target mask image.

[0026] Among them, the image segmentation model can accurately segment the objects in the input image, so as to obtain the mask image corresponding to the input image. The mask image can be a binary mask image or a probabilistic mask image, which is used to indicate the possibility that each pixel in the image belongs to the target object. If the target object to be marked is not specified when the image segmentation model is used to mark the image to be marked, the image segmentation model will try to segment all objects in the input image. Since the embodiment of the present application focuses on the graphic code in the image, the image segmentation model needs to output the relevant information of the graphic code in the image, ignoring the relevant information of other objects in the image. For example, if the target mask image is a binary mask image, and the pixels with a pixel value of 1 are the pixels of the graphic code, and the pixels with a pixel value of 0 are the background pixels other than the pixels of the graphic code, then all the pixels with a pixel value of 1 are combined, and the position of the graphic code can be determined from the image, and then the graphic code in the image can be identified.

[0027] Therefore, in order to obtain the mask image of the graphic code, the image to be annotated can be processed based on the prompt information and the image segmentation model, wherein the prompt information is used to specify the graphic code in the image so that the mask image output by the image segmentation model can segment the graphic code from the image.

[0028] It is understandable that the graphic code can be a barcode or a QR code. In this embodiment, the graphic code has a corner point. The corner point is the intersection of two edge lines of the graphic code.

[0029] In some embodiments, the image segmentation model may be a SAM (segment-anything-model) model, a DINO (Detection and Instance Segmentation with Transformers) model, or the like.

[0030] The prompt information is used to segment the target object in the input image. The target object is, for example, a graphic code, so the prompt information can be a point, a box, or text related to the graphic code. The point can be any point on the graphic code. The rectangular box can be a box of any shape surrounding the graphic code. The text can be, for example, words related to the graphic code such as "barcode" and "QR code".

[0031] Exemplarily, the server inputs the prompt information and the image to be annotated into the image segmentation model. The image segmentation model processes the image to be annotated based on the prompt information to obtain a target mask image, so that the graphic code can be segmented from the image to be annotated, and further training data for the graphic code annotation model can be obtained.

[0032] S104, based on the image to be annotated and the target mask image, hierarchically train each network layer of the image decoding network of the image segmentation model to obtain a graphic code annotation model, which is used to annotate the graphic code in the image input to the model.

[0033] Among them, the image segmentation model includes image encoding network, hint encoding network and image decoding network.

[0034] The image encoding network converts the image to be annotated into a feature representation. Optionally, the image segmentation model uses a pre-trained visual model (such as the ViT-H model, the Vision Transformer Hierarchical model) as the basis of its image encoding network to extract high-level features of the image to be annotated. Next, the image to be annotated passes through the backbone network to obtain the corresponding feature map. The feature map contains rich information about the image to be annotated, and each pixel is encoded into a high-dimensional vector. After that, the image segmentation model further processes and fuses these features through a series of convolutional layers or other layers to generate a more powerful feature representation.

[0035] The hint encoding network is used to process hint information. The hint encoding network can encode hint information into a form compatible with image features. For example, the position of a point is encoded into a heat map, and the box can be encoded into a heat map of two points. The feature representation generated by the hint encoding network is then merged with the feature representation generated by the image encoding network to enable the model to understand the target segmentation area.

[0036] The image decoding network based on the large model uses image features and cue features to predict a segmentation mask through multiple network layers (such as convolutional layers, fully connected layers, etc.). The segmentation mask can be further refined through some optimization processes, such as conditional random fields (CRFs), graph cuts, etc., to improve the accuracy of segmentation.

[0037] Since the embodiment of the present application needs to obtain the target mask image of the concerned graphic code, and the image segmentation model always requires the user to input prompt information to obtain the target mask image. In order to directly obtain the target mask image related to the graphic code, the image to be annotated and the corresponding target mask image can be used to train the image segmentation model, so that the mask image output based on the trained image segmentation model (graphic code annotation model) can obtain the relevant information of the barcode in the input image (such as the corner points of the barcode, the position of the barcode in the image, etc.).

[0038] Due to the difference in input data, the image segmentation model needs to input the image and the corresponding prompt information, while the graphic code annotation model only needs to input the image. Both models need to encode the image to obtain the feature representation of the image, so the image encoding network does not need to be processed. However, the image decoding network responsible for obtaining the corresponding mask image based on the image features needs to be retrained based on the image to be annotated and the corresponding target mask image, so that the graphic code annotation model can generate a mask image that can accurately identify the graphic code.

[0039] In addition, the image decoding network in the image segmentation model includes multiple network layers, and each network layer is relatively complex. If all network layers are trained at the same time, it may affect the training efficiency and increase the training time. Therefore, in this embodiment, the network layers in the image decoding network can be trained separately. When one of the network layers is trained, the other network layers are not processed, that is, the other network layers are frozen. Since the image features encoded by the image coding network enter the image decoding network, the order in which they arrive at the network layers in the image decoding network in sequence is fixed. For example, the image features will first pass through the fully connected layer, then the upsampling layer, the first convolutional layer, the activation function, the second convolutional layer, etc., and then output the final mask image. Therefore, the order of training the network layers can be determined according to the order of the network layers through which the image features pass.

[0040] Exemplarily, in the process of training the image segmentation model using the image to be annotated and the corresponding target mask image, the various network layers of the image decoding network of the image segmentation model are hierarchically trained based on the image to be annotated and the target mask image to obtain a graphic code annotation model. The graphic code annotation model is used to annotate the graphic code in the image. Optionally, annotating the graphic code in the image includes: segmenting the graphic code from the image; or obtaining the corner coordinates of the graphic code in the image.

[0041] In some embodiments, in order to make the final graphic code annotation model more applicable, for example, it can be used on a mobile terminal, the scale of the image segmentation model can be reduced, and a smaller graphic code annotation model can be trained to achieve local deployment.

[0042] It can be seen that in the embodiment of the present application, by processing the image to be annotated based on the prompt information and the image segmentation model to obtain the target mask image, it is possible to capture more complex patterns and relationships in image processing based on the large model, thereby improving the generation efficiency of training samples and increasing the data volume of training samples. Then, based on the annotated image and the target mask image, the various network layers of the image decoding network of the image segmentation model are layered trained to obtain a graphic code annotation model, which can improve the accuracy of training and reduce the influence of other network layers on the network layer during training, thereby affecting the accuracy and convergence rate of the network layer parameters. Moreover, compared with manually annotated graphic codes, the generated graphic code annotation model can directly annotate the graphic codes in the image, thereby improving the annotation efficiency of the graphic codes, and further improving the data volume and generation efficiency of the training samples required for the graphic code recognition model.

[0043] In some embodiments, since the structure of large models such as image segmentation models is relatively complex, during the hierarchical training of each network layer of the image decoding network in the image segmentation model, only the network layer that needs to be trained currently is trained, and other network layers are frozen. After the network layer that needs to be trained currently is trained, the next network layer that needs to be trained is unfrozen until all network layers are trained. That is to say, the network layer to be adjusted is determined from multiple network layers of the image decoding network to obtain a temporary segmentation model. Among them, the parameters of the network layer to be adjusted in the temporary segmentation model are variable, and the parameters of the network layer to be adjusted will be adjusted based on the input image to be annotated and the target mask image. The difference between the trained temporary segmentation model and the image segmentation model will not be too large. The parameters of the network layers other than the network layer to be adjusted in the temporary segmentation model will not change during training, so as to reduce the adverse effects on the training of the network layer to be adjusted, thereby improving the efficiency and reliability of the training graphic code annotation model. After obtaining the temporary segmentation model, the temporary segmentation model is trained based on the image to be annotated and the target mask image until the temporary segmentation model converges. In this way, the training of the temporary segmentation model corresponding to the current network to be adjusted is completed, and the training of the next network to be adjusted can be started.

[0044] Optionally, the learning rate of each network layer of the image decoding network of the image segmentation model for layered training can be 1e-7, and the optimizer can select AdamW.

[0045] It is understandable that during the training process, the image encoding network does not change.

[0046] In some embodiments, the image to be annotated can be processed based on the prompt information and the image segmentation model to obtain a large number of target mask images. The large number of target mask images obtained are divided into a training set, a validation set and a test set. Among them, the training set is used for model learning, the validation set is used to adjust the hyperparameters, and the test set is used to evaluate the performance of the final model. The trained graphic code annotation model or temporary segmentation model can use the test set to verify the performance. The evaluation indicators used here include IOU and key point distance. After the test set is verified, it can be deployed to the graphic code annotation software through trt quantization. Since model fine-tuning requires a large amount of data, the existing data may still not meet the iteration requirements. The amount of data can be increased by data enhancement methods, and almost hundreds of strategies such as random crop, randm erasing, cutout, etc. are used to enrich the breadth of the training data set.

[0047] As mentioned above, in order to enable the graphic code annotation model to run efficiently in environments with limited computing power such as mobile devices and embedded systems, the image segmentation model can be lightweight processed to obtain a lightweight coding network. The number of parameters of the lightweight coding network is smaller than that of the image segmentation model, the processing speed is fast, and the required memory is small. In some possible implementations, as shown in Figure 2, the embodiment of the present application also includes the following steps S204 to S208. Among them:

[0048] S202, performing lightweight processing on the image coding network in the image segmentation model to obtain a lightweight coding network.

[0049] The image coding network in the image segmentation model needs to extract image features, so the image coding network accounts for 90% of the image segmentation model. Therefore, in order to reduce the parameters of the image segmentation model, reducing the size of the image coding network is the key. For example, for the SAM model, the ViT model corresponding to the image coding network has more than 600M parameters, which is not conducive to running on mobile devices. Therefore, the image coding network in the image segmentation model can be lightweighted to obtain a lightweight coding network.

[0050] S204, inputting the training image into the image segmentation model, and obtaining a first image feature obtained by the image encoding network in the image segmentation model performing image encoding on the training image.

[0051] S206: Train the lightweight coding network based on the first image feature to obtain a trained lightweight coding network.

[0052] After obtaining the lightweight coding network, since the number of parameters of the model is reduced, in order to make the encoding result of the image by the lightweight coding network consistent with the encoding result of the image by the image segmentation model, the lightweight coding network can be trained based on the image segmentation model. That is, based on the image features obtained by extracting features of the input image by the image coding network in the image segmentation model, the image features obtained by extracting features of the input image by the image coding network in the lightweight coding network are trained, thereby adjusting the parameters of the image coding network in the lightweight coding network to obtain a trained lightweight coding network. That is, the training image is input into the image segmentation model to obtain the first image feature obtained by the image coding network in the image segmentation model encoding the training image, and then the lightweight coding network is trained based on the first image feature to obtain a trained lightweight coding network.

[0053] In some embodiments, training the lightweight feature based on the first image feature to obtain a trained lightweight coding network may include the following steps: inputting the training image corresponding to the first image feature into the lightweight coding network, and the image coding network in the lightweight coding network encodes the training image to obtain a second image feature corresponding to the training image; obtaining the loss between the first image feature and the second image feature; and training the lightweight coding network based on the loss to obtain a trained lightweight coding network.

[0054] In order to make the mask image of the image generated by the trained lightweight coding network consistent with the mask image of the image generated by the image segmentation model, the same training image can be input into the lightweight coding network and the image segmentation model respectively, and the difference between the image features extracted by the lightweight coding network and the image features extracted by the image segmentation model is obtained. Based on the difference, the parameters of the lightweight coding network are adjusted to obtain a trained lightweight coding network.

[0055] In some embodiments, the difference between the mask images generated by the two models can also be used to adjust the parameters of the lightweight coding network, or the parameters of the lightweight coding network can be adjusted based on the difference between the image features and the difference between the mask images.

[0056] S208, based on the image to be annotated and the target mask image, the image decoding network of the trained lightweight encoding network is trained to obtain a graphic code annotation model.

[0057] For example, after obtaining a trained lightweight coding network, the image decoding network of the trained lightweight coding network can be trained based on the image to be annotated and the target mask image to obtain a graphic code annotation model. The graphic code annotation model can be applied to systems with limited computing power, such as mobile devices and embedded systems.

[0058] It can be seen that in this embodiment, by comparing the differences between the image segmentation model and the lightweight coding network and training the lightweight coding network based on such differences, it is possible to ensure the accuracy of the model's image processing and the accuracy of the generated mask image while reducing the model size.

[0059] After the above steps, a graphic code annotation model can be obtained to segment the graphic code in the image from the image. Figure 3 As shown, in order to improve the richness of the annotation data, the information of the graphic code, such as corner point information, boundary information, etc., can be obtained based on the mask image of the image, which is also convenient for screening the accuracy of the graphic code annotation. For details, please refer to the following steps.

[0060] S302: Obtain an image to be annotated.

[0061] S304: Obtain a mask image of the image to be annotated based on the graphic code annotation model.

[0062] S306, extracting the target contour in the mask image, performing quadrilateral fitting processing on the target contour, and obtaining the graphic code corner points corresponding to the image to be annotated.

[0063] S308: Render the image to be annotated based on the graphic code corner points to annotate the image to be annotated with the graphic code.

[0064] In this embodiment, an image to be annotated is obtained, and the image to be annotated is input into the graphic code annotation model. Based on the graphic code annotation model, a mask image corresponding to the image to be annotated can be obtained. In the mask image, the values ​​of the pixel points of the graphic code are different from those of the other remaining pixels. Therefore, by analyzing the mask image, detailed information of the graphic code can be obtained. Specifically, the target contour in the mask image can be extracted, and a quadrilateral fitting is performed on the target contour to obtain the corner points of the graphic code corresponding to the image to be annotated. Among them, the target contour is the contour of the graphic code in the image to be annotated.

[0065] After obtaining the corner points of the graphic code, the pixels where the corner points of the graphic code are located can be determined, and then the image to be annotated can be rendered based on the corner points of the graphic code to annotate the image with the graphic code. For example, after determining the pixels where the corner points of the graphic code are located, the pixels where the corner points of the graphic code are located are set to a preset color to highlight the corner points of the graphic code. In the process of rendering the image to be annotated based on the corner points of the graphic code, the corner points of the graphic code can be highlighted. For example, the corner points of the graphic code can be displayed using red dots or red boxes. This can achieve automatic annotation, and it is also convenient for downstream task staff to quickly check whether the model has correctly identified the graphic code in the image based on the rendering results.

[0066] It can be seen that in this embodiment, a mask image of the image to be annotated is obtained based on the graphic code annotation model, and then the graphic code corner points in the image are highlighted based on the mask image, and the information of the graphic code corner points (such as their locations) is obtained. In this way, automatic annotation of the graphic codes in the image can be achieved, and the correctness of the model annotation can be detected easily.

[0067] In some embodiments, the number of network layers of the image coding network can be reduced, the vector length of the embedding layer output, and the dimension of the category embedding can be reduced, and the number of parallel processing heads in the self-attention module in the network layer of the image coding network can be reduced, so that the number of parameters of the lightweight coding network is less than a preset percentage of the parameters of the image coding network. The preset percentage can be, for example, 2%. For example, the number of parameters of the image coding network is about 632M, and the number of parameters of the lightweight coding network is about 13M. In this way, the lightweight coding network is obtained by reducing the number of parameters and computational complexity in the image segmentation model.

[0068] Optionally, a lightweight encoding network can be obtained in the following ways. (1) Reduce the embedding dimension: reduce the dimension of the patch embedding and reduce the number of features represented by each patch. Reduce the dimension of the category embedding and the position embedding. A "patch" refers to a series of small blocks into which the input image is divided. These small blocks are usually square and evenly distributed throughout the image. Each patch is regarded as a separate input unit, similar to the receptive field in a convolutional neural network. The input image is divided into patches of fixed size. For example, if the input image is 224x224 pixels and the size of each patch is 16x16 pixels, the image will be divided into 196 patches. Each patch is converted into a fixed-length vector through a linear layer (i.e., a fully connected layer). This linear layer is called a patch embedding layer. In order to preserve the spatial information between patches, position encoding is usually added after the patch embedding so that the model can understand the relative position of the patches in the original image. (2) Reduce the number of Transformer layers: reduce the number of layers in the Transformer encoder (i.e., reduce the number of attention modules and feedforward networks). Among them, Transformer is a deep learning architecture based on the self-attention mechanism, which is applied to computer vision tasks. The main components of the Transformer model include: self-attention mechanism (Self-Attention), multi-head attention (Multi-Head Attention), feed-forward neural network (Feed-Forward Neural Networks), layer normalization (Layer Normalization) and residual connection (Residual Connections). Among them, self-attention allows the model to dynamically focus on different parts of the input sequence when processing the input data. In ViT, this means that the model can learn the relationship between different patches in the image. Multi-head attention is an extension of self-attention. It divides the input data into multiple heads, each with its own attention mechanism. The outputs of these heads are then combined to form the final attention output. Feed-forward neural network After the self-attention layer, the Transformer model usually contains one or more feed-forward neural network layers, which further perform nonlinear transformations on the output of the attention layer. Layer normalization and residual connections are used to improve training stability and performance. (3) Reduce the number of attention heads: Reduce the number of heads in multi-head self-attention (Multi-Head Self-Attention) in each Transformer layer.(4) Reduce the patch size: Increase the patch size of image segmentation, which will reduce the number of patches and thus reduce the amount of data processed by the model. (5) Use lightweight attention mechanisms: For example, use linear attention or sparse attention mechanisms to replace the standard self-attention mechanism to reduce computational complexity. (6) Introduce depthwise separable convolutions: Introduce depthwise separable convolutions in the Transformer layer to replace or combine standard convolution operations to reduce the number of parameters. (6) Knowledge distillation: Use knowledge distillation techniques to transfer the knowledge of the image encoding network of a large image decomposition model to a small lightweight encoding network, so that the small lightweight encoding network can maintain similar prediction performance. (7) Parameter sharing and compression: Share certain parameters across layers or use lightweight network architectures with parameter sharing. Use techniques such as model pruning, quantization, and low-rank decomposition to compress the model. (8) Network architecture search (NAS): Use automatic machine learning techniques, such as network architecture search, to find model structures with fewer parameters and minimal performance loss.

[0069] It can be seen that in this embodiment, by reducing the parameters of the image coding network in the image decomposition model, the scale of the graphic code annotation model can be reduced, local deployment and operation can be achieved, the dependence of large models on hardware is reduced, and the risk of data leakage caused by cloud operation is avoided.

[0070] In a specific embodiment, in order to realize automatic labeling of graphic codes in mobile devices or embedded systems, the image coding network in the image segmentation model can be lightweight processed to obtain a lightweight coding network; the training image is input into the image segmentation model to obtain a first image feature obtained by the image coding network in the image segmentation model encoding the training image; the training image corresponding to the first image feature is input into the lightweight coding network, and the image coding network in the lightweight coding network encodes the training image to obtain a second image feature corresponding to the training image; the loss between the first image feature and the second image feature is obtained; and the lightweight coding network is trained based on the loss to obtain a trained lightweight coding network.

[0071] The network layer to be adjusted is determined from multiple network layers of the image decoding network in the trained lightweight coding network to obtain a temporary segmentation model, wherein the parameters of the network layer to be adjusted in the temporary segmentation model are variable, and the parameters of the network layers other than the network layer to be adjusted in the temporary segmentation model remain unchanged; the temporary segmentation model is trained based on the image to be annotated and the target mask image until the temporary segmentation model converges; the above steps are performed on the network layers in the image decoding network according to the preset network layer adjustment order to obtain a graphic code annotation model.

[0072] For example, the image decomposition model can be the SAM model. The image encoding network of the SAM model can be the ViT-H model. The Vit-H model used by the SAM model has more than 600M parameters and is very heavyweight. It can be replaced by Tiny-Vit. There are obvious differences between TinyViT and ViT-H in design concept, structural complexity, computing requirements and application scenarios.

[0073] The TinyViT model is a lightweight encoding network designed for resource-constrained devices, with the goal of running efficiently in environments with limited computing power, such as mobile devices and embedded systems. To achieve this goal, TinyViT adopts a hierarchical structure and a step-by-step downsampling strategy, which enables the model to gradually reduce the resolution of the input feature map while increasing the number of channels at different stages. This design not only effectively captures multi-scale features, but also significantly reduces the computational complexity and the number of parameters. Specifically, the TinyViT model has a smaller number of Transformer blocks, a lower embedding dimension, and uses fewer multi-head self-attention heads. For example, the embedding dimension varies from 64 to 512, and the number of attention heads ranges from 1 to 8. In addition, the TinyViT model further optimizes the computational efficiency of the model by introducing lightweight feedforward networks (FFNs) and regularization techniques such as DropPath at each stage. This enables the TinyViT model to maintain high accuracy in an environment with low computing resources, and is particularly suitable for applications with high real-time processing requirements, such as image classification and object detection on mobile devices. The ViT-H (Vision Transformer Huge) model is a large-scale model designed for high-performance computing environments to maximize the performance and accuracy of the model. ViT-H uses a flat structure, that is, all Transformer blocks process feature maps of the same resolution without downsampling. This design maintains global consistency and enables the model to process high-resolution images, but it also greatly increases the computational requirements. The ViT-H model has more Transformer blocks and higher embedding dimensions, usually on the order of 1280 or higher, and uses a large number of multi-head self-attention heads to capture the rich features and global context information of the image. In addition, the feedforward network (FFN) of the ViT-H model also has a very large hidden layer dimension, which enables the model to learn and represent more complex features. However, this also means that the ViT-H model requires a lot of computing resources for training and inference, usually running on a high-performance computing cluster with multiple TPUs or GPUs. The ViT-H model is able to solve complex visual tasks on large-scale datasets, such as image generation, large-scale image classification, etc., so it is mainly used in high-end visual applications in research and industry. The TinyViT model emphasizes efficient operation in an environment with limited computing resources, and achieves lightweight and high efficiency by streamlining the model structure and optimizing the computing process, while ViT-H pursues the highest model accuracy and performance, and copes with the most demanding visual tasks by increasing the model size and computational complexity. TinyViT is suitable for mobile devices and edge computing, while ViT-H is suitable for high-performance computing and large-scale visual tasks.

[0074] During the training process, the original Vit-H model can be directly changed to the TinyViT model. Then the model can be retrained. However, since the encoder (image encoding network) and decoder (image decoding network) need to be coupled and optimized, it is difficult to achieve the desired effect with direct training. Figure 4 As shown in Figure 2, in order to make the model converge faster and reduce the difficulty of iteration, the knowledge distillation technology is used during training, and the original SAM model is used as the first model ( Figure 4 The teacher model shown in the figure), and the lightweight SAM model as the second model ( Figure 4 student model shown). At the same time, a divide-and-conquer approach is adopted, and the parameters of the mask decoder are fixed in the second model and the first model to achieve decoupling of the encoder and decoder. The image features output by the Vit model calculate the loss of the image features output by the studet model; this loss weights the difference with the output mask image as the loss of the model iteration. At the same time, since there is a certain randomness in the selection of prompt information, we record the selection of prompt information (prompt) of the first model each time, and then the second model also uses the same operation to ensure the consistency of the two random operations. Through this training method, only 1% of the data in the SAM training set is used. Through 8 iterations in a single GPU RTX3090, the lightweight training of the model is completed in less than a day, thereby improving the training efficiency, and then the accuracy and precision of the annotation can be improved. In this embodiment, based on the large model, a wider range of image data features can be learned during the training process, so that the generalization ability is strong. In addition, the target mask image is obtained based on the prompt information and the image segmentation model, that is, the training data is obtained based on the large model, which can conveniently obtain a large amount of training data, so that in the subsequent training of the graphic code annotation model, it is not easy to overfit to a specific data set, so that good performance can be maintained in different application scenarios.

[0075] After obtaining the trained lightweight encoding network (graphic code annotation model), Figure 5 As shown, the graphic code annotation model can be used to obtain the mask image of each image to be annotated. Then, the corner point annotation information of the graphic code is obtained based on these mask images, so that the annotation degree can be further enriched and the annotation type can be expanded. In addition, the image to be annotated can be rendered based on the corner points of the graphic code to annotate the image to be annotated with the graphic code. This makes it easy to manually review whether the annotation is accurate based on the rendered pre-annotation results. If inaccurate, modify the annotation position. The output JSON file includes the image to be annotated and the corresponding corner point information and mask image. This facilitates the completion of subsequent downstream tasks.

[0076] It should be understood that, although the steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0077] Based on the same inventive concept, the embodiment of the present application also provides a training device for a graphic code annotation model. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more graphic code annotation models provided below can refer to the limitations of the training method for the graphic code annotation model above, and will not be repeated here.

[0078] like Figure 6 As shown, the embodiment of the present application provides a training device 600, including:

[0079] The mask image acquisition module 601 is used to process the image to be annotated based on the prompt information and the image segmentation model to obtain a target mask image, wherein the prompt information is used to specify the graphic code in the image to be annotated, and the target mask image is used to segment the graphic code from the image to be annotated;

[0080] The training module 602 is used to hierarchically adjust the various network layers of the image decoding network of the image segmentation model based on the image to be annotated and the target mask image to obtain the graphic code annotation model, and the graphic code annotation model is used to annotate the graphic code in the image.

[0081] In some embodiments, in terms of training the image decoding network, the training module 602 is specifically used to: determine the network layer to be adjusted from multiple network layers of the image decoding network to obtain a temporary segmentation model, wherein the parameters of the network layer to be adjusted in the temporary segmentation model are variable, and the parameters of the network layers other than the network layer to be adjusted in the temporary segmentation model remain unchanged; train the temporary segmentation model based on the image to be annotated and the target mask image until the temporary segmentation model converges; and perform the above steps on the network layers in the image decoding network according to a preset network layer adjustment order to obtain the graphic code annotation model.

[0082] In some embodiments, the training device 600 also includes a lightweight model processing device, which is specifically used to: perform lightweight processing on the image coding network in the image segmentation model to obtain a lightweight coding network; input the training image into the image segmentation model to obtain a first image feature obtained by the image coding network in the image segmentation model encoding the training image; train the lightweight coding network based on the first image feature to obtain a trained lightweight coding network; and train the image decoding network of the trained lightweight coding network based on the image to be labeled and the mask information to obtain the graphic code labeling model.

[0083] In some embodiments, the lightweight model processing device is also used to input the training image corresponding to the first image feature into the lightweight coding network, and the image coding network in the lightweight coding network encodes the training image to obtain the second image feature corresponding to the training image; obtains the loss between the first image feature and the second image feature; and trains the lightweight coding network based on the loss to obtain the trained lightweight coding network.

[0084] In some embodiments, the prompt information includes at least one of a dot, a rectangular box, and text.

[0085] In some embodiments, the embodiments of the present application also include a labeling device, which is specifically used to obtain an image to be labeled; obtain a mask image of the image to be labeled based on the graphic code labeling model; extract the target contour in the mask image, perform quadrilateral fitting on the target contour, and obtain the graphic code corner points corresponding to the image to be labeled; render the image to be labeled based on the graphic code corner points to perform graphic code labeling on the image to be labeled.

[0086] Each module in the training device of the graphic code annotation model can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0087] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG7. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in the training method of the above-mentioned graphic code annotation model are implemented.

[0088] Those skilled in the art will understand that the structure shown in FIG. 7 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0089] In some embodiments, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0090] In some embodiments, Figure 8 The figure shows an internal structure diagram of a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0091] In some embodiments, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0092] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0093] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0094] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0095] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for a graphic code annotation model, characterized in that: include: Processing the image to be annotated based on the prompt information and the image segmentation model to obtain a target mask image, wherein the prompt information is used to specify the graphic code in the image to be annotated, and the target mask image is used to segment the graphic code from the image to be annotated; Based on the image to be annotated and the target mask image, each network layer of the image decoding network of the image segmentation model is hierarchically trained to obtain the graphic code annotation model, and the graphic code annotation model is used to annotate the graphic code in the image.

2. The method according to claim 1, characterized in that The step of performing hierarchical training on each network layer of the image decoding network of the image segmentation model based on the image to be annotated and the target mask image to obtain the graphic code annotation model includes: Determine a network layer to be adjusted from multiple network layers of the image decoding network, and obtain a temporary segmentation model, wherein parameters of the network layer to be adjusted in the temporary segmentation model are variable, and parameters of network layers other than the network layer to be adjusted in the temporary segmentation model remain unchanged; Training the temporary segmentation model based on the image to be annotated and the target mask image until the temporary segmentation model converges; The above steps are performed on the network layers in the image decoding network according to a preset network layer adjustment sequence to obtain the graphic code annotation model.

3. The method according to claim 1, characterized in that The method further comprises: Performing lightweight processing on the image coding network in the image segmentation model to obtain a lightweight coding network; Inputting a training image into the image segmentation model to obtain a first image feature obtained by an image encoding network in the image segmentation model performing image encoding on the training image; Training the lightweight coding network based on the first image feature to obtain a trained lightweight coding network; Based on the image to be annotated and the target mask image, the image decoding network of the trained lightweight encoding network is trained to obtain the graphic code annotation model.

4. The method according to claim 3, characterized in that The step of training the lightweight coding network based on the first image feature to obtain a trained lightweight coding network includes: Inputting the training image corresponding to the first image feature into the lightweight coding network, and the image coding network in the lightweight coding network encodes the training image to obtain a second image feature corresponding to the training image; Obtaining a loss between the first image feature and the second image feature; The lightweight coding network is trained based on the loss to obtain the trained lightweight coding network.

5. The method according to claim 3, characterized in that: The step of performing lightweight processing on the image coding network in the image segmentation model to obtain a lightweight coding network includes: Reduce the number of network layers of the image coding network, reduce the vector length of the embedding layer output, and reduce the number of parallel processing heads in the self-attention module in the network layer of the image coding network, so that the number of parameters of the obtained lightweight coding network is less than a preset percentage of the number of parameters of the image coding network.

6. The method according to claim 1, characterized in that The prompt information includes at least one of a point, a rectangular box, and text.

7. The method according to claim 1, characterized in that The step of obtaining the graphic code annotation model includes: Get the image to be annotated; Obtaining a mask image of the image to be annotated based on the graphic code annotation model; Extracting the target contour in the mask image, performing quadrilateral fitting processing on the target contour, and obtaining the graphic code corner points corresponding to the image to be annotated; The image to be annotated is rendered based on the graphic code corner points to annotate the image to be annotated with graphic codes.

8. A training device for a graphic code annotation model, characterized in that: include: A mask image acquisition module, used for processing the image to be annotated based on the prompt information and the image segmentation model to obtain a target mask image, wherein the prompt information is used to specify the graphic code in the image to be annotated, and the target mask image is used to segment the graphic code from the image to be annotated; A training module is used to hierarchically adjust the various network layers of the image decoding network of the image segmentation model based on the image to be annotated and the target mask image to obtain the graphic code annotation model, and the graphic code annotation model is used to annotate the graphic code in the image.

9. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images