Model training method and device, computer equipment and storage medium

Abstract

The invention discloses a model training method and device, computer equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: a first device obtains a target weight matrix corresponding to each sample image from a plurality of target weight matrixes; inputting the image sequence corresponding to each sample image and the target weight matrix corresponding to each sample image into a feature extraction module in the initial model to obtain a global image feature corresponding to each sample image; and training a feature extraction module and a task processing module of the initial model based on the global image feature corresponding to each sample image to obtain a target model. Thus, in the model training process, the weight matrix does not need to be calculated in real time based on an attention mechanism in the training process, namely, the number of times of calculation of the attention weight matrix in the training process is reduced, and then the efficiency and speed of model training on the first device are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, computer equipment and storage medium. Background Art

[0002] In the related art, when training a model for a network structure based on the self-attention mechanism, there will be a large number of attention (i.e., softmax function) calculations to obtain the weight matrix required for extracting global features in a long sequence. However, due to the high complexity of the softmax function calculation, the efficiency and speed of model training are low. Summary of the invention

[0003] The present application proposes a model training method, apparatus, computer equipment and storage medium to improve the accuracy of model training.

[0004] In a first aspect, an embodiment of the present application provides a model training method, which is applied to a first device, and the method includes: dividing each sample image in an image sample set into multiple sub-images to obtain an image sequence corresponding to each sample image; obtaining a target weight matrix corresponding to each sample image from multiple target weight matrices, wherein the matrix parameters in the target weight matrix are used to characterize the degree of association between the multiple sub-images in the image sequence, and the multiple target weight matrices are pre-calculated by a second device; inputting the image sequence corresponding to each sample image and the target weight matrix corresponding to each sample image into a feature extraction module in an initial model to obtain a global image feature corresponding to each sample image; based on the global image feature corresponding to each sample image, training the feature extraction module and the task processing module of the initial model to obtain a target model, wherein the initial model is a network structure based on a self-attention mechanism.

[0005] In a second aspect, an embodiment of the present application provides a model training method, which is applied to a second device, and the method includes: obtaining a sample image set; splitting each sample image in the sample image set into multiple sub-images to obtain an image sequence corresponding to each sample image; determining a target weight matrix corresponding to each sample image based on the degree of association between the sub-images in the image sequence corresponding to each sample image, wherein the target weight matrix is ​​used in a process in which the first device trains an initial model based on the sample image, and the image sequence corresponding to the sample image is input into a feature extraction module in the initial model to obtain a global image feature corresponding to the sample image, and the initial model is a network structure based on a self-attention mechanism.

[0006] In the third aspect, the embodiment of the present application provides a model training device, which is applied to the first device, and the device includes: an image segmentation module, a weight acquisition module, a feature extraction module, a feature extraction module and a model training module. Among them, the image segmentation module is used to divide each sample image in the image sample set into multiple sub-images to obtain an image sequence corresponding to each sample image; the weight acquisition module is used to obtain a target weight matrix corresponding to each sample image from multiple target weight matrices, and the matrix parameters in the target weight matrix are used to characterize the degree of association between the multiple sub-images in the image sequence, and the multiple target weight matrices are pre-calculated by the second device; the feature extraction module is used to input the image sequence corresponding to each sample image and the target weight matrix corresponding to each sample image into the feature extraction module in the initial model to obtain the global image features corresponding to each sample image; the model training module is used to train the feature extraction module and the task processing module of the initial model based on the global image features corresponding to each sample image to obtain the target model, and the initial model is a network structure based on the self-attention mechanism.

[0007] In a fourth aspect, an embodiment of the present application provides a model training device, which is applied to a second device, and the device includes: a sample acquisition module, an image segmentation module, and a weight calculation module. Among them, the sample acquisition module is used to acquire a sample image set; the image segmentation module is used to split each sample image in the sample image set into multiple sub-images to obtain an image sequence corresponding to each sample image; the weight calculation module is used to determine the target weight matrix corresponding to each sample image based on the degree of association between the sub-images in the image sequence corresponding to each sample image. The target weight matrix is ​​used in the process of training an initial model based on the sample image in the first device, and the image sequence corresponding to the sample image is input into the feature extraction module in the initial model to obtain the global image feature corresponding to the sample image. The initial model is a network structure based on the self-attention mechanism.

[0008] In a fifth aspect, an embodiment of the present application provides a computer device, comprising: one or more processors; a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above-mentioned method.

[0009] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the above method.

[0010] In the solution provided by the present application, the first device can divide each sample image in the image sample set into multiple sub-images to obtain an image sequence corresponding to each sample image; obtain a target weight matrix corresponding to each sample image from multiple target weight matrices, wherein the matrix parameters in the target weight matrix are used to characterize the degree of association between the multiple sub-images in the image sequence, and the multiple target weight matrices are pre-calculated by the second device; input the image sequence corresponding to each sample image and the target weight matrix corresponding to each sample image into the feature extraction module in the initial model to obtain the global image feature corresponding to each sample image; based on the global image feature corresponding to each sample image, the feature extraction module and the task processing module of the initial model are trained to obtain the target model, and the initial model is a network structure based on the self-attention mechanism. In this way, in the process of model training, the weight matrix corresponding to each sample image is called from the multiple target weight matrices pre-calculated by the second device to extract the global image feature, and there is no need to calculate the weight matrix in real time based on the attention mechanism during the training process, that is, the number of calculations of the attention weight matrix during the training process is reduced, thereby improving the efficiency and speed of model training on the first device. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A schematic diagram of a process for extracting global image features provided by the present application is shown.

[0013] Figure 2 A flow chart of a model training method provided in one embodiment of the present application is shown.

[0014] Figure 3 A schematic diagram of a model structure for model training provided in an embodiment of the present application is shown.

[0015] Figure 4 Shows Figure 2 A flowchart of the sub-steps of step S130 in one implementation manner.

[0016] Figure 5 A schematic diagram of another process of extracting global image features provided in an embodiment of the present application is shown.

[0017] Figure 6 Shows Figure 2 A flowchart of the sub-steps of step S140 in one implementation manner.

[0018] Figure 7 A flow chart of a model training method provided in another embodiment of the present application is shown.

[0019] Figure 8 A schematic diagram of the architecture of a model training method provided in an embodiment of the present application is shown.

[0020] Fig. 9 It is a block diagram of a model training device provided according to an embodiment of the present application.

[0021] Fig.10 It is a block diagram of a model training device provided according to another embodiment of the present application.

[0022] Fig.11 It is a block diagram of a computer device according to an embodiment of the present application for executing the model training method according to an embodiment of the present application.

[0023] Fig.12 It is a storage unit of an embodiment of the present application for storing or carrying program codes for implementing the model training method according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0025] It should be noted that in some processes described in the specification, claims and the above-mentioned drawings of the present application, multiple operations that appear in a specific order are included, and these operations may not be executed or executed in parallel in the order in which they appear in this article. The sequence numbers of operations such as S110, S120, etc. are only used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or less operations, and these operations may be executed in sequence or in parallel. And, the terms "first", "second", etc. in the specification, claims and the above-mentioned drawings of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or server that includes a series of steps or sub-modules is not necessarily limited to those steps or sub-modules explicitly listed, but may include other steps or sub-modules that are not explicitly listed or that are inherent to these processes, methods, products, or devices.

[0026] Among related technologies, Transformer is a network structure based on a unique attention mechanism that can establish better feature expressions; and its standard input data flow mode is also helpful for coordination with other modal data. However, Transformer also has the following problems:

[0027] Unfriendly to Neural Network Processing Uni (NPU) architecture: Transformer is not particularly user-friendly for computing and deployment on mobile devices, especially NPU. As a chip specially designed to process neural networks, NPU is good at matrix calculation multiplication and addition. However, the attention module in the traditional Transformer has a large number of softmax calculation processes. Figure 1 As shown, in the attention module, the attention weight matrix is ​​calculated based on the softmax function, matrix Q and matrix K, for example, Then, the calculated attention weight matrix and matrix V are input into the feature aggregation unit for matrix multiplication to obtain the final extracted global image features. Obviously, in the calculation process of the attention weight matrix, the NPU is not suitable for the large number of division calculations in softmax, and the complexity of softmax exponential calculation is high, which will affect the efficiency and speed of the NPU inference task in Transformer; on the other hand, the existence of softmax leads to the complexity of attention calculation O(n 2), which greatly increases the computational cost of training in the case of long sequences. In summary, the existence of softmax is very unfavorable for the deployment of the Transformer model on mobile devices or computing devices with NPU.

[0028] The inventors proposed a model training method, apparatus, computer equipment and storage medium. During the model training process, the process of attention calculation is replaced by the process of reading the weight matrix, thereby reducing the number of softmax calculations during the model training process for a network structure based on the attention mechanism, and improving the speed and efficiency of model training.

[0029] Please refer to Figure 2 , Figure 2 A flow chart of a model training method provided in an embodiment of the present application is applied to a first device. Figure 2 The model training method provided in the embodiment of the present application is described in detail. The model training method may include the following steps:

[0030] Step S110: Divide each sample image in the image sample set into a plurality of sub-images to obtain an image sequence corresponding to each sample image.

[0031] In this embodiment, the first device may be an electronic device with a data processing function configured with an NPU, and the electronic device includes but is not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3) player, and an MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) player, etc. Of course, the first device may also be an electronic device with a data processing function configured with other processors, and the other processors include but are not limited to the brain processor (Brain Processing Unit, BPU) and the deep learning processor (Deep-Learning Processing Unit, DPU) and other processors, and this embodiment does not limit this.

[0032] Optionally, the initial model used for model training in the present application is a network structure based on the attention mechanism, for example, a vision transformer (ViT) model designed based on the transformer network structure. Since the input of the transformer network structure requires a sequence format, it cannot accept two-dimensional image data as input. Therefore, after the first device obtains the image sample set, it first divides each sample image in the image sample set into multiple sub-images to obtain an image sequence corresponding to each sample image. For example, Figure 3 As shown, a sample image can be divided into 9 sub-images, and after the 9 sub-images are arranged in a certain order, an image sequence corresponding to each sample image can be obtained. Of course, in practical applications, it is not necessary to divide the sample image into 9 sub-images, and it can also be divided into other numbers of sub-images. For example, if the sample image is an image with a resolution of 224*224, the sample image can be divided into 14 sub-images, and the resolution of each sub-image is 16*16. It can be understood that the number of sub-images into which the sample image is divided can be determined based on the computing power of the first device for a long sequence. If the computing power of the first device for a long sequence is strong, a larger number of sub-images can be divided to obtain a longer image sequence, so as to extract more detailed and comprehensive global image features based on the longer image sequence; if the computing power of the first device for a long sequence is weak, a smaller number of sub-images can be divided to obtain a shorter image sequence, so that the first device can process the image sequence based on the current computing power to successfully extract the global image features.

[0033] Step S120: Obtain a target weight matrix corresponding to each sample image from a plurality of target weight matrices, wherein matrix parameters in the target weight matrix are used to characterize the degree of association between the plurality of sub-images in the image sequence, and the plurality of target weight matrices are pre-calculated by the second device.

[0034] In this embodiment, the target weight matrix is ​​the aforementioned attention weight matrix calculated for the matrix Q and the matrix K, wherein the multiple target weight matrices are pre-calculated by the second device, and the first device can obtain the multiple target weight matrices from the second device before model training, and store the multiple target weight matrices. The multiple target weight matrices correspond one-to-one to all sample images in the image sample set, that is, there is a mapping relationship between the multiple target weight matrices and all sample images in the image sample set, and then the first device can quickly call the target weight matrix corresponding to any sample image from the multiple target weight matrices according to the mapping relationship.

[0035] Step S130: inputting the image sequence corresponding to each sample image and the target weight matrix corresponding to each sample image into the feature extraction module in the initial model to obtain the global image feature corresponding to each sample image.

[0036] Furthermore, after obtaining the image sequence corresponding to each sample image and the target weight matrix corresponding to each sample image. For each sub-image, it can be regarded as a Patch, and its data format is a three-dimensional matrix, which does not conform to the input format of the Transformer model. Therefore, each sub-image can be transformed through an Embedding layer, and each sub-image can be mapped to a one-dimensional vector through linear mapping. Taking each sub-image as [16,16,3] as an example, after mapping through the Embedding layer, a vector with a length of 768 can be obtained. This vector with a length of 768 is also called a token. In this way, a one-dimensional feature vector corresponding to each sub-image can be obtained, and multiple one-dimensional feature vectors corresponding to multiple sub-images can constitute the first image feature sequence corresponding to the sample images corresponding to the multiple sub-images.

[0037] In other words, after obtaining the image sequence corresponding to each sample image, the image features of each sub-image in the image sequence corresponding to each sample image (i.e., the aforementioned one-dimensional feature vector) can be extracted to obtain the first image feature sequence corresponding to each sample image. In addition, it is necessary to add a category block, i.e., a class token, to the head of the first image feature sequence, and then embed the position code for the first image feature sequence with the added category block.

[0038] Furthermore, the first image feature sequence corresponding to each sample image and the sample weight matrix corresponding to each sample image are input into the feature extraction module in the initial model to obtain the second image feature sequence corresponding to each sample image as the global image feature corresponding to each sample image. The global image feature is the output feature corresponding to the category block.

[0039] In some embodiments, the feature extraction module is an encoding module based on a self-attention mechanism, and the encoding module includes a key value acquisition unit and a feature aggregation unit, see Figure 4 , step S130 may include the following contents from step S131 to step S132:

[0040] Step S131: inputting the first feature sequence corresponding to each sample image into the key value acquisition unit to obtain the key value matrix corresponding to each sample image.

[0041] In this embodiment, the key value matrix corresponding to each sample image is the aforementioned matrix V. After the first feature sequence corresponding to each sample image is input into the key value acquisition unit, the first feature sequence and the weight matrix W are combined. v By performing matrix multiplication (i.e., linear transformation), we can obtain the key-value matrix V corresponding to each sample image.

[0042] Step S132: Input the key-value matrix corresponding to each sample image and the sample weight matrix corresponding to each sample image into the feature aggregation unit to obtain a second feature sequence corresponding to each sample image as the global image feature corresponding to each sample image. The feature aggregation unit is used to perform weighted processing on the matrix parameters in the key-value matrix based on the sample weight matrix.

[0043] Furthermore, the key-value matrix corresponding to each sample image and the sample weight matrix corresponding to each sample image are input into the feature aggregation unit. In the feature aggregation unit, the key-value matrix corresponding to each sample image and the sample weight matrix corresponding to each sample image are matrix multiplied, that is, the matrix parameters in the key-value matrix are weighted to obtain the second feature sequence corresponding to each sample image as the global image feature corresponding to each sample image.

[0044] See also Figure 5 In this application, the sample weight matrices used to input into the feature aggregation unit to extract global image features are all called from multiple target weight matrices, not as Figure 1 The result is calculated in real time by the softmax function. Therefore, the number of calculations of the attention weight matrix during the training process is reduced, thereby improving the efficiency and speed of model training on the first device.

[0045] Step S140: Based on the global image features corresponding to each sample image, the feature extraction module and the task processing module of the initial model are trained to obtain a target model, wherein the initial model is a network structure based on a self-attention mechanism.

[0046] In some embodiments, each image sample carries first label information corresponding to the target image processing task, see Figure 6 , step S140 may include the following contents from step S141 to step S143:

[0047] Step S141: input the global image features corresponding to each sample image into the task processing module in the initial model to obtain the second label information of the target image processing task for each sample image, and the task processing module is used to execute the target image processing task.

[0048] In this embodiment, the target image processing task may be an image classification task, or other image processing tasks, which is not limited in this embodiment. The global image features corresponding to each sample image are input into the task processing module of the initial model, and the task processing module analyzes the global image features and then outputs the second label information of the target image processing task for each sample image.

[0049] Step S142: determining a target loss value based on the degree of difference between the first label information and the second label information of each sample image.

[0050] Step S143: Based on the target loss value, iteratively update the network parameters of the feature extraction module and the task processing module in the initial model until the model training conditions are met, and obtain the trained initial model as the target model.

[0051] Furthermore, after obtaining the second label information of the target image processing task for each sample image, the difference between the first label information and the second label information of each sample image can be calculated according to a preset loss function to determine the target loss value. The preset loss function includes but is not limited to a hinge loss function, a cross entropy loss function, and an exponential loss function. The network parameters of the feature extraction module and the task processing module in the initial model are adjusted according to the detected target loss value. At the same time, as the network parameters are adjusted, the initial model can learn as many and more comprehensive global image features as possible. Finally, until the target loss value is detected to meet the model training conditions, the trained initial model is obtained as the target model.

[0052] Among them, the model training conditions can be: the detection target loss value is less than the preset value, the detection target loss value no longer changes, or the number of training times reaches the preset number of times, etc. It can be understood that after the initial model is iteratively trained for multiple training cycles according to the image sample set, wherein each training cycle includes multiple iterative trainings, the parameters in the initial model are continuously optimized, so that the above detection target loss value becomes smaller and smaller, and finally becomes a fixed value, or is less than the above preset value. At this time, it means that the initial model has converged; of course, it can also be determined that the initial model has converged after the number of training times reaches the preset number of times. At this time, the initial model can be used as the above target model. Among them, the preset value and the preset number of times are pre-set, and their values ​​can also be adjusted according to different application scenarios. This embodiment does not limit this.

[0053] In this embodiment, during the model training process, the weight matrix corresponding to each sample image is called from multiple target weight matrices pre-calculated by the second device to extract global image features. There is no need to calculate the weight matrix in real time based on the attention mechanism during the training process, which reduces the number of calculations of the attention weight matrix during the training process, thereby improving the efficiency and speed of model training on the first device.

[0054] Please refer to Figure 7 , Figure 7 A flow chart of a model training method provided in another embodiment of the present application is applied to a second device. Figure 7 The model training method provided in the embodiment of the present application is described in detail. The model training method may include the following steps:

[0055] Step S210: Acquire a sample image set.

[0056] In this embodiment, the second device can be an electronic device with strong computing power or a server. The server can be an independent physical server or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), as well as basic cloud computing services such as big data and artificial intelligence platforms. This embodiment does not limit this.

[0057] Optionally, combined Figure 8 The architectural diagram shown explains the present application, and the second device may include a data preprocessing module, a segmentation and distillation feature processing module, and a self-attention calculation module. The data preprocessing module can obtain the original image data set and divide it into a training data set and a verification data set, which correspond to the subsequent training process and verification process respectively. In addition to being loaded into the Dataloader module for a series of data enhancement processes such as re-cropping and rotation in preparation for training, the training data set will also be copied and sent to the segmentation and distillation feature processing module for feature extraction. Among them, the proportion of the training data set and the verification data set compared to the original image data set may be 0.9 and 0.1, respectively. It can be understood that the sample image set may include the aforementioned training data set and verification data set.

[0058] Furthermore, the segmentation and distillation feature processing module may include a segmentation unit and a classification unit. After acquiring the training data set, the segmentation unit may identify the objects contained in each sample image. If multiple types of objects are identified, a corresponding color is added to each sample image for each type of object, and each sample image after adding color is obtained. Different types of objects have different corresponding colors added. In this way, objects of the same type in the sample image can have the same color gamut and clearer contours, and objects of different types in the sample image can have different color gamuts. Among them, if the probability of identifying an object of the target type contained in any sample image is lower than a threshold, it is determined that any sample image does not contain an object of the target type, and the sample image is discarded.

[0059] In some embodiments, to avoid excessive data volume, the segmentation and distillation feature processing module can filter out a target number of sample images from the training data set for color addition operations, for example, randomly identifying objects contained in 10% of the sample images.

[0060] Step S220: split each sample image in the sample image set into a plurality of sub-images to obtain an image sequence corresponding to each sample image.

[0061] Specifically, the same parameter index as in the ViT training process is used to split each sample image after adding color into multiple sub-images to obtain an image sequence corresponding to each sample image. In layman's terms, the sample image is split according to the same parameter index during the model training process and the processing process of the segmentation and distillation feature processing module. Only in this way can it be ensured that the target weight matrix pre-calculated by the second device can be adapted to the weight matrix required by the first device during the model training process, thereby ensuring the effect of the model training.

[0062] Furthermore, the classification unit can obtain the pixel value at the target pixel position in each of the sub-images in the image sequence corresponding to each of the sample images, and obtain the reference pixel value corresponding to each of the sub-images. The target pixel position can be the center point position of each sub-image, or any other pixel position, which is not limited in this embodiment. At this time, the pixel value at the center point position of each sub-image can be used to characterize the category of the sub-image. Based on this, a list named attention_class can be established, and sub-images of different classes in the same image sequence can be classified and stored in the attention_class list.

[0063] Based on this, the self-attention calculation module can determine the similarity between each two sub-images in the image sequence corresponding to each sample image based on the reference pixel value corresponding to each sub-image, as the degree of association between each two sub-images, and obtain the degree of association between sub-images in the image sequence corresponding to each sample image. Specifically, if the pixel values ​​of the center points of any two sub-images in the image sequence are the same, it can be determined that the two sub-images with the same pixel values ​​are of the same type, and the similarity between the two is determined to be 99.9%, that is, 99.9% is used as the degree of association between the two sub-images; if the pixel values ​​of the center points of any two sub-images in the image sequence are different, it can be determined that the two sub-images with the same pixel values ​​are of different types, that is, the similarity between the two is determined to be 0.1%, that is, 0.1% is used as the degree of association between the two sub-images. In this way, the introduction of a 0% association degree can be avoided to cause data non-smoothing, and the occurrence of problems such as possible calculation errors of subsequent data.

[0064] Step S230: Based on the degree of association between sub-images in the image sequence corresponding to each sample image, determine the target weight matrix corresponding to each sample image, and the target weight matrix is ​​used in the process of training an initial model based on the sample image on the first device. The image sequence corresponding to the sample image is input into the feature extraction module in the initial model to obtain the global image feature corresponding to the sample image, and the initial model is a network structure based on the self-attention mechanism.

[0065] Furthermore, after calculating the degree of association between the sub-images in the image sequence corresponding to each sample image, a matrix named DisAttn_matrix can be established, the size of which can be the number of samples * n 2 *n 2 , where n is the number of sub-images divided for a sample image. For example, if the sample image is an image with a resolution of 224*224, the sample image can be divided into 14 sub-images, in which case n is 196. The degree of association between the sub-images in the image sequence corresponding to each sample image calculated above is stored in the DisAttn_matrix matrix, and the DisAttn_matrix matrix is ​​converted into matrix information with the same structure as the traditional attention, namely the DisAttn matrix, to obtain the target weight matrix corresponding to each sample image. Optionally, optimization can also be performed by adjusting the ratio of related sequences to eliminate redundant information within the DisAttn_Matrix.

[0066] In this way, when the second device calculates the target weight matrix corresponding to each sample image in advance and obtains multiple target weight matrices, the first device can directly call the target weight matrix corresponding to each sample image from the multiple target weight matrices during the training of the initial model, and combine the image sequence corresponding to each sample image to accurately extract the global image features corresponding to each sample image. The specific implementation method of the first device training the initial model can refer to the content in the aforementioned embodiment, which will not be repeated here.

[0067] In this embodiment, feature information of a certain type of image is extracted through segmentation and distillation to assist the attention training process. Because the distilled feature information itself already has some global features, it gives the model an effect similar to pre-training. This allows the training process to converge faster, thereby obtaining better and more stable results. In addition, the first device does not need to calculate the weight matrix in real time based on the attention mechanism during the training process, which reduces the number of calculations of the attention weight matrix during the training process, thereby improving the efficiency and speed of model training on the first device.

[0068] Please refer to Fig. 9 , which shows a structural block diagram of a model training device 300 provided in an embodiment of the present application, which is applied to a first device. The device 300 may include: an image segmentation module 310, a weight acquisition module 320, a feature extraction module 330 and a model training module 340.

[0069] The image splitting module 310 is used to divide each sample image in the image sample set into a plurality of sub-images to obtain an image sequence corresponding to each sample image.

[0070] The weight acquisition module 320 is used to obtain a target weight matrix corresponding to each sample image from multiple target weight matrices, wherein the matrix parameters in the target weight matrix are used to characterize the degree of association between the multiple sub-images in the image sequence, and the multiple target weight matrices are pre-calculated by the second device.

[0071] The feature extraction module 330 is used to input the image sequence corresponding to each sample image and the target weight matrix corresponding to each sample image into the feature extraction module in the initial model to obtain the global image feature corresponding to each sample image.

[0072] The model training module 340 is used to train the feature extraction module and the task processing module of the initial model based on the global image features corresponding to each sample image to obtain a target model. The initial model is a network structure based on the self-attention mechanism.

[0073] In some embodiments, the model training device 300 may further include: a first feature sequence extraction module. The first feature sequence extraction module may be used to extract the image features of each sub-image in the image sequence corresponding to each sample image, and obtain the first image feature sequence corresponding to each sample image, before inputting the image sequence corresponding to each sample image and the sample weight matrix corresponding to each sample image into the feature extraction module in the initial model to obtain the global image features corresponding to each sample image.

[0074] In this manner, the feature extraction module 330 can be specifically used to input the first image feature sequence corresponding to each sample image and the sample weight matrix corresponding to each sample image into the feature extraction module in the initial model to obtain the second image feature sequence corresponding to each sample image as the global image feature corresponding to each sample image.

[0075] In some embodiments, the feature extraction module is an encoding module based on a self-attention mechanism, and the encoding module includes a key value acquisition unit and a feature aggregation unit. The key value acquisition unit is used to extract a key value matrix corresponding to each sample image based on a first feature sequence corresponding to each sample image. The feature aggregation unit is used to perform weighted processing on matrix parameters in the key value matrix based on a sample weight matrix corresponding to each sample image and a sample weight matrix corresponding to each sample image, to obtain a second feature sequence corresponding to each sample image as a global image feature corresponding to each sample image.

[0076] In some embodiments, each image sample carries first label information corresponding to a target image processing task, and the model training module 340 can be specifically used to: input the global image features corresponding to each sample image into the task processing module in the initial model to obtain the second label information of the target image processing task for each sample image, and the task processing module is used to perform the target image processing task; determine a target loss value based on the degree of difference between the first label information and the second label information of each sample image; based on the target loss value, iteratively update the network parameters of the feature extraction module and the task processing module in the initial model until the model training conditions are met, and obtain the trained initial model as the target model.

[0077] Please refer to Fig.10 , which shows a structural block diagram of a model training device 400 provided in another embodiment of the present application, which is applied to a second device. The device 400 may include: a sample acquisition module 410, an image segmentation module 420 and a weight calculation module 430.

[0078] The sample acquisition module 410 is used to acquire a sample image set.

[0079] The image splitting module 420 is used to split each sample image in the sample image set into multiple sub-images to obtain an image sequence corresponding to each sample image.

[0080] The weight calculation module 430 is used to determine the target weight matrix corresponding to each sample image based on the degree of association between sub-images in the image sequence corresponding to each sample image. The target weight matrix is ​​used in the process of training the initial model based on the sample image by the first device. The image sequence corresponding to the sample image is input into the feature extraction module in the initial model to obtain the global image features corresponding to the sample image. The initial model is a network structure based on the self-attention mechanism.

[0081] In some embodiments, the model training device 400 may further include: a recognition module and a coloring module. The recognition module may be used to recognize objects contained in each sample image. The coloring module may be used to add a corresponding color to each type of object in each sample image if multiple types of objects are contained, to obtain each sample image after adding colors, and different types of objects have different colors added.

[0082] In this manner, the image splitting module 420 can be specifically used to split each sample image after adding color into multiple sub-images to obtain an image sequence corresponding to each sample image.

[0083] In some embodiments, the model training device 400 may further include: a pixel value acquisition module and a correlation degree acquisition module. The pixel value acquisition module may be used to acquire the pixel value at the target pixel position in each sub-image in the image sequence corresponding to each sample image before determining the target weight matrix corresponding to each sample image based on the correlation degree between sub-images in the image sequence corresponding to each sample image, so as to obtain the reference pixel value corresponding to each sub-image. The correlation degree acquisition module may be used to determine the similarity between each two sub-images in the image sequence corresponding to each sample image based on the reference pixel value corresponding to each sub-image, as the correlation degree between each two sub-images, and obtain the correlation degree between sub-images in the image sequence corresponding to each sample image.

[0084] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0085] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0086] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0087] In summary, in the solution provided by the present application, the first device can divide each sample image in the image sample set into multiple sub-images to obtain an image sequence corresponding to each sample image; obtain a target weight matrix corresponding to each sample image from multiple target weight matrices, wherein the matrix parameters in the target weight matrix are used to characterize the degree of association between the multiple sub-images in the image sequence, and the multiple target weight matrices are pre-calculated by the second device; input the image sequence corresponding to each sample image and the target weight matrix corresponding to each sample image into the feature extraction module in the initial model to obtain the global image feature corresponding to each sample image; based on the global image feature corresponding to each sample image, the feature extraction module and the task processing module of the initial model are trained to obtain the target model, and the initial model is a network structure based on the self-attention mechanism. In this way, in the process of model training, the weight matrix corresponding to each sample image is called from the multiple target weight matrices pre-calculated by the second device to extract the global image feature, and there is no need to calculate the weight matrix in real time based on the attention mechanism during the training process, that is, the number of calculations of the attention weight matrix during the training process is reduced, thereby improving the efficiency and speed of model training on the first device.

[0088] The following will be combined Fig.11 A computer device provided by the present application is described.

[0089] Reference Fig.11 , Fig.11The structural block diagram of a computer device 500 provided in an embodiment of the present application is shown, and the above method provided in an embodiment of the present application can be executed by the computer device 500. The computer device can be an electronic terminal with a data processing function, and the electronic terminal includes but is not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3) player, an MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4) player, and smart home devices, etc.; of course, the computer device can also be a server, and the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0090] The computer device 500 in the embodiment of the present application may include one or more of the following components: a processor 501, a memory 502, and one or more applications, wherein the one or more applications may be stored in the memory 502 and configured to be executed by one or more processors 501, and the one or more programs are configured to execute the method as described in the aforementioned method embodiment.

[0091] The processor 501 may include one or more processing cores. The processor 501 uses various interfaces and lines to connect various parts of the entire computer device 500, and executes various functions and processes data of the computer device 500 by running or executing instructions, programs, code sets or instruction sets stored in the memory 502, and calling data stored in the memory 502. Optionally, the processor 501 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 501 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem can also be integrated into the processor 501 and implemented separately through a communication chip.

[0092] The memory 502 may include a random access memory (RAM) or a read-only memory (ROM). The memory 502 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 502 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as the various corresponding relationships described above) created by the computer device 500 during use.

[0093] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0094] In several embodiments provided in the present application, the coupling or direct coupling or communication connection between the modules shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or modules, which may be electrical, mechanical or other forms.

[0095] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0096] Please refer to Fig.12 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 600 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0097] The computer readable storage medium 600 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 600 includes a non-transitory computer-readable storage medium. The computer readable storage medium 600 has storage space for program code 610 that performs any method step of the above method. These program codes can be read from or written to one or more computer program products. The program code 610 can be compressed, for example, in an appropriate form.

[0098] In some embodiments, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the steps in the above-mentioned method embodiments.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that: Applied to a first device, the method includes: Divide each sample image in the image sample set into a plurality of sub-images to obtain an image sequence corresponding to each sample image; Acquire a target weight matrix corresponding to each sample image from a plurality of target weight matrices, wherein matrix parameters in the target weight matrix are used to characterize the degree of association between the plurality of sub-images in the image sequence, and the plurality of target weight matrices are pre-calculated by the second device; Inputting the image sequence corresponding to each sample image and the target weight matrix corresponding to each sample image into the feature extraction module in the initial model to obtain the global image feature corresponding to each sample image; Based on the global image features corresponding to each sample image, the feature extraction module and the task processing module of the initial model are trained to obtain a target model, wherein the initial model is a network structure based on a self-attention mechanism.

2. The method according to claim 1, characterized in that Before inputting the image sequence corresponding to each sample image and the sample weight matrix corresponding to each sample image into the feature extraction module in the initial model to obtain the global image feature corresponding to each sample image, the method further includes: Extracting image features of each sub-image in the image sequence corresponding to each sample image to obtain a first image feature sequence corresponding to each sample image; The step of inputting the image sequence corresponding to each sample image and the sample weight matrix corresponding to each sample image into the feature extraction module in the initial model to obtain the global image feature corresponding to each sample image includes: The first image feature sequence corresponding to each sample image and the sample weight matrix corresponding to each sample image are input into the feature extraction module in the initial model to obtain the second image feature sequence corresponding to each sample image as the global image feature corresponding to each sample image.

3. The method according to claim 2, characterized in that The feature extraction module is an encoding module based on a self-attention mechanism, and the encoding module includes a key value acquisition unit and a feature aggregation unit; The step of inputting the first image feature sequence corresponding to each sample image and the sample weight matrix corresponding to each sample image into a feature extraction module in the initial model to obtain the second image feature sequence corresponding to each sample image as the global image feature corresponding to each sample image includes: Inputting the first feature sequence corresponding to each sample image into the key value acquisition unit to obtain a key value matrix corresponding to each sample image; The key-value matrix corresponding to each sample image and the sample weight matrix corresponding to each sample image are input into the feature aggregation unit to obtain a second feature sequence corresponding to each sample image as the global image feature corresponding to each sample image. The feature aggregation unit is used to perform weighted processing on the matrix parameters in the key-value matrix based on the sample weight matrix.

4. The method according to any one of claims 1 to 3, characterized in that: Each image sample carries first label information corresponding to a target image processing task. Based on the global image features corresponding to each sample image, the feature extraction module and the task processing module of the initial model are trained to obtain a target model. The initial model is a network structure based on a self-attention mechanism, including: Inputting the global image features corresponding to each sample image into the task processing module in the initial model to obtain second label information of the target image processing task for each sample image, wherein the task processing module is used to perform the target image processing task; Determining a target loss value based on a degree of difference between the first label information and the second label information of each sample image; Based on the target loss value, the network parameters of the feature extraction module and the task processing module in the initial model are iteratively updated until the model training conditions are met, so as to obtain the trained initial model as the target model.

5. A model training method, characterized in that: Applied to the second device, the method includes: Get a sample image set; Splitting each sample image in the sample image set into a plurality of sub-images to obtain an image sequence corresponding to each sample image; Based on the degree of association between sub-images in the image sequence corresponding to each sample image, a target weight matrix corresponding to each sample image is determined, and the target weight matrix is ​​used in the process of training an initial model based on the sample image by the first device. The image sequence corresponding to the sample image is input into a feature extraction module in the initial model to obtain the global image feature corresponding to the sample image, and the initial model is a network structure based on a self-attention mechanism.

6. The method according to claim 5, characterized in that Before splitting each sample image in the sample image set into a plurality of sub-images to obtain an image sequence corresponding to each sample image, the method further includes: Identify the object contained in each sample image; If multiple types of objects are included, then a corresponding color is added to each type of object in each sample image to obtain each sample image after adding the color, and different types of objects have different corresponding colors added; The step of splitting each sample image in the sample image set into a plurality of sub-images to obtain an image sequence corresponding to each sample image includes: Each sample image after adding color is split into multiple sub-images to obtain an image sequence corresponding to each sample image.

7. The method according to claim 6, characterized in that Before determining the target weight matrix corresponding to each sample image based on the degree of association between sub-images in the image sequence corresponding to each sample image, the method further includes: Acquire a pixel value at a target pixel position in each sub-image in an image sequence corresponding to each sample image, and obtain a reference pixel value corresponding to each sub-image; Based on the reference pixel value corresponding to each of the sub-images, the similarity between each two of the sub-images in the image sequence corresponding to each of the sample images is determined as the degree of association between each two of the sub-images, and the degree of association between the sub-images in the image sequence corresponding to each of the sample images is obtained.

8. A model training device, characterized in that: Applied to a first device, the apparatus comprises: An image splitting module is used to divide each sample image in the image sample set into multiple sub-images to obtain an image sequence corresponding to each sample image; A weight acquisition module, used to acquire a target weight matrix corresponding to each sample image from a plurality of target weight matrices, wherein matrix parameters in the target weight matrix are used to characterize the degree of association between the plurality of sub-images in the image sequence, and the plurality of target weight matrices are pre-calculated by the second device; A feature extraction module, used for inputting the image sequence corresponding to each sample image and the target weight matrix corresponding to each sample image into the feature extraction module in the initial model to obtain the global image feature corresponding to each sample image; The model training module is used to train the feature extraction module and the task processing module of the initial model based on the global image features corresponding to each sample image to obtain the target model. The initial model is a network structure based on the self-attention mechanism.

9. A model training device, characterized in that: Applied to a second device, the apparatus comprises: A sample acquisition module, used to acquire a sample image set; An image splitting module, used for splitting each sample image in the sample image set into multiple sub-images to obtain an image sequence corresponding to each sample image; A weight calculation module is used to determine a target weight matrix corresponding to each sample image based on the degree of association between sub-images in the image sequence corresponding to each sample image. The target weight matrix is ​​used in the process of training an initial model based on the sample image by the first device. The image sequence corresponding to the sample image is input into a feature extraction module in the initial model to obtain the global image feature corresponding to the sample image. The initial model is a network structure based on a self-attention mechanism.

10. A computer device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, and the program code can be called by a processor to execute the method according to any one of claims 1 to 7.

Images