Image encoder training method and apparatus, device, medium, product

Abstract

The application relates to an image encoder training method and device, equipment, medium and product, the method comprising: acquiring image encoders with the same network architecture but different feature scale sizes, the image encoders comprising a plurality of serially arranged feature representation layers, wherein a first image encoder with a larger feature scale serves as a teacher model, and a second image encoder with a smaller feature scale serves as a student model; synchronously inputting an image sample into the teacher model and the student model, determining an average loss value between features extracted by corresponding feature representation layers of each group of the teacher model and the student model; determining a prediction loss value corresponding to an inference result output by the student model, fusing the prediction loss value with the average loss value to obtain a model loss value corresponding to the image sample; and correcting a weight of the student model according to the model loss value. The application optimizes a knowledge distillation process, realizes model compression and maintains model performance.

Description

TECHNICAL FIELD

[0001] The present application relates to e-commerce information retrieval technology, and in particular to an image encoder training method and device, equipment, medium, and product. BACKGROUND

[0002] With the development of science and technology, the demand for searching for related commodities through taking photos of objects is increasing, such as users using the commodity search of buyer show-seller show to purchase commodities, and sellers searching for corresponding goods sources by taking photos of hot-selling commodities to purchase goods, and the like. Therefore, commodity image retrieval is a crucial step, and the understanding and recognition of commodity images are the core links.

[0003] In the technology of understanding images, a deep learning model is often used to implement it. The deep learning model has strong semantic abstract understanding ability. In theory, the more layers of neural network in the deep learning model or the larger the network structure, the better the feature representation effect, and the higher the image retrieval performance. However, in the actual deployment process, due to the limitation of computing resources, a model of too large a scale is not conducive to online, and thus a contradiction is formed between the model size and the model performance.

[0004] In the prior art, with the help of the knowledge distillation idea, the size of the deep learning model can be compressed. In the case of maintaining a given performance target, the model size is continuously compressed to find a balanced instance of the model to overcome the contradiction. In the traditional knowledge distillation technology, the output of the teacher model is used to calculate the distillation loss of the student model, and then the real loss between the student model and the real label is combined to determine the overall loss, or the output of the teacher model is directly used to supervise the output of the student model to determine the overall loss, so as to realize the correction of the model weight. In practice, due to the optimization only on the model inference result, the model compression effect is not ideal. SUMMARY

[0005] The present application aims to solve the above problems and provide an image encoder training method and corresponding device, equipment, non-volatile readable storage medium, and computer program product.

[0006] According to one aspect of the present application, an image encoder training method is provided, comprising the following steps:

[0007] An image encoder with the same network architecture but different feature scale sizes is obtained, the image encoder comprising a plurality of serially arranged feature representation layers, wherein a first image encoder with a larger feature scale size has been trained to convergence to constitute a teacher model in a knowledge distillation network, and a second image encoder with a smaller feature scale size is to be trained to constitute a student model in the knowledge distillation network;

[0008] synchronously inputting the image sample into the teacher model and the student model for feature extraction by feature representation layers in the teacher model and the student model;

[0009] determining a prediction loss value corresponding to the inference result output by the student model, and fusing the prediction loss value with the average loss value to obtain a model loss value corresponding to the image sample;

[0010] when the student model does not reach a convergence state, correcting the weight of the student model according to the model loss value, and then iterating the above process to synchronously input other image samples into the teacher model and the student model until the student model is trained to the convergence state.

[0011] Optionally, the determining of the average loss value between the features extracted by the feature representation layers corresponding to each group of the teacher model and the student model comprises:

[0012] performing scale transformation on the feature representation obtained by each feature representation layer of the student model to transform the feature representation to the same feature scale as the feature representation obtained by the corresponding feature representation layer of the teacher model;

[0013] based on the feature representation of each group of feature representation layers, calculating a feature loss value between corresponding features of each group of feature representation as the feature loss value corresponding to the feature representation layer of the group;

[0014] averaging the feature loss values of the feature representation layers of each group to obtain the average loss value corresponding to all groups of feature representation layers.

[0015] Optionally, the calculating of the feature loss value between corresponding features of each group of feature representation comprises:

[0016] based on a plurality of pooling steps, performing average pooling on each feature representation in the same group of feature representation respectively, and compressing each feature representation into a plurality of feature maps corresponding to the plurality of steps, each feature map comprising a plurality of channel vectors;

[0017] based on the corresponding relationship between the pooling step, the feature map and the channel vector, calculating a data distance between each two corresponding channel vectors of the same group of feature representation;

[0018] averaging the data distances between all corresponding channel vectors in the same group of feature representation to obtain the feature loss value corresponding to the group of feature representation.

[0019] Optionally, the determining of the prediction loss value corresponding to the inference result output by the student model, and the fusing of the prediction loss value with the average loss value to obtain the model loss value corresponding to the image sample comprises:

[0020] The smoothing loss function is applied to calculate a smoothing loss of an inference result output by the student model, as a prediction loss value of the student model, based on the inference result and reference to feature representations of a plurality of product images in a preset image feature library, the inference result being an output of a last feature representation layer in the student model.

[0021] The prediction loss value and the average loss value are weighted and fused to obtain a model loss value.

[0022] Optionally, until the student model is trained to a convergent state, the method comprises:

[0023] Obtaining a product image of a product item in a product information library;

[0024] Inputting the product image into the converged student model to obtain an inference result output by a last feature representation layer thereof as a feature representation of the product image;

[0025] Storing the feature representation of the product image in association with the product image in the product information library.

[0026] Optionally, after storing the feature representation of the product image in association with the product image in the product information library, the method comprises:

[0027] In response to a product recall request, obtaining a feature representation of a product image of a product item specified by the request as a to-be-retrieved feature;

[0028] Calculating a similarity between the to-be-retrieved feature and feature representations of product images of each product item in the product information library;

[0029] Filtering out part of the product items from the product information library according to the similarity, and packaging product information of the part of the product items to obtain a product information list;

[0030] Responding to the product recall request with the product information list.

[0031] Optionally, until the student model is trained to a convergent state, the method comprises:

[0032] Taking the converged student model as a teacher model, obtaining a third image encoder with a same network architecture but a smaller feature scale as a student model, reconfiguring a knowledge distillation network and continuing to train the knowledge distillation network to train the third image encoder to a convergent state.

[0033] According to another aspect of the present application, an image encoder training apparatus is provided, comprising:

[0034] The training preparation module is configured to obtain image encoders with the same network architecture but different feature scale sizes, and the image encoders include a plurality of feature representation layers arranged in series, wherein a first image encoder with a larger feature scale size is trained to convergence to constitute a teacher model in a knowledge distillation network, and a second image encoder with a smaller feature scale size is to be trained to constitute a student model in the knowledge distillation network.

[0035] The feature representation module is configured to synchronously input an image sample into the teacher model and the student model for feature extraction by feature representation layers in the teacher model and the student model, and determine an average loss value between features extracted by corresponding feature representation layers of the teacher model and the student model.

[0036] The loss determination module is configured to determine a prediction loss value corresponding to an inference result output by the student model, fuse the prediction loss value with the average loss value to obtain a model loss value corresponding to the image sample.

[0037] The iterative processing module is configured to correct weights of the student model according to the model loss value when the student model does not reach a convergence state, and then iteratively perform the above process, and synchronously input other image samples into the teacher model and the student model until the student model is trained to the convergence state.

[0038] According to another aspect of the present application, there is provided an image encoder training device, comprising a central processing unit and a memory, wherein the central processing unit is configured to invoke a computer program stored in the memory to execute steps of the image encoder training method according to the present application.

[0039] According to another aspect of the present application, there is provided a non-volatile readable storage medium, which stores a computer program implemented according to the image encoder training method in the form of computer readable instructions, and the computer program is invoked to run by a computer to execute steps included in the method.

[0040] According to another aspect of the present application, there is provided a computer program product, comprising computer program / instructions, which are executed by a processor to implement steps of the method described in any one of the embodiments of the present application.

[0041] Compared with the prior art, the present application has many advantages, including but not limited to:

[0042] Firstly, the teacher model and the student model both belong to image encoders with the same network architecture, only the feature scale of the teacher model is larger than that of the student model. In the knowledge distillation process, when the teacher model and the student model perform reasoning synchronously, the corresponding multiple sets of feature representation layers based on the same network architecture can obtain corresponding feature representations. After the feature representations are aligned in scale, the loss between each corresponding feature is calculated to obtain an average loss value. The average loss value is used to correct the prediction loss value obtained by the reasoning result of the student model to determine the model loss value. The student model is modified according to the model loss value. Since the correlation between the corresponding features of each set of feature representation layers is considered in advance, the consistency of the corresponding features between the teacher model and the student model is better. The reasoning ability of the student model can be more efficiently improved by using the higher reasoning ability of the teacher model learned, so that the network performance of the student model is greatly improved, and the student model is more easily trained to converge. While realizing the miniaturization of the model, an image encoder with excellent network performance can also be obtained.

[0043] Secondly, since the image encoder is realized to be miniaturized by means of knowledge distillation training, and the miniaturized image encoder can inherit the reasoning ability learned by the larger scale image encoder, when the image encoder finally trained to converge is used for online reasoning, the image data can be accurately represented in semantics, and the corresponding feature representation can be obtained. On the basis of the feature representation, the retrieval performance corresponding to the semantic retrieval based on images can be further improved.

[0044] In addition, the image encoder trained by the present application is suitable for deployment in an e-commerce platform, providing a basic support framework for commodity retrieval technology based on commodity images, so that the e-commerce platform can achieve economies of scale. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0046] Figure 1 The network architecture schematic diagram of the application environment of the technical solutions of the present application;

[0047] Figure 2 The model architecture schematic diagram of the image encoder exemplarily adopted by the present application;

[0048] Figure 3 The principle block diagram of the knowledge distillation network exemplarily adopted by the present application;

[0049] Figure 4 Flowchart of an embodiment of the image encoder training method of the present application;

[0050] Figure 5 Flowchart of an embodiment of the image encoder training method of the present application;

[0051] Figure 6 Flowchart of an embodiment of the image encoder training method of the present application;

[0052] Figure 7 Flowchart of an embodiment of the image encoder training method of the present application;

[0053] Figure 8 Flowchart of an embodiment of the image encoder training method of the present application;

[0054] Figure 9 Flowchart of an embodiment of the image encoder training method of the present application;

[0055] Figure 10 Flowchart of an embodiment of the image encoder training method of the present application; DETAILED DESCRIPTION

[0056] The models referred to or possibly referred to in the present application, including traditional machine learning models or deep learning models, can be deployed on remote servers and remotely called by clients, or directly called by clients with sufficient device capabilities, unless otherwise specified. In some embodiments, when the models are run on the client, the corresponding intelligence can be obtained through transfer learning to reduce the requirement for client hardware resources and avoid excessive occupation of client hardware resources.

[0057] Referring to Figure 1 An exemplary application scenario of the present application employs a network architecture including a terminal device 80, a standalone station 81, and an application server 82. The application server 82 can be used to deploy the image encoder of the present application to open an image encoding service. The standalone station 81 can be used to deploy and open an online store of an e-commerce service. A user on the terminal device 80 can submit a product image to the standalone station 81 in a page of the online store. The standalone station 81 can store the product image in its own product information database and further call an interface opened by the image encoder of the application server 82 to submit the product image to the image encoder, which extracts the feature representation of the product image and returns it.

[0058] Further, a commodity retrieval service can be implemented in the application server, which further encapsulates the image encoder, the independent station 81 can submit the commodity image to the commodity retrieval service, call the image encoder to obtain the corresponding feature representation, and then perform semantic retrieval in the commodity information library to determine other on-shelf commodity items whose commodity images are semantically similar to the feature representation of the commodity image, and return these commodity items and their commodity images to the user.

[0059] It should be noted that, Figure 1 The application scenario is only an example of platform deployment, and in some other exemplary embodiments, the image encoder of the present application can also run in any computer device with sufficient computing power. For example, it can run in the terminal device 80 or the independent station 81 to provide an image encoder service.

[0060] As Figure 1 Any one of the terminal device 80, the independent station 81, and the application server 82 can be used to run the computer program product programmed and implemented according to the image encoder training method of the present application, as long as it has sufficient computing power, so as to train an image encoder meeting the requirements of the present application, which is used to deploy a corresponding image encoding service.

[0061] Please refer to Figure 2 The present application provides an exemplary image encoder, which can include multiple serially arranged feature representation layers, each feature representation layer extracts the feature information of the image sample input therein from shallow to deep, obtains the corresponding feature representation of each layer, and the inference result output by the last feature representation layer can be used as the effective corresponding feature representation of the image sample. The feature representation layer in the image encoder can be any basic neural network model suitable for extracting features from images, such as convolutional neural network, residual module, recurrent neural network, and multi-head attention-based encoding module. In some embodiments, the image encoder can be a basic model such as Swin Transformer model or ResNet model, each of which can be based on the same network architecture and adapted to different feature scale sizes to derive multiple instances, for example, the Swin Transformer model has different instances of large and small versions, and the ResNet model has different instances of ResNet152, ResNet34, etc.

[0062] Please refer to Figure 3In order to train an image encoder meeting the expectation, after selecting instances of different feature scale sizes, a first image encoder with the largest feature scale is trained to a convergent state, and then is taken as a teacher model, a second image encoder with a relatively smaller feature scale is taken as a student model, the teacher model and the student model are constructed into a knowledge distillation network, knowledge distillation is implemented from the teacher model to the student model, the student model is trained to a convergent state, and thus an image encoder meeting the expectation is obtained.

[0063] As shown in the knowledge distillation network, Figure 3 between the feature representation layers corresponding to each group of the teacher model and the student model, a loss calculation module is accessed to calculate the feature loss value between two feature representations obtained between the feature representation layers in the same group, finally, the average loss value is obtained by averaging all feature loss values corresponding to all groups of feature representation layers, and the inference result obtained by the student model corresponding to an image sample can be determined by using a smoothing loss function or a cross-entropy function without relying on the inference result of the teacher model, finally, the average loss value and the prediction loss value are fused into a model loss value, the student model is updated by using the model loss value, and the convergence of the student model is promoted.

[0064] In order to implement the training of the teacher model and the student model, a training data set can be prepared, in which a large number of image samples are stored, and the supervision label corresponding to the image sample can be determined in real time according to the calculation mode corresponding to the prediction loss value during training, so as to embody the advantage of self-supervised training idea.

[0065] Based on the principles disclosed above, please refer to Figure 4 According to an image encoder training method provided by the present application, in one embodiment, the method comprises the following steps:

[0066] Step S1100, an image encoder with the same network architecture but different feature scale sizes is obtained, the image encoder comprises a plurality of serially arranged feature representation layers, wherein a first image encoder with a larger feature scale has been trained to a convergent state and constitutes a teacher model in a knowledge distillation network, and a second image encoder with a smaller feature scale is to be trained and constitutes a student model in the knowledge distillation network;

[0067] For ease of understanding, it is assumed that two image encoder instances corresponding to the same base model are prepared, wherein the first image encoder and the second image encoder have the same network architecture and comprise the same number of a plurality of feature representation layers, the plurality of feature representation layers are serially arranged, so as to extract semantic features of corresponding scales from given image feature information from shallow to deep by each feature representation layer in turn, and the first image encoder has a larger feature scale than the second image encoder,Figure 2 The feature representation layer is shown.

[0068] The feature representation layer may, for example, include a convolution layer, an activation layer, a pooling layer, a normalization layer, etc. The convolution layer performs a convolution operation on the image feature information input thereto to extract corresponding semantic features, which are activated by the activation layer. The pooling layer performs a pooling operation on the activated features to constrain their scale, and the normalization layer performs a batch normalization operation on the features, thereby obtaining the feature representation of the corresponding layer.

[0069] The corresponding groups of feature representation layers between the first image encoder and the second image encoder have different feature scale constraints, thereby embodying different scales. For example, the first feature representation layer of the first image encoder can process feature information of a 128*128*C scale to a 64*64*4C scale, where C is the number of channels. Correspondingly, the first feature representation layer of the second image encoder can process feature information of a 64*64*C scale to a 32*32*4C scale, thereby embodying different feature scales between the corresponding two feature representation layers. In other embodiments, individual groups of feature representation layers may

[0070] To implement knowledge distillation, the first image encoder can be trained to a converged state using a training data set prepared for the training task of the present application. In one embodiment, when training the first image encoder, the first image encoder can be connected to a classifier, and the classification task can be iteratively performed on the image samples in the training data set to train the first image encoder to a converged state, so that the first image encoder acquires the ability to perform feature representation on a given image, and outputs a final determined feature representation after the last feature representation layer completes the inference process. In other embodiments, the first image encoder can also be unsupervised trained based on the idea of generative adversarial training. The like can be flexibly determined by those skilled in the art.

[0071] The trained first image encoder has certain feature representation capability, and thus can be used as a teacher model in a knowledge distillation architecture. Therefore, the first image encoder and the second image encoder are built into a knowledge distillation network according to the architecture shown. Figure 3 The image samples in the training data set are used to implement a training task based on the knowledge distillation network, so as to implement knowledge distillation on the student model through the inference ability learned by the teacher model, and guide the student model to quickly train to a converged state through the teacher model.

[0072] Step S1200, synchronously inputting an image sample into the teacher model and the student model for feature extraction by feature representation layers therein, and determining an average loss value between features extracted by feature representation layers corresponding to each group of the teacher model and the student model;

[0073] When the training of the student model is started based on the built knowledge distillation network, the image sample is iteratively called from the training data set for training. For each iteration of the image sample, it is synchronously input into the teacher model and the student model, and the teacher model and the student model perform feature representation at different feature scales in multiple levels for the training sample respectively, complete the inference process, and finally output the inference result.

[0074] Specifically, as shown in the structure Figure 2 In the structure as exemplified, the image sample is first extracted by the first feature representation layer of any image encoder to obtain the semantic features of the first scale at the shallow layer, and then the second feature representation layer connected with the first feature representation layer further extracts the semantic features of the second scale at a deeper layer. In this way, the fourth feature representation layer directly obtains the semantic features of the fourth scale, which can be directly used as the final inference result, and the inference process of feature representation of the image sample is completed. In this process, the teacher model and the student model extract corresponding feature representations at their corresponding scales through multiple feature representation layers, respectively. Therefore, the characteristics that the teacher model and the student model have the same network architecture and the features of the feature representations of each other are related can be used to calculate the feature loss value between the feature representation layers corresponding to each group, and the overall average loss value between the teacher model and the student model at the feature representation layer level can be calculated by using the feature loss values of all groups.

[0075] In one embodiment, for the features with different feature scales between the feature representation layers corresponding to each group of the teacher model and the student model, and the feature representation layers in the student model have smaller feature scales than the corresponding feature representation layers of the teacher model, the feature representation layers in the student model can be further scaled. On the basis of the scaling, the feature loss value between the feature representations obtained by the corresponding two feature representation layers is calculated.

[0076] In another embodiment, when the feature representation layers corresponding to each group of the teacher model and the student model have the same feature scale, the corresponding feature representations obtained by the student model can not be scaled.

[0077] It can be seen that, based on the same set of corresponding feature representation layers, the feature representation obtained by the teacher model and the student model calculates the loss value and finally determines the average loss value, which can be deeply into each specific link of the inference process of the image encoder, benefit the correlation of the isomorphic network, realize the feature alignment calculation loss value, so that when the average loss value determined accordingly is used to determine the model loss value and correct the weight of the student model, the student model and the teacher model can obtain better feature consistency, thereby realizing greater improvement of the performance of the small-scale network, while ensuring smaller resource cost, the better feature representation effect can be realized, and after further application in the commodity retrieval service, the commodity retrieval performance can be improved.

[0078] Step S1300, determining the prediction loss value corresponding to the inference result output by the student model, fusing it with the average loss value to obtain the model loss value corresponding to the image sample;

[0079] For the inference result obtained after the student model infers the image sample, in one embodiment, that is, the feature representation obtained by inferring the last feature representation layer, the prediction loss value corresponding to the feature representation can be calculated in various ways.

[0080] In one embodiment, the feature representation obtained by the student model can be classified and mapped by a classifier to obtain a classification result, and the cross-entropy loss corresponding to the classification result of the image sample can be calculated as the corresponding prediction loss value according to the supervision label of the image sample.

[0081] In another embodiment, the smooth loss of the feature representation can be calculated according to the following process, which can be processed as follows:

[0082] Step S1310, applying a smooth loss function, calculating the smooth loss of the inference result based on the inference result output by the student model, referring to the feature representations of a plurality of commodity images in a preset image feature library, as the prediction loss value of the student model, the inference result being the output of the last feature representation layer in the student model;

[0083] The smooth loss function refers to the Smooth-AP function, which allows the deep network to perform end-to-end training based on the ranking task, by retrieving the feature representation corresponding to the feature representation of the image sample in a given image feature as the result, and then determining the average precision AP(Average Precision) as the average loss. Smooth-AP is a simple, elegant and scalable method, which is superior to the ordinary AP algorithm, it adopts the form of plug-and-play objective function, uses the sigmoid function to relax the indicator function in the non-differentiable AP, and essentially replaces the discrete indicator function with a sigmoid function.

[0084] In an embodiment, the feature representation stored in the image feature library can be a feature representation of a product image of a product item in a product information library of an e-commerce platform, so that the student model can be trained corresponding to a product retrieval service of the e-commerce platform.

[0085] After the inference result of the student model is applied to a smoothing loss function to calculate a result, a smoothing loss corresponding to the inference result is obtained, which can be used as a prediction loss value of the student model.

[0086] As can be seen from the above embodiments for calculating the prediction loss value, the calculation of the prediction loss value of the student model no longer needs to rely on the distillation loss calculation of the inference result of the teacher model and the inference result of the student model.

[0087] Step S1320, weighting and fusing the prediction loss value and the average loss value to obtain a model loss value.

[0088] Further, in order to realize the gradient update of the weight parameters of the student model, the average loss value related to the feature level between the teacher model and the student model and the prediction loss value obtained based on the inference result level of the student model are fused by using the following formula, and then the model loss value of the student model corresponding to a single input image sample can be obtained. In an embodiment, the average loss value can be associated with a preset weight according to step S1320, and then fused with the prediction loss value to obtain the model loss value. For example, the formula is as follows:

[0089] L = L SAP + aL corr

[0090] Wherein, L corr represents the average loss value corresponding to the feature level, L SAP represents the prediction loss value corresponding to the inference result level, and a is a preset weight.

[0091] Wherein:

[0092]

[0093] N1, N2, N3, and N4 represent each feature representation layer in the image encoder, respectively, corr function represents a data distance algorithm function, f tin ,f sin represent the feature representations of the teacher model and the student model in the same scale obtained at the corresponding feature representation layer, respectively.

[0094] Step S1400, when the student model does not reach the convergence state, the weight of the student model is corrected according to the model loss value, and then the above process is iterated, and other image samples are input into the teacher model and the student model simultaneously until the student model is trained to the convergence state.

[0095] When the model loss value corresponding to each iteration is calculated for each image sample, the model loss value can be compared with a preset decision threshold value for indicating that the model reaches the convergence state. When the model loss value reaches the decision threshold value, it is determined that the student model has reached the convergence state, and the training thereof can be terminated. When the model loss value does not reach the decision threshold value, it is indicated that the student model has not reached the convergence state. In this case, the gradient update of the student model is performed according to the model loss value, the weight parameters of each feature representation layer are corrected by back propagation, and then the next image sample is input into the teacher model and the student model to start the next iteration training. In this way, the student model is determined to have reached the convergence state directly according to the model loss value of the corresponding image sample.

[0096] It should be noted that, in the entire knowledge distillation process, the teacher model does not need to be trained again, and thus the weight parameters thereof can be frozen before the training is started, and the weight thereof does not need to be corrected in the entire process.

[0097] In some embodiments, after step S1400, the converged student model can be further used as a teacher model, then a third image encoder with the same network architecture but a smaller feature scale is obtained as a student model, a knowledge distillation network is reconstructed, and the above steps S1200 to S1400 are iterated to continue training the reconstructed knowledge distillation network to train the third image encoder to the convergence state. As can be seen, through iteration, the feature scale of the student model can be continuously contracted to obtain a corresponding converged instance, and the converged instance obtained can effectively inherit the reasoning ability learned by the teacher model in the entire iteration process, so that the converged instance finally obtained has stronger reasoning ability but smaller network scale, and is more suitable for lightweight deployment.

[0098] According to the above embodiments, it can be known that the present application has many advantages, including but not limited to:

[0099] Firstly, the teacher model and the student model both belong to image encoders with the same network architecture, only the feature scale of the teacher model is larger than that of the student model. In the knowledge distillation process, when the teacher model and the student model perform reasoning synchronously, the corresponding multiple sets of feature representation layers based on the same network architecture can obtain corresponding feature representations. After the feature representations are aligned in scale, the loss between each corresponding feature is calculated to obtain an average loss value. The average loss value is used to correct the prediction loss value obtained by the reasoning result of the student model to determine the model loss value. The student model is modified according to the model loss value. Since the correlation between the corresponding features of each set of feature representation layers is considered in advance, the consistency of the corresponding features between the teacher model and the student model is better, the reasoning ability of the student model can be more efficiently improved by using the higher reasoning ability of the teacher model, so that the network performance of the student model is greatly improved, and the student model is more easily trained to converge. While realizing the miniaturization of the model, an image encoder with excellent network performance can also be obtained.

[0100] Secondly, since the image encoder is realized to be miniaturized by means of knowledge distillation training, and the miniaturized image encoder can inherit the reasoning ability learned by the larger scale image encoder, when the image encoder finally trained to converge is used for online reasoning, it can realize accurate semantic representation of image data, obtain corresponding feature representation, and further improve the retrieval performance of semantic retrieval based on image.

[0101] In addition, the image encoder trained in the present application is suitable for deployment in an e-commerce platform, providing a basic support framework for commodity image-based commodity retrieval technology, so that the e-commerce platform can achieve economies of scale.

[0102] On the basis of any embodiment of the present application, please refer to Figure 5 to determine the average loss value between the features extracted by each corresponding feature representation layer of the teacher model and the student model, comprising:

[0103] Step S1210, performing scale transformation on the feature representation obtained by each feature representation layer of the student model, and transforming it to the same feature scale as the feature representation obtained by the corresponding feature representation layer of the teacher model;

[0104] As described above, in order to calculate the feature loss value between the feature representations of the same corresponding group of the teacher model and the student model, the feature representation obtained by the corresponding feature representation layer of the student model can be first subjected to scale transformation, and the transformation target is to unify the scale of the feature representation to the same scale as the feature representation obtained by the corresponding feature representation layer of the teacher model.

[0105] For example, a 384*384 image sample is input into the teacher model, and four feature representations F t1 t2 t3 t4 Similarly, the student model also obtains four feature representations from the image sample. For each feature representation obtained by the student model, the scale of the corresponding feature representation output by the teacher model is referred to, a 1*1 convolution kernel is first applied to the feature representation of the student model for scale transformation, and the transformed feature representation is used for batch normalization in the standardization layer (BN, Batch Normalization) to obtain a feature representation belonging to the student model with the same scale as the corresponding feature representation of the teacher model. For the student model, the feature representations obtained after scale transformation can be denoted as F s1 s2 s3 s4 .

[0106] In step S1220, the feature loss value between the corresponding features of each group of feature representations is calculated based on the feature representations of each group of feature representations, as the corresponding feature loss value of the feature representation layer of the group.

[0107] Referring to the formula for calculating the average loss value above:

[0108]

[0109] According to the formula, for the corresponding group of feature representations of the teacher model and the student model, the loss value between the features can be calculated by using a preset data distance algorithm, and the feature loss value between the corresponding feature representation layers of the corresponding group can be obtained.

[0110] In step S1230, the feature loss values of the feature representation layers of each group are averaged to obtain the average loss value corresponding to all the groups of feature representation layers.

[0111] Similarly, according to the formula above, for the feature loss values obtained by all groups between the teacher model and the student model, a simple average or weighted average can be used to further statistically average all the feature loss values, thereby obtaining the average loss value corresponding to all the groups of feature representation layers.

[0112] ​​​​​​According to the above embodiments, although the feature scales of the feature representations obtained by the respective groups of feature representation layers of the teacher model and the student model may not be uniform, after the uniform scale is achieved through the scale transformation, the correlation between the feature representations of the same group can be established, so that the feature loss value can be calculated for the two feature representations of the same group, and finally the average loss value of the entire student model at the feature level relative to the teacher model can be effectively obtained by averaging the feature loss values of all groups, which replaces the traditional distillation loss calculation method and also achieves the effect of feature alignment correction, which helps the student model to converge more quickly and the correction effect to be better.

[0113] On the basis of any embodiment of the present application, refer to Figure 6 , calculate the feature loss value between the corresponding features of each group of feature representations, including:

[0114] Step S1221, based on a plurality of pooling steps, respectively average pool each feature representation in the same group of feature representations, and compress each feature representation into a plurality of feature maps corresponding to the plurality of steps, each feature map including a plurality of channel vectors;

[0115] For the two feature representations of the corresponding groups of the teacher model and the student model that need to be used to calculate the feature loss value, since they have been unified to the same scale, the same principle operation can be calculated. Specifically, each feature representation in the same group is subjected to an average pooling operation by using a plurality of pooling layers with different pooling steps, so as to compress it from a three-dimensional feature map format to a two-dimensional feature map format. Thus, along the channel direction, each feature representation will include a plurality of channel vectors in the feature map obtained after the action of one pooling step.

[0116] As can be seen, after the pooling operation, each feature representation obtains a plurality of feature maps, each of which includes a plurality of channel vectors, and this is true for the feature representations of the teacher model and the student model. Thus, a one-to-one correspondence between the channel vectors is established between the teacher model and the student model, and since the feature dimensions between the channel vectors are also the same, a one-to-one correspondence between the specific feature values is actually established.

[0117] Step S1222, based on the correspondence between the pooling steps, the feature maps, and the channel vectors, calculate the data distance between each two corresponding channel vectors of the same group of feature representations;

[0118] At this point, for the same group of feature representations, the data distance can be calculated for the one-to-one corresponding channel vectors in the corresponding feature maps obtained by the same pooling step, and this data distance can effectively represent the similarity loss between the teacher model and the student model at the specific features.

[0119] In one embodiment, the data distance between the channel vectors can be calculated using the Pearson correlation coefficient algorithm, and in another embodiment, the cosine similarity algorithm can be used for calculation. Those skilled in the art can choose flexibly.

[0120] Step S1223, the data distance between all corresponding channel vectors in the same group of feature representations is averaged to obtain the feature loss value corresponding to the group of feature representations.

[0121] Therefore, for the same group of feature representations, it is decomposed into feature maps corresponding to different pooling steps, and then a plurality of channel vectors in each feature map are decomposed, and the data distance between the channel vectors is calculated to represent the corresponding similarity loss. Accordingly, the data distance between all channel vectors corresponding to two feature representations is averaged to obtain a certain mean corr avg , which can be used as the feature loss value corresponding to the group of feature representations.

[0122] According to the above embodiments, by decomposing the same group of corresponding feature representations into channel vectors in the feature maps obtained by different pooling steps, the feature loss value is calculated based on the correspondence between the channel vectors of the teacher model and the student model, the feature alignment calculation model is used to calculate the loss, which is more accurate and delicate, which is beneficial to improve the weight correction amplitude in each gradient update process, and promotes the training efficiency of the student model, and ensures that the reasoning ability learned by the teacher model can be accurately learned.

[0123] On the basis of any embodiment of the present application, please refer to Figure 7 , until the student model is trained to a convergent state, including:

[0124] Step S2100, obtaining the product image of the product item in the product information library;

[0125] In one embodiment, the present application provides a product information library, which stores the product images of the product items in the independent station of the e-commerce platform. The product images can be obtained from the product image files of the listed products already stored in the product information library, so as to determine the corresponding feature representations for the product image files of the listed products.

[0126] The product image refers to an image used to show the shape, structure, use state and other contents of the product, which can be a detail image of a local area of a single product, or a full view image of the entire product. In the e-commerce platform application scenario, the product image is usually a display image of the listed product of the online store, including but not limited to the main image and detail image displayed on the product detail page of the listed product. Similarly, as long as the content of the image in the product image is the product, it can be regarded as the product image of the present application.

[0127] In an embodiment, the product image can be submitted by a user on a terminal device, the user can take a product image through a camera in the terminal device, or obtain a product image from a video file or a picture file, and then submit it to an independent station server for further processing to determine the feature representation of the product image.

[0128] In another embodiment, the product image can be stored in any available storage space, including but not limited to a distributed storage system, a local hard disk, a local memory, etc. When the product image needs to be called, the corresponding binary data can be directly read from the corresponding storage space.

[0129] In some embodiments, the image format preprocessing of the product image can be adapted to the input parameter requirements of the image encoder of the present application. For example, the product image can be adjusted to a target size, the product image can be cropped to highlight the product, etc. Those skilled in the art can flexibly implement it.

[0130] Step S2200, input the product image into the converged student model to obtain the inference result output by the last feature representation layer as the feature representation of the product image;

[0131] The student model of the present application, when trained to a converged state, can be used as an image encoder in the inference phase and can be used to implement image encoding services. Accordingly, after obtaining the product image, the image encoding service is called, and the image encoder is called to perform feature extraction on the product image, thereby extracting the feature representation of the product image as image feature information.

[0132] Step S2300, store the feature representation of the product image in the product information library in association with the product image.

[0133] In order to facilitate subsequent image semantic retrieval, after obtaining the feature representation of the product image, it can be stored in association with the product image, and can be stored in the product information library. The association mapping relationship between the product item, the product image of the product item, and the feature representation of the product image can be maintained.

[0134] According to the above embodiments, the converged student model trained by the present application can be used as an image encoder. Due to its lightweight characteristics, it can run efficiently and can be used to implement batch feature representation of massive product images in the product information library of each independent station in the e-commerce platform, and provide effective basic information for product retrieval services.

[0135] On the basis of any embodiment of the present application, refer to Figure 8 After storing the feature representation of the product image in association with the product image in the product information library, comprising:

[0136] Step S3100, in response to a product recall request, obtaining the feature representation of the product image of the product item specified by the request as a feature to be searched;

[0137] In an exemplary application scenario, the image encoder of the present application can be used to pre-extract the corresponding feature representation of each product image of each product item in the product information library of the independent station as image feature information, and the image feature information and its corresponding product image, the source product are constructed as mapping relationship data, which can also be stored in the product information library or its mapping view. Thus, semantic retrieval can be performed through the image feature information in the product information library, thereby achieving the purpose of image retrieval.

[0138] In some more specific business scenarios, for example, in a photo search recommendation scenario, the independent station obtains a product image as a search product image, which can be a product image obtained by a user enabling the camera of a terminal device to take a photo, a picture file of the user, or a picture file specified by the user and obtained by the independent station from the corresponding storage space. For the search product image, the independent station can trigger a product recall request, which is submitted to the image encoding service. The image encoding service responds to the product recall request, obtains the search product image specified in the product recall request, and then further calls the image encoder of the present application to obtain the corresponding feature representation by feature extraction, as the corresponding feature to be searched.

[0139] Step S3200, calculating the similarity between the feature to be searched and the feature representation of the product image of each product item in the product information library;

[0140] Further, using a product search service, calculating the data distance between the feature to be searched of the search product image and the feature representation of each product image in the product information library, converting the data distance into a representation of semantic similarity, so that each product image in the product information library corresponds to the search product image and obtains its own corresponding semantic similarity. When representing the semantic similarity, the greater the numerical value, the higher the similarity. The product search service can be deployed in the current independent station server or in the application server of the e-commerce platform for the current independent station to call.

[0141] When calculating the data distance between two feature representations, any feasible data distance algorithm can be used, including but not limited to: cosine similarity algorithm, Euclidean distance algorithm, Pearson correlation coefficient algorithm, Jaccard coefficient algorithm, etc., which can be flexibly selected by those skilled in the art.

[0142] It is easy to understand that the semantic similarity obtained for each product image in the product information database represents the degree of semantic similarity between it and the product image to be searched. The higher the semantic similarity, the more similar the two are in terms of image content, and vice versa.

[0143] Step S3300: Select a portion of product items from the product information database based on similarity, and encapsulate the product information of the portion of product items to obtain a product information list;

[0144] Furthermore, based on the semantic similarity calculated from the product image to be searched, the product images in the product information database can be filtered.

[0145] In one embodiment, a preset number of images to be selected is set, and the images of each product in the product information database are sorted from high to low according to semantic similarity. Then, one or more of the top-ranked product images are selected according to the preset number. These product images are similar product images that match the product image to be searched.

[0146] In another embodiment, a similarity threshold is set, and then the semantic similarity obtained from the image feature information of each product image in the product information database is compared with the threshold. When the semantic similarity is higher than the threshold, the corresponding product image can be confirmed as a similar product image that matches the product image to be searched, thereby filtering out one or more similar product images.

[0147] For one or more similar product images selected, based on their correspondence with their respective source product items, product information for each source product item can be further obtained to encapsulate a product information list. This product information typically includes the corresponding similar product images of the source product item, the product title of the source product item, the price of the source product item, and product links, etc. The specific product information selected can be flexibly set according to business needs and is not limited to the above example.

[0148] Step S3400: Respond to the product recall request with the product information list.

[0149] After obtaining the product information list, it can be pushed to the corresponding user's terminal device. The terminal device parses the product information list and constructs different product display controls for each source product item in the list, which are then displayed in the terminal device's graphical user interface. Each product display control can display its corresponding product information, including product title, product image, and product price. Corresponding product links can also be added to this product information. Therefore, when a corresponding product display control is touched, the user can jump to the corresponding product link, directly accessing the product details page of the corresponding source product item, thus achieving the business effect of image search.

[0150] As can be seen from the above embodiments, when the image encoder of this application is used to generate feature representations for product images in the product information database and product images to be searched, the semantic similarity between the feature representations can be used to realize the image search business. Since the image encoder trained by this application has the advantage of accurately understanding the deep semantics in the image, the obtained feature representation is more accurate, so as to accurately search for similar product images in the product information database for the product image to be searched, thereby improving the accuracy and distinguishability of the search.

[0151] Please see Figure 9 According to one aspect of this application, an image encoder training apparatus includes a training preparation module 1100, a feature representation module 1200, a loss determination module 1300, and an iterative processing module 1400, wherein: the training preparation module 1100 is configured to acquire image encoders with the same network architecture but different feature scales, the image encoders comprising multiple sequentially arranged feature representation layers, wherein a first image encoder with a larger feature scale has been trained to convergence and constitutes a teacher model in a knowledge distillation network, and a second image encoder with a smaller feature scale is to be trained and constitutes a student model in a knowledge distillation network; the feature representation module 1200 is configured to synchronously train the teacher model and the student model... The student model inputs image samples for feature representation layer extraction, and determines the average loss value between the features extracted by the feature representation layers of each group of the teacher model and the student model. The loss determination module 1300 is configured to determine the prediction loss value corresponding to the inference result output by the student model, and fuse it with the average loss value to obtain the model loss value corresponding to the image sample. The iterative processing module 1400 is configured to, when the student model has not reached convergence, adjust the weights of the student model according to the model loss value, and then iterate the above process, calling other image samples to synchronously input the teacher model and the student model until the student model is trained to convergence.

[0152] Based on any embodiment of this application, the feature representation module 1200 includes: a scaling unit, configured to scale the feature representations obtained by each feature representation layer of the student model to maintain the same feature scale as the feature representations obtained by the corresponding feature representation layer of the teacher model; a loss calculation unit, configured to calculate the feature loss value between corresponding features of each group of feature representations based on the feature representations of each group of feature representations, as the feature loss value corresponding to that group of feature representations; and an averaging unit, configured to average the feature loss values ​​of each group of feature representations to obtain the average loss value corresponding to all multiple groups of feature representations.

[0153] Based on any embodiment of this application, the loss calculation unit includes: a pooling processing subunit, configured to perform average pooling on each feature representation in the same group of feature representations based on multiple pooling steps, compressing each feature representation into multiple feature maps corresponding to multiple steps, each feature map including multiple channel vectors; a distance calculation subunit, configured to calculate the data distance between every two corresponding channel vectors of the same group of feature representations based on the correspondence between pooling steps, feature maps, and channel vectors; and a loss averaging subunit, configured to take the average of the data distances between all corresponding channel vectors in the same group of feature representations as the feature loss value corresponding to that group of feature representations.

[0154] Based on any embodiment of this application, the loss determination module 1300 includes: a smoothing loss calculation unit, configured to apply a smoothing loss function, calculate the smoothing loss of the inference result based on the inference result output by the student model and referencing the feature representations of multiple product images in a preset image feature library, and use it as the predicted loss value of the student model, wherein the inference result is the output of the last feature representation layer in the student model; and a weighted fusion processing unit, configured to perform weighted fusion of the predicted loss value and the average loss value to obtain the model loss value.

[0155] Based on any embodiment of this application, the iterative processing module 1400 further includes: an image acquisition module, configured to acquire product images of product items in a product information database; a model inference module, configured to input the product image into the converged student model to obtain the inference result output by the last feature representation layer as the feature representation of the product image; and a feature storage module, configured to associate the feature representation of the product image with the product image and store it in the product information database.

[0156] Based on any embodiment of this application, the feature storage module further includes: a request response module, configured to respond to a product recall request and obtain the feature representation of the product image of the product item specified in the request as a feature to be retrieved; a similarity calculation module, configured to calculate the similarity between the feature to be retrieved and the feature representation of the product image of each product item in the product information database; a list encapsulation module, configured to filter out a portion of product items from the product information database based on the similarity and encapsulate the product information of the portion of product items to obtain a product information list; and a request response module, configured to respond to the product recall request with the product information list.

[0157] Based on any embodiment of this application, the iterative processing module 1400 includes: a compression iteration module, configured to use the converged student model as the teacher model, obtain a third image encoder with the same network architecture but smaller feature scale as the student model, reconstruct the knowledge distillation network and continue to train it to train the third image encoder to a converged state.

[0158] Another embodiment of this application also provides an image encoder training device. For example... Figure 10 The diagram shows the internal structure of an image encoder training device. This device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable, non-volatile storage medium stores an operating system, a database, and computer-readable instructions. The database stores information sequences, and when executed by the processor, these computer-readable instructions enable the processor to implement an image encoder training method.

[0159] The processor of the image encoder training device provides computational and control capabilities, supporting the operation of the entire device. The memory of the device can store computer-readable instructions, which, when executed by the processor, cause the processor to perform the image encoder training method of this application. The network interface of the device is used for communication with a terminal.

[0160] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the image encoder training device to which the present application is applied. A specific image encoder training device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0161] In this embodiment, the processor is used to execute... Figure 9The specific functions of each module are described, and the memory stores the program code and various data required to execute the above modules or sub-modules. The network interface is used to realize data transmission between user terminals or servers. The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the image encoder training device of this application, and the server can call the server's program code and data to execute the functions of all modules.

[0162] This application also provides a non-volatile readable storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the image encoder training method of any embodiment of this application.

[0163] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the method described in any embodiment of this application.

[0164] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM).

[0165] In summary, this application provides an image encoder with miniaturized feature scale and excellent network performance, suitable for deployment on e-commerce platforms or their independent websites, providing fundamental support for product retrieval technology based on product images.

Claims

1. An image encoder training method, characterized by, The method comprises the following steps: obtaining an image encoder with the same network architecture but different feature scale sizes, wherein the first image encoder with a larger feature scale size is trained to convergence to constitute a teacher model in a knowledge distillation network, and the second image encoder with a smaller feature scale size is to be trained to constitute a student model in the knowledge distillation network; synchronously inputting an image sample into the teacher model and the student model for feature extraction by feature representation layers in the teacher model and the student model, determining an average loss value between features extracted by corresponding feature representation layers of the teacher model and the student model; determining a prediction loss value corresponding to an inference result output by the student model, and fusing the prediction loss value with the average loss value to obtain a model loss value corresponding to the image sample; when the student model does not reach a convergence state, correcting weights of the student model according to the model loss value, and then continuing to synchronously input other image samples into the teacher model and the student model to start the next iteration training, and iteratively training until the student model is trained to the convergence state; obtaining a product image of a product item in a product information library; inputting the product image into the converged student model to obtain an inference result output by a last feature representation layer of the student model as a feature representation of the product image; storing the feature representation of the product image in association with the product image in the product information library.

2. The image encoder training method of claim 1, wherein The method comprises the following steps: performing scale transformation on feature representations obtained by each feature representation layer of the student model to transform the feature representations to the same feature scale as feature representations obtained by corresponding feature representation layers of the teacher model; calculating a feature loss value between corresponding features of each group of feature representations based on the feature representations of each group of feature representation layers, as a feature loss value corresponding to the feature representation layer of each group; averaging the feature loss values of each group of feature representation layers to obtain the average loss value corresponding to all groups of feature representation layers.

3. The image encoder training method of claim 2, wherein, The method comprises the following steps: averaging pool each feature representation in the same group of feature representations based on a plurality of pool steps to compress each feature representation into a plurality of feature maps corresponding to the plurality of steps, and each feature map comprises a plurality of channel vectors; calculating a data distance between each two corresponding channel vectors of the same group of feature representations based on the corresponding relationship among the pool steps, the feature maps and the channel vectors; averaging the data distances between all corresponding channel vectors in the same group of feature representations to obtain the feature loss value corresponding to the group of feature representations.

4. The image encoder training method of claim 1, wherein, The method comprises the following steps: The smoothing loss function is applied to calculate a smoothing loss of an inference result output by the student model based on a feature representation of a plurality of product images in a preset image feature library, as a prediction loss value of the student model, the inference result being an output of a last feature representation layer in the student model; The prediction loss value and the average loss value are fused by weighting to obtain a model loss value.

5. The image encoder training method of claim 1, wherein After storing the feature representation of the product image in the product information library in association with the product image, the method further includes: In response to a product recall request, obtaining a feature representation of a product image of a product item specified by the request as a to-be-retrieved feature; Calculating a similarity between the to-be-retrieved feature and the feature representation of the product image of each product item in the product information library; According to the similarity, filtering out part of the product items from the product information library, packaging product information of the part of the product items to obtain a product information list; The product information list is used to respond to the product recall request.

6. The image encoder training method according to any one of claims 1 to 4, characterized by, Until the student model is trained to a convergent state, the method further includes: Taking the converged student model as a teacher model, taking a third image encoder with the same network architecture but a smaller feature scale as a student model, reconfiguring a knowledge distillation network and continuing to train the knowledge distillation network to train the third image encoder to a convergent state.

7. An image encoder training apparatus, characterized by comprising: The method further includes: A training preparation module is configured to obtain image encoders with the same network architecture but different feature scale sizes, the image encoders including a plurality of serially arranged feature representation layers, wherein a first image encoder with a larger feature scale size has been trained to a convergent state to constitute a teacher model in a knowledge distillation network, and a second image encoder with a smaller feature scale size is to be trained to constitute a student model in the knowledge distillation network; A feature representation module is configured to synchronously input image samples to the teacher model and the student model for feature extraction by feature representation layers in the teacher model and the student model, and determine an average loss value between features extracted by corresponding feature representation layers of the teacher model and the student model; A loss determination module is configured to determine a prediction loss value corresponding to an inference result output by the student model, and fuse the prediction loss value with the average loss value to obtain a model loss value corresponding to the image sample; An iterative processing module is configured to, when the student model does not reach a convergent state, correct weights of the student model according to the model loss value, and then continue to call other image samples to be synchronously input to the teacher model and the student model to start the next iteration training, and so on, until the student model is trained to a convergent state; An image acquisition module is configured to obtain product images of product items in a product information library; A model inference module is configured to input the product images to the converged student model to obtain an inference result output by a last feature representation layer in the student model as a feature representation of the product images; A feature storage module is configured to store the feature representation of the product images in the product information library in association with the product images.

8. The image encoder training apparatus of claim 7, wherein, The device further includes: The request response module is configured to respond to a product recall request, obtain a feature representation of a product image of a product item specified in the request as a feature to be searched; The similarity calculation module is configured to calculate similarities between the feature to be searched and feature representations of product images of each product item in the product information library; The list packaging module is configured to filter out part of the product items from the product information library according to the similarities, package product information of the part of the product items to obtain a product information list; and the request response module is configured to respond to the product recall request with the product information list.

9. An image encoder training apparatus comprising a central processing unit and a memory, characterized in that, The central processing unit is configured to call and run a computer program stored in the memory to perform steps of the method according to any one of claims 1 to 6.

10. A non-volatile readable storage medium, characterized by The computer program is stored in the form of computer readable instructions and is implemented according to the method of any one of claims 1 to 6. When the computer program is called and run by a computer, steps included in the corresponding method are performed.

Images