Picture analysis model training method, advertisement picture selection method and electronic device

Abstract

Embodiments of the present application disclose a picture analysis model training method, an advertisement picture selection method and an electronic device. The picture analysis model training method comprises: inputting at least one group of sample vectors into a picture analysis model to obtain a first probability corresponding to each group of sample vectors; each group of sample vectors is generated based on a first advertisement and a second advertisement with the same advertisement language on a same advertisement position, and comprises the advertisement language, a first picture used by the first advertisement, a first text used by the first picture, a second picture used by the second advertisement, and a second text used by the second picture; the first probability represents a probability that the first picture is better than the second picture in the corresponding sample vector; a click rate of the first advertisement is higher than a click rate of the second advertisement; a loss value corresponding to each group of sample vectors in the at least one group of sample vectors is calculated based on the first probability corresponding to each group of sample vectors; and a weight parameter of the picture analysis model is updated according to the calculated loss value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a picture analysis model training method, an advertisement picture selection method and an electronic device. BACKGROUND

[0002] Internet advertisements are mostly composed of texts and pictures. In the related art, the visual sensory effect of a picture is evaluated by the resolution, texture, blank amount, edge gradient and other features of the picture, so that the picture with the best visual sensory effect is used for advertising, however, this may cause the content of the advertisement picture to be mismatched with the corresponding text, thereby resulting in a low click rate of the advertisement. SUMMARY

[0003] Therefore, the embodiments of the present application provide a picture analysis model training method, an advertisement picture selection method and an electronic device to solve the technical problem of a low click rate of an advertisement caused by the content of the selected advertisement picture being mismatched with the corresponding text in the related art.

[0004] To achieve the above object, the technical scheme of the present application is as follows:

[0005] The embodiments of the present application provide a picture analysis model training method, comprising:

[0006] inputting at least one group of sample vectors into a picture analysis model to obtain a first probability corresponding to each group of sample vectors; wherein each group of sample vectors is generated based on a first advertisement and a second advertisement with the same advertisement language on the same advertisement position, and includes the advertisement language, a first picture used by the first advertisement, a first text used by the first picture, a second picture used by the second advertisement and a second text used by the second picture; the first probability represents the probability that the first picture is better than the second picture in the corresponding sample vector; and the click rate of the first advertisement is higher than that of the second advertisement;

[0007] calculating a loss value corresponding to each group of sample vectors in the at least one group of sample vectors based on the first probability corresponding to each group of sample vectors;

[0008] updating the weight parameters of the picture analysis model according to the calculated loss value.

[0009] In the above scheme, the picture analysis model includes a text processing network, an image processing network, an attention network and a binary classification network; when the at least one group of sample vectors is input into the picture analysis model to obtain the first probability corresponding to each group of sample vectors, the method comprises:

[0010] input the advertisement, the first text and the second text into the text processing network to obtain a first global feature vector, a second global feature vector, a first text feature vector and a second text feature vector; wherein the first global feature vector represents global features of the advertisement and the first text, the second global feature vector represents global features of the advertisement and the second text, the first text feature vector is obtained based on the first global feature vector, and the second text feature vector is obtained based on the second global feature vector;

[0011] input the first picture and the first global feature vector into the image processing network to obtain a first image feature vector, and input the second picture and the second global feature vector into the image processing network to obtain a second image feature vector;

[0012] input the first text feature vector and the first image feature vector into the attention network to obtain a first picture vector, and input the second text feature vector and the second image feature vector into the attention network to obtain a second picture vector;

[0013] input the first picture vector and the second picture vector into the binary classification network to obtain corresponding first probabilities.

[0014] In the above scheme, the text processing network comprises a text encoder and a feedforward neural network; the inputting of the advertisement, the first text and the second text into the text processing network to obtain the first global feature vector, the second global feature vector, the first text feature vector and the second text feature vector comprises:

[0015] concatenate the advertisement and the first text to obtain a first sentence, and concatenate the advertisement and the second text to obtain a second sentence;

[0016] input the first sentence and the second sentence into the text encoder to obtain the first global feature vector and the second global feature vector;

[0017] input the first global feature vector and the second global feature vector into the feedforward neural network to obtain the first text feature vector and the second text feature vector.

[0018] In the above scheme, the image processing network comprises a convolution layer, a residual network and a fully connected layer; when the first picture and the first global feature vector are input into the image processing network to obtain the first image feature vector, and the second picture and the second global feature vector are input into the image processing network to obtain the second image feature vector, the method comprises:

[0019] input a first picture set into the convolution layer to obtain a first image tensor; the first picture set is composed of all the first pictures or all the second pictures in the at least one group of sample vectors;

[0020] inputting the first image tensor and the first text tensor into a residual network to obtain a second image tensor; the first text tensor is generated by all first global feature vectors or all second global feature vectors;

[0021] inputting the second image tensor into a full connection layer to obtain an image feature vector corresponding to each picture in the first picture set; wherein the image feature vector is a first image feature vector or a second image feature vector.

[0022] In the above scheme, the residual network includes at least one residual block, each residual block includes at least one residual unit, and each residual unit includes a convolution network, a first convolution layer, a first full connection layer, a second full connection layer and a normalization layer; when the first image tensor and the first text tensor are input into the residual unit to obtain the second image tensor, the method comprises:

[0023] inputting the first image tensor into the convolution network to obtain a third image tensor;

[0024] inputting the first image tensor into the first convolution layer to obtain a fourth image tensor;

[0025] adding the third image tensor and the fourth image tensor to obtain a fifth image tensor;

[0026] inputting the first text tensor into the first full connection layer to obtain a first parameter tensor; the first parameter tensor represents a first association strength between each picture in the first picture set and the corresponding text;

[0027] inputting the first text tensor into the second full connection layer to obtain a second parameter tensor; the second parameter tensor represents a second association strength between each picture in the first picture set and the corresponding text;

[0028] inputting the fifth image tensor, the first parameter tensor and the second parameter tensor into the normalization layer to obtain the second image tensor.

[0029] In the above scheme, when the first picture vector and the second picture vector are input into the binary classification network to obtain the corresponding first probability, the method comprises:

[0030] inputting the first picture vector and the second picture vector into the binary classification network, and using a set activation function to normalize the difference between the first product and the second product to obtain the corresponding first probability; wherein the first product represents the product between the first weight parameter and the first picture vector; the second product represents the product between the second weight parameter and the second picture vector.

[0031] In the above scheme, before the at least one group of sample vectors is input into the picture analysis model to obtain the first probability corresponding to each group of sample vectors, the method further comprises:

[0032] obtaining advertisement data of the advertisement; the advertisement data comprises an advertisement position identifier, an advertisement text, a picture, and a click rate;

[0033] determining, from the obtained picture corresponding to the advertisement, at least one second picture set in which the advertisement position identifier and the advertisement text are the same; each second picture set comprises one picture corresponding to the advertisement;

[0034] sorting the pictures in each second picture set based on the click rate corresponding to the picture, to obtain a corresponding third picture set;

[0035] generating a first sample set based on the arrangement order of the pictures in each third picture set and the corresponding advertisement text; wherein the first sample set comprises at least one group of sample vectors.

[0036] The embodiment of the application further provides an advertisement picture selection method, comprising:

[0037] generating at least one group of first vectors corresponding to the plurality of candidate pictures based on a first advertisement text, each two candidate pictures, and corresponding text information;

[0038] inputting each group of generated first vectors into a first model to obtain a second probability corresponding to each group of first vectors;

[0039] determining an advertisement picture from the plurality of candidate pictures based on the arrangement order of the picture corresponding to each group of first vectors and the second probability corresponding to each group of first vectors; wherein

[0040] The first model is a picture analysis model trained by using any one of the picture analysis model training methods described above; the second probability represents a probability that a first candidate picture is better than a second candidate picture; and the first candidate picture is before the second candidate picture.

[0041] The embodiment of the application further provides an electronic device, comprising:

[0042] a training unit configured to input at least one group of sample vectors into a picture analysis model to obtain a first probability corresponding to each group of sample vectors; wherein each group of sample vectors is generated based on a first advertisement and a second advertisement with the same advertisement text on a same advertisement position, and comprises an advertisement text, a first picture used by the first advertisement, a first text used by the first picture, a second picture used by the second advertisement, and a second text used by the second picture; the first probability represents a probability that the first picture is better than the second picture in the corresponding sample vector; and a click rate of the first advertisement is higher than a click rate of the second advertisement;

[0043] a calculation unit, configured to calculate a loss value corresponding to each of the at least one set of sample vectors based on a first probability corresponding to each of the at least one set of sample vectors;

[0044] an updating unit, configured to update a weight parameter of the picture analysis model according to the calculated loss value.

[0045] Embodiments of the present application further provide an electronic device, comprising:

[0046] a generation unit, configured to generate at least one set of first vectors corresponding to the plurality of candidate pictures based on the first advertisement sentence, each two candidate pictures, and corresponding text information;

[0047] a processing unit, configured to input each of the generated set of first vectors into the first model to obtain a second probability corresponding to each of the set of first vectors;

[0048] a determination unit, configured to determine an advertisement picture from the plurality of candidate pictures based on a picture arrangement order corresponding to each of the set of first vectors and the second probability corresponding to each of the set of first vectors; and

[0049] The first model is a picture analysis model trained by using any one of the picture analysis model training methods described above; the second probability represents a probability that a first candidate picture is better than a second candidate picture; and the first candidate picture is before the second candidate picture.

[0050] Embodiments of the present application further provide an electronic device, comprising a processor and a memory for storing a computer program capable of running on the processor,

[0051] When the processor runs the computer program, the processor is configured to perform at least one of the following:

[0052] steps of any one of the picture analysis model methods described above;

[0053] steps of the advertisement picture selection method described above.

[0054] Embodiments of the present application further provide a storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement at least one of the following:

[0055] steps of any one of the picture analysis model training methods described above;

[0056] steps of the advertisement picture selection method described above.

[0057] In the embodiment of the present application, the picture analysis model is trained by at least one set of sample vectors. In the training process, the loss value corresponding to each set of sample vectors in the at least one set of sample vectors is calculated based on the first probability corresponding to each set of sample vectors in the at least one set of sample vectors; and the weight parameters of the picture analysis model are updated according to the calculated loss value. Since in the training process, each set of training samples is generated based on the same first advertisement and second advertisement of the same advertisement position, including the advertisement language, the first picture used by the first advertisement, the first text used by the first picture, the second picture used by the second advertisement, and the second text used by the second picture; the first probability represents the probability that the first picture is better than the second picture in the corresponding sample vector, and the click rate of the first advertisement is higher than that of the second advertisement, that is, the advertisement position and the advertisement language corresponding to each set of sample vectors are the same, and the difference in click rate corresponding to different pictures in each set of sample vectors is caused by the difference in picture content. Therefore, based on the probability output by the trained picture analysis model, the picture with higher click rate and matching with the advertisement can be accurately predicted from the two pictures, and the picture with the best advertisement effect can be selected from the multiple pictures to launch the advertisement. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 An implementation flowchart of the picture analysis model training method provided by the embodiment of the present application is shown in the figure.

[0059] Figure 2 An implementation flowchart of the picture analysis model training method provided by the embodiment of the present application is shown in the figure.

[0060] Figure 3 An implementation flowchart of the picture analysis model training method provided by the embodiment of the present application is shown in the figure.

[0061] Figure 4 An implementation flowchart of the picture analysis model training method provided by the embodiment of the present application is shown in the figure.

[0062] Figure 5 An implementation flowchart of the picture analysis model training method provided by the embodiment of the present application is shown in the figure.

[0063] Figure 6 An implementation flowchart of the picture analysis model training method provided by the embodiment of the present application is shown in the figure.

[0064] Figure 7 An implementation flowchart of the picture analysis model training method provided by the embodiment of the present application is shown in the figure.

[0065] Figure 8 An implementation flowchart of the picture analysis model training method provided by the embodiment of the present application is shown in the figure.

[0066] Figure 9 An implementation flowchart of the picture analysis model training method provided by the embodiment of the present application is shown in the figure.

[0067] Figure 10 An implementation flowchart of the advertisement picture selection method provided by the embodiment of the present application is shown in the following table.

[0068] Figure 11 A structural schematic diagram of the electronic device provided by the embodiment of the present application is shown in the following table.

[0069] Figure 12 A structural schematic diagram of the electronic device provided by another embodiment of the present application is shown in the following table.

[0070] Figure 13 A hardware component structural schematic diagram of the electronic device provided by the embodiment of the present application is shown in the following table. DETAILED DESCRIPTION

[0071] In recent years, Internet advertisement has become one of the most common advertisement forms. Whether it is search engine advertisement, display package advertisement, or real-time bidding advertisement, it is generally composed of text and pictures. Among them, the text is the theme of the advertisement. In order to improve the click rate of the advertisement, it is necessary to select an advertisement picture for the advertisement. A suitable advertisement picture has the following characteristics:

[0072] 1. High picture quality, for example, resolution that allows users to distinguish details, and suitable picture size;

[0073] 2. Suitable picture content, for example, the advertisement text is a certain brand and model of mobile phone, and the picture is a picture of the same brand and model of mobile phone;

[0074] 3. There are attractive content elements in the picture, such as text description, picture color, and product details displayed in the picture.

[0075] Based on this, in each embodiment of the present application, the electronic device determines a set of sample vectors based on each two advertisements with the same advertisement text on the same advertisement position, trains a picture analysis model based on at least one set of sample vectors, and obtains a trained picture analysis model. Since each set of sample vectors is generated based on a first advertisement and a second advertisement with the same advertisement text on the same advertisement position, it includes the advertisement text, a first picture used by the first advertisement, a first text used by the first picture, a second picture used by the second advertisement, and a second text used by the second picture; the click rate of the first advertisement is higher than that of the second advertisement, that is, the advertisement position and the advertisement text corresponding to each set of sample vectors are the same, and the difference in click rate corresponding to different pictures in each set of sample vectors is caused by the difference in picture content. Therefore, the probability output by the trained picture analysis model can accurately predict the picture with high click rate and matching with the advertisement in each two pictures, and then select the picture with the best advertisement effect from multiple pictures to launch the advertisement.

[0076] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0077] Figure 1 The implementation flowchart of the picture analysis model training method provided by the embodiments of the present application is shown in FIG. 1. The execution subject of the flowchart is an electronic device such as a terminal or a server. As shown in FIG. 1, the picture analysis model training method comprises the following steps. Figure 1

[0078] Step 101: inputting at least one group of sample vectors into a picture analysis model to obtain a first probability corresponding to each group of sample vectors; wherein each group of sample vectors is generated based on the same first and second advertisements with the same advertisement language on the same advertisement position, and comprises the advertisement language, a first picture used by the first advertisement, a first text used by the first picture, a second picture used by the second advertisement, and a second text used by the second picture; the first probability represents the probability that the first picture is better than the second picture in the corresponding sample vector; and the click rate of the first advertisement is higher than that of the second advertisement.

[0079] Here, the electronic device determines at least one group of sample vectors, inputs the at least one group of sample vectors into the picture analysis model, and processes each group of sample vectors through the picture analysis model to obtain a first probability corresponding to each group of sample vectors. Each group of sample vectors in the at least one group of sample vectors can be generated based on each two advertisements with the same advertisement language on the same advertisement position. Of course, the advertisement language and the advertisement position corresponding to different groups of sample vectors can also be different. The electronic device can obtain each group of sample vectors according to any of the following methods.

[0080] The electronic device can generate a group of sample vectors based on the first and second advertisements with the same advertisement language on the same advertisement position. The schematic diagram of the advertisement language and the advertisement position is shown in FIG. 2. Figure 2

[0081] The electronic device can generate a group of sample vectors based on each two pictures in a picture set corresponding to the advertisements with the same advertisement position and the same advertisement language. The picture set includes one picture of each advertisement.

[0082] The electronic device can also determine a group of samples from a sample set corresponding to the advertisements with the same advertisement position and the same advertisement language, and determine a vector corresponding to each sample in each group of samples to obtain at least one group of sample vectors. Each sample set includes a plurality of groups of samples, and each group of samples includes at least a first picture used by a first advertisement, a first text used by the first picture, a second picture used by a second advertisement, and a second text.

[0083] ​​The picture set or the sample set can exist in a local database of the electronic device or a remote database. The first text and the second text are recognized from the corresponding pictures by the electronic device using an optical character recognition (OCR) technology.

[0084] In consideration of the fact that the more characters included in the first text or the second text, the longer the training time of the picture analysis model, in order to reduce the training time of the picture analysis model, in a case where the total length of the characters corresponding to the recognized text is greater than a first set threshold, the corresponding first text or second text is obtained by pruning in the order of small to large characters.

[0085] In actual application, in a case where the corresponding picture includes multiple texts, the multiple recognized texts are sorted in the order of large to small fonts; in a case where the total length of the characters corresponding to the recognized text is greater than a first set threshold, the corresponding first text or second text is obtained by pruning the sorted multiple texts in the order from back to front based on the first set threshold.

[0086] The implementation process of processing each group of sample vectors by the picture analysis model will be described in detail below.

[0087] Please refer to Figure 3 , Figure 3 The schematic diagram of processing each group of sample vectors by the picture analysis model provided by the embodiments of the present application is shown in FIG. 1. Figure 3 As shown in FIG. 1, the picture analysis model includes a text processing network, an image processing network, an attention network, and a binary classification network.

[0088] In order to improve the accuracy of the predicted first probability, in some embodiments, when the at least one group of sample vectors is input into the picture analysis model to obtain the first probability corresponding to each group of sample vectors, the method includes:

[0089] The advertisement text, the first text, and the second text are input into the text processing network to obtain a first global feature vector, a second global feature vector, a first text feature vector, and a second text feature vector; wherein the first global feature vector represents the global features of the advertisement text and the first text, the second global feature vector represents the global features of the advertisement text and the second text, the first text feature vector is obtained based on the first global feature vector, and the second text feature vector is obtained based on the second global feature vector.

[0090] The first picture and the first global feature vector are input into the image processing network to obtain a first image feature vector, and the second picture and the second global feature vector are input into the image processing network to obtain a second image feature vector.

[0091] inputting the first text feature vector and the first image feature vector into the attention network to obtain a first picture vector, and inputting the second text feature vector and the second image feature vector into the attention network to obtain a second picture vector;

[0092] inputting the first picture vector and the second picture vector into the binary classification network to obtain corresponding first probabilities.

[0093] Here, the electronic device performs the following processing on each group of sample vectors through the picture analysis model:

[0094] inputting the advertisement, the first text, and the second text in the first group of sample vectors into the text processing network to obtain a first global feature vector output by the text processing network based on the advertisement and the first text, a first text feature vector output based on the first global feature vector, and to obtain a second global feature vector output by the text processing network based on the advertisement and the second text, a second text feature vector output based on the second global feature vector. The first group of sample vectors is any group of sample vectors input into the picture analysis model. The first global feature vector, the second global feature vector, the first text feature vector, and the second text feature vector are all column vectors. Figure 3 In the above, characterize the advertisement, characterize the first text; characterize the second text, characterize the first global feature vector, characterize the second global feature vector, characterize the first text feature vector, characterize the second text feature vector. and are column vectors, and the dimensions of the column vectors are d T .

[0095] In order to enhance the nonlinear features of the global feature vector, in some embodiments, the text processing network includes a text encoder and a feedforward neural network; and the inputting the advertisement, the first text, and the second text into the text processing network to obtain the first global feature vector, the second global feature vector, the first text feature vector, and the second text feature vector includes:

[0096] concatenating the advertisement and the first text to obtain a first sentence, and concatenating the advertisement and the second text to obtain a second sentence;

[0097] inputting the first sentence and the second sentence into the text encoder to obtain the first global feature vector and the second global feature vector;

[0098] inputting the first global feature vector and the second global feature vector into the feedforward neural network to obtain the first text feature vector and the second text feature vector.

[0099] Here, as shown in Figure 4 , the text processing network comprises a text encoder and a feedforward neural network, the text encoder being a Bidirectional Encoder Representations from Transformers (BERT) model based on a transformer, the hidden layer width of the BERT model being d T . The feedforward neural network is composed of two cascaded fully connected layers, both of which use a Relu function as an activation function. Among them, the width of the fully connected layer 1 connected with the BERT model is 2048, which is used for amplification processing (or upsampling) of the global feature vector; the width of the fully connected layer 2 is d T , which is used for downsampling (or downsampling) of the feature vector output by the fully connected layer 1. The feedforward neural network is used to strengthen the nonlinear features of the global feature vector, that is, to enhance the expression ability of the text feature vector, so as to express the difference of the features in different scenarios.

[0100] The advertisement and the first text are spliced by elements to obtain a first sentence, and the advertisement and the second text are spliced by elements to obtain a second sentence; the first sentence and the second sentence are input into the text encoder to obtain a first global feature vector corresponding to the first sentence and a second global feature vector corresponding to the second sentence; the first global feature vector and the second global feature vector are input into the feedforward neural network to obtain a first text feature vector corresponding to the first global feature vector, and a second text feature vector corresponding to the second global feature vector.

[0101] Exemplarily, in the case of the advertisement , the first text , the advertisement and the first text are spliced by elements to obtain , wherein sep represents a separator of the BERT model, used to distinguish between two sentences.

[0102] In actual application, when the advertisement, the first text and the second text of each group of feature vectors in the at least one group of feature vectors are input into the text processing network, the text processing network outputs a first text tensor corresponding to all first global feature vectors or all second global feature vectors, and outputs corresponding based on the first text tensor , wherein characterize all the first text feature vectors or all the second text feature vectors.b characterizes the number of groups of the at least one group of sample vectors, characterize a real number.

[0103] In the case of obtaining the first global feature vector and the second global feature vector, input the first picture and the first global feature vector into the image processing network to obtain the first image feature vector, and input the second picture and the second global feature vector into the image processing network to obtain the second image feature vector. Wherein, in Figure 3 characterize the first picture, characterize the second picture, characterize the first image feature vector, characterize the second image feature vector.

[0104] In order to extract the correlation strength between the image features in the picture and the text features of the corresponding text, and improve the accuracy of the predicted first probability, in some embodiments, the image processing network comprises a cascaded convolutional layer, a residual network and a fully connected layer; when the first picture and the first global feature vector are input into the image processing network to obtain the first image feature vector, and the second picture and the second global feature vector are input into the image processing network to obtain the second image feature vector, the method comprises: inputting the first picture set into the convolutional layer to obtain the first image tensor; the first picture set is composed of all the first pictures or all the second pictures in the at least one group of sample vectors;

[0105] input the first image tensor and the first text tensor into the residual network to obtain the second image tensor; the first text tensor is generated by all the first global feature vectors or all the second global feature vectors;

[0106] input the second image tensor into the fully connected layer to obtain the image feature vector corresponding to each picture in the first picture set; wherein, the image feature vector is the first image feature vector or the second image feature vector.

[0107] Here, the electronic device processes each group of sample vectors through the picture processing network as follows: input the first picture set into the convolutional layer for convolution processing to obtain the first image tensor; input the first image tensor and the first text tensor into the residual network for processing to obtain the second image tensor; input the second image tensor into the fully connected layer for processing to obtain the image feature vector corresponding to each picture in the first picture set. The width of the fully connected layer is d T .

[0108] ​The residual network includes at least one residual block. It should be noted that when the image processing network includes a plurality of cascaded residual blocks, the input of the first residual block is the first image tensor, the input of the non-first residual block is the image tensor output by the connected last residual block, and the last residual block outputs the second image tensor. The number of layers in each residual block in the plurality of cascaded residual blocks can be partially the same or completely different, and is set according to actual conditions.

[0109] In some embodiments, each residual block includes at least one residual unit, and each residual unit includes a convolution network, a first convolution layer, a first full connection layer, a second full connection layer and a normalization layer; wherein the convolution network is composed of a plurality of cascaded convolution layers, the parameters of the convolution kernel of the first convolution layer are the same as those of the last convolution layer in the convolution network, the parameters of the convolution kernel include the size and the total number of the convolution kernel, and the total number of the convolution kernel represents the width of the convolution layer; the width of the first full connection layer and the second full connection layer is the same as that of the first convolution layer. In actual application, the image processing network is a ResNet network. As shown in Figure 5 The residual network in the image processing network includes four residual blocks, and the number of layers in the residual block 1 and the residual block 4 is the same. Each residual block includes nine cascaded residual units, and each residual unit includes a convolution network, a first convolution layer, a first full connection layer, a second full connection layer and a normalization layer.

[0110] In order to enhance the image features related to the text features in the picture and improve the accuracy of the predicted first probability, when the first image tensor and the first text tensor are input into the residual unit to obtain the second image tensor, the method includes:

[0111] The first image tensor is input into the convolution network to obtain a third image tensor;

[0112] The first image tensor is input into the first convolution layer to obtain a fourth image tensor;

[0113] The third image tensor and the fourth image tensor are added to obtain a fifth image tensor;

[0114] The first text tensor is input into the first full connection layer to obtain a first parameter tensor; the first parameter tensor represents the first association strength between each picture in the first picture set and the corresponding text;

[0115] The first text tensor is input into the second full connection layer to obtain a second parameter tensor; the second parameter tensor represents the second association strength between each picture in the first picture set and the corresponding text;

[0116] The fifth image tensor, the first parameter tensor and the second parameter tensor are input into the normalization layer to obtain the second image tensor.

[0117] Here, taking an example of a residual network including one residual block and the residual block including one residual unit, the first image tensor is input into the convolution network for convolution processing to obtain a third image tensor; the first image tensor is input into the convolution network for convolution processing to obtain a fourth image tensor; the third image tensor and the fourth image tensor are added to obtain a fifth image tensor; the first text tensor is input into the first fully connected layer to obtain a first parameter tensor The first text tensor is input into the second fully connected layer to obtain a second parameter tensor The fifth image tensor, the first parameter tensor and the second parameter tensor are input into the normalization layer for batch normalization processing to obtain a second image tensor. The second image tensor is a normalized vector.

[0118] It should be noted that, in the case where the residual block includes at least two residual units, the input of the first residual unit in the residual block is the input of the corresponding residual block, the input of the non-first residual unit is the output of the normalization layer in the connected last residual unit, and the output of the normalization layer in the last residual block is the output of the residual block.

[0119] It should be noted that, in the case where the residual network includes one residual block and the residual block includes one residual unit, the second image vector is the output of the residual network; in the case where the residual network includes at least two residual blocks and the residual block includes at least two residual units, the output of the last residual unit in the first residual block is the output of the first residual block, and the output of the first residual block is the input of the connected second residual block, and the output of the last residual unit in the last residual block is the output of the residual network, i.e., the image tensor output by the last residual unit in the last residual block is the second image tensor.

[0120] In actual application, the following formula is used for batch normalization processing:

[0121]

[0122] Wherein, represents the fifth image tensor, c represents the width of the first convolution layer, H represents the height of the picture, and W represents the width of the picture; represents the mathematical expectation of the vector represented by the fifth image tensor; represents the variance of the vector represented by the fifth image tensor; ε represents a very small constant, for example, ε = 10 -6 , used to avoid the case where the denominator is zero; * represents element-wise multiplication, also known as Hadamard product, i.e., tf.math.multiply function in tensorflow framework; α I and β Iall represent the weight parameters to be trained corresponding to the residual units, represent the first parameter tensor, represent the first text tensor, ω α and b α all represent the weight parameters to be trained corresponding to the first full connection layer; represent the second parameter tensor, ω β and b β all represent the weight parameters to be trained corresponding to the second full connection layer.

[0123] and respectively affect the variance and expectation of the input of each residual unit, and their values can strengthen the features related to the text in the picture. For example, the BERT model receives the text “mobile phone screen” and obtains the features related to “mobile phone screen” and parameters, which will strengthen the parameters of the channels related to “mobile phone screen” in the convolutional network in the residual unit, so that the ResNet finally extracts the features related to the text.

[0124] It should be noted that, is a tensor of b x c x H x W, and all are tensors of b x c, so the tensor multiplication and addition between them are broadcast mechanisms (such as the broadcast mechanism in numpy and tensorflow). That is, the same value is used to fill in and so that and all are tensors of b x c x H x W, and then element-wise multiplication and matrix addition operations are performed.

[0125] After obtaining the first image feature vector and the second image feature vector, the first text feature vector and the first image feature vector are input into the attention network to obtain a first picture vector, and the second text feature vector and the second image feature vector are input into the attention network to obtain a second picture vector. Wherein, in Figure 3 , represent the first picture vector, The second picture vector is characterized. The following takes the first text feature vector and the first image feature vector as an example to illustrate the specific implementation process of the attention network for processing the first text feature vector and the first image feature vector in detail:

[0126] The electronic device splices the first text feature vector and the first image feature vector to obtain wherein, d T The dimension of the first text feature vector is characterized.

[0127] The is input to the softmax layer in the attention network for processing to obtain the first attention matrix Att1, wherein the first attention matrix characterizes the mutual attention between the first text and the first picture, W1, W2 and C are all weight parameters to be trained in the attention network.

[0128] The first attention matrix and the first image feature vector are reduced in dimension in a summation manner to obtain the first picture vector with text attention appended wherein,

[0129] The second text feature vector and the second image feature vector are processed by the attention network in the above manner to obtain the second picture vector with text attention appended wherein,

[0130] In the case where the first picture vector and the second picture vector are obtained, the first picture vector and the second picture vector are input to the binary classification network, and the binary classification network is used to compare the first picture vector and the second picture vector to obtain the corresponding first probability.

[0131] In some embodiments, when the first picture vector and the second picture vector are input to the binary classification network to obtain the corresponding first probability, the method comprises:

[0132] The first picture vector and the second picture vector are input to the binary classification network, and a difference between a first product and a second product is normalized by using a set activation function to obtain the corresponding first probability; wherein the first product characterizes the product between the first weight parameter and the first picture vector; and the second product characterizes the product between the second weight parameter and the second picture vector.

[0133] Here, the electronic device inputs the first picture vector and the second picture vector into the binary classification network, calculates the difference between the first product and the second product through the binary classification network, and normalizes the difference by using a set activation function to obtain the corresponding first probability.

[0134] In actual application, in the case that the first picture is arranged before the second picture in each group of sample vectors, the first probability is calculated by using the formula , wherein p represents the first probability; W A and W B are weight parameters to be trained; W A represents the first weight parameter; W B represents the second weight parameter; and δ represents a Sigmoid function.

[0135] It should be noted that, in some embodiments, in the case that the second picture is arranged before the first picture in each group of sample vectors, the corresponding first probability is calculated by using the formula .

[0136] Step 102: Based on the corresponding first probability of each group of sample vectors in the at least one group of sample vectors, a loss value corresponding to each group of sample vectors in the at least one group of sample vectors is calculated.

[0137] Here, the electronic device adopts a set loss function to calculate the loss value corresponding to each group of sample vectors in the at least one group of sample vectors based on the corresponding first probability of each group of sample vectors in the at least one group of sample vectors.

[0138] In actual application, the set loss function is L i i = Relu [1-p i i]. L i represents the loss value corresponding to the i th group of sample vectors, and p i represents the first probability corresponding to the i th group of sample vectors. The Relu function is a linear function.

[0139] Step 103: The weight parameters of the picture analysis model are updated according to the calculated loss value.

[0140] Here, the electronic device can calculate an average loss value based on the loss value corresponding to each group of sample vectors in the at least one group of sample vectors to obtain the loss value of the picture analysis model; and update the weight parameters of the picture analysis model according to the loss value of the picture analysis model to improve the accuracy of the first probability output by the picture analysis model.

[0141] The electronic device back-propagates the loss value of the picture analysis model in the picture analysis model, calculates a gradient of a loss function according to the loss value in the process of back-propagating the loss value to each layer of the picture analysis model, and updates a weight parameter back-propagated to a current layer in a descending direction of the gradient.

[0142] The electronic device uses the updated weight parameter as a weight parameter used by the trained picture analysis model.

[0143] Here, an update stop condition can be set, and when the update stop condition is met, the weight parameter obtained by the last update is determined as the weight parameter used by the trained picture analysis model. The update stop condition can be, for example, a set training round (epoch), which is a process of training the picture analysis model according to at least one set of sample vectors. Of course, the update stop condition is not limited to this, and can be, for example, a set average accuracy (mAP, mean Average Precision).

[0144] It should be noted that in actual applications, model training is achieved by minimizing the loss function.

[0145] Back-propagation is relative to forward-propagation. Forward-propagation refers to a feedforward processing process of the model, and the direction of back-propagation is opposite to that of forward-propagation. Back-propagation refers to updating the weight parameters of each layer of the model according to the output result of the model. For example, the picture analysis model includes a text processing network, an image processing network, an attention network, and a binary classification network. Forward-propagation refers to processing in the order of the text processing network and the image processing network-attention network-binary classification network, and back-propagation refers to updating the weight parameters of each layer in the order of the binary classification network-attention network-image processing network and text processing network.

[0146] In the scheme provided by the embodiment, the picture analysis model is trained by at least one set of sample vectors. In the training process, the loss value corresponding to each set of sample vectors in the at least one set of sample vectors is calculated based on the first probability corresponding to each set of sample vectors in the at least one set of sample vectors; and the weight parameter of the picture analysis model is updated according to the calculated loss value. Since in the training process, each set of training samples is generated based on the same first advertisement and second advertisement of the same advertisement position, including the advertisement language, the first picture used by the first advertisement, the first text used by the first picture, the second picture used by the second advertisement, and the second text used by the second picture; the first probability represents the probability that the first picture is better than the second picture in the corresponding sample vector, and the click rate of the first advertisement is higher than that of the second advertisement, that is, the advertisement position and the advertisement language corresponding to each set of sample vectors are the same, and the difference in click rate corresponding to different pictures in each set of sample vectors is caused by the difference in picture content. Therefore, based on the probability output by the trained picture analysis model, the picture with higher click rate and better matching with the advertisement can be accurately predicted from each two pictures, and the picture with the best advertisement effect can be selected from the multiple pictures to launch the advertisement.

[0147] As shown in Figure 6 In some embodiments, before training the picture analysis model, it is also necessary to generate a sample set based on the advertisement data of the launched advertisement. Before step 101, the method further comprises:

[0148] Step 001: obtaining the advertisement data of the advertisement; the advertisement data includes the advertisement position identifier, the advertisement language, the picture and the click rate.

[0149] Here, the electronic device obtains the advertisement position identifier, the advertisement language, the picture and the click rate corresponding to each advertisement from the launched advertisement.

[0150] Step 002: determining at least one second picture set with the same advertisement position identifier and advertisement language from the obtained pictures corresponding to the advertisement; each second picture set includes one picture corresponding to the advertisement.

[0151] Considering that the embodiment of the present application mainly focuses on the advertisement effect of the picture, it is necessary to collect the pictures of different advertisements with the same advertisement position and the same advertisement language. Here, the electronic device classifies the obtained pictures corresponding to the advertisement based on the advertisement position identifier and the advertisement language in the advertisement data, and obtains at least one second picture set, and the advertisement position identifier and the advertisement language corresponding to each picture in each second picture set are the same.

[0152] Considering that in the training of the picture analysis model, two pictures corresponding to each set of sample vectors correspond to different advertisements, therefore, in order to facilitate the generation of sample vectors, when determining the second picture set, the second picture set includes one picture corresponding to the advertisement.

[0153] Step 003: Sort the images in each of the second image sets based on the click-through rate corresponding to the images, and obtain the corresponding third image set.

[0154] In actual applications, ads with the same ad position and ad slogan but different images are grouped as a unit group. The ad data corresponding to the j-th unit group can be expressed as (T j , I j , V j , P j ), where T j represents the ad slogan, and the ad slogan can be a string; P j represents the ad position identifier, I j represents the image set of the j-th unit group, and V j represents the click-through rate corresponding to each image in I j . Both I j and V j are vectors of the same length. Since the ad identifier and the ad slogan are unique within this unit group, T j and P j can be vectors of the corresponding length or both can be scalars.

[0155] Here, the click-through rate corresponding to each image in V j can be sorted in descending order of the click-through rate first. Since the subscripts of each element in I j and V​​​​​​​​​​​​​​​​​​​​​​​​​​​​​

[0158] Step 004: generating a first sample set based on the arrangement order of the pictures in each of the third picture set and the corresponding advertisement language; wherein the first sample set comprises the at least one group of sample vectors.

[0159] Here, the electronic device determines a plurality of groups of picture samples based on the arrangement order of the pictures in the third picture set and the corresponding advertisement language, determines a group of sample vectors based on each group of picture samples, and thus obtains the first sample set. Each group of picture samples comprises an advertisement language and two pictures with known relative advantages and disadvantages. The two pictures in each group of picture samples can be arranged in descending order of click rate or in ascending order of click rate. Each group of picture samples can be represented as In the case where the third picture set comprises z pictures, a total of groups of picture samples can be determined.

[0160] The method for determining a group of sample vectors based on each group of picture samples is to identify the corresponding text from each picture in each group of picture samples, and determine a group of sample vectors based on each group of picture samples and the corresponding text of each picture in each group of picture samples. Each group of sample vectors can be represented as characterizes the first picture, characterizes the second picture, from identifies, characterizes the first text used; from identifies, characterizes the second text used.

[0161] Considering that pictures with text descriptions can attract more user attention than pictures without text descriptions, pictures used for advertising usually have text descriptions. Therefore, in this embodiment, pictures with text descriptions are used to train the picture analysis model. However, as shown in Figure 7 not all text in the pictures helps the advertising effect of the pictures, therefore, the text content in the pictures is quite important for understanding the content of the pictures. Based on this, in this embodiment, the corresponding text is identified from each picture in each group of picture samples using an image-text recognition technology, for example, an OCR technology.

[0162] In the process of identifying the text corresponding to the picture, the text in different positions is regarded as a sentence, and all the recognized sentences are spliced into a piece of text. Different sentences can be separated by a period. Considering that the text with larger font is more eye-catching and more important, and the accuracy of the recognized text is also higher, therefore, when splicing different sentences, the splicing is performed in the order of font size from large to small. For example, the text in different positions is "Redmi K40" (large font) and "with old for new up to 3000 yuan E card" (small font), and the spliced text is: "Redmi K40. with old for new up to 3000 yuan E card".

[0163] In actual application, a maximum threshold can be set for the length of the text, and in the case that the length of the text recognized from the picture is greater than the maximum threshold d, the recognized text needs to be cropped before generating each group of sample vectors. Among them, considering that the text with larger font in the picture is relatively more important, therefore, in order to avoid important information being cut, the recognized text can be cropped in the order of character from small to large. In the case that the corresponding picture includes multiple pieces of text, the multiple pieces of recognized text are sorted in the order of font size from large to small; in the case that the total length of the characters corresponding to the recognized text is greater than d, the multiple pieces of sorted text are cropped in the order from back to front based on the set character length.

[0164] It should be noted that the same picture may appear in different unit groups. For example, a picture of sports shoes (picture 1) and a picture of high-heeled shoes (picture 2), if the advertising language is "shock absorption and breathability, cool summer", then obviously picture 1 is more suitable; and if the advertising language is "the latest style, highlighting leg shape", then picture 2 is more suitable. Therefore, in order to obtain a picture matching the advertising language, each group of picture samples determined contains not only pictures but also advertising languages.

[0165] In actual application, in this embodiment, the width of all pictures is W and the height is H. Among them, W and H can be set according to the maximum value of the picture size allowed by the advertising position. In the case that the width of the picture is not equal to W and / or the height is not equal to H, the picture is stretched or reduced based on the bilinear interpolation algorithm, so that the width of the processed picture is W and the height is H.

[0166] It should be noted that the larger the values of W, H and d are, the better the effect of the picture analysis model trained based on at least one group of sample vectors is, but correspondingly the training time is also longer, therefore, the effect of the picture analysis model and the training time need to be considered, and appropriate W, H and d are selected. In actual application, W and H can be 546, and d can be 64.

[0167] It should be noted that, in practical applications, the electronic devices that perform steps 001 to 004 may be the same as or different from the electronic devices that perform steps 101 to 103.

[0168] As another embodiment of this application, the image analysis model can be put into use after it has been trained. For example, in a scenario of selecting advertising images, the electronic device can use the image analysis model trained through the above embodiments to analyze the candidate images corresponding to the advertisement to be displayed, so as to select the image with the best advertising effect. It should be noted that the electronic device in the embodiment that trains the image analysis model may be different from the electronic device that uses the image analysis model to select advertising images in this embodiment.

[0169] Reference Figure 8 The process by which electronic devices select advertising images using a trained image analysis model is as follows:

[0170] Step 801: Based on the first advertising slogan, every two candidate images and the corresponding text information, generate at least one first vector corresponding to multiple candidate images.

[0171] Here, the electronic device acquires the first advertising slogan and multiple candidate images of the advertisement to be displayed, uses OCR technology to identify the corresponding text information from the candidate images, and generates at least one first vector corresponding to the multiple candidate images based on the first advertising slogan, every two candidate images and the corresponding text information.

[0172] For example, the first ad tagline of the ad to be delivered is τ, and there are n candidate images in sequence: v0, v1, v2…v n-1 The OCR technology is used to process each candidate image and extract the corresponding text information as t0, t1, t2…t n-1 Based on τ, every two candidate images and their corresponding text information, generate multiple sets of first vectors corresponding to n candidate images. Each set of first vectors can be represented as (τ, v i ,t i ,v g ,t g ), where i and g are both positive integers less than n.

[0173] It should be noted that if the length of the text information identified from the candidate image is greater than the set maximum text length, the identified text information will be cropped based on the set maximum text length. Please refer to the relevant description above for the cropping method, which will not be repeated here.

[0174] Each first vector group includes a first advertising slogan, a first candidate image, first text information used by the first candidate image, a second candidate image, and second text information used by the second candidate image.

[0175] Step 802: input each generated first vector into the first model to obtain a second probability corresponding to each first vector; wherein the first model is a picture analysis model trained by using any one of the picture analysis model training methods described above; the second probability represents a probability that a first candidate picture is better than a second candidate picture; and the first candidate picture is before the second candidate picture.

[0176] Here, the first model processes each first vector in a manner similar to the method of processing each sample vector by the picture analysis model described above, which is not repeated here.

[0177] Step 803: determine an advertisement picture from the plurality of candidate pictures based on the picture arrangement order corresponding to each first vector and the second probability corresponding to each first vector.

[0178] Here, the electronic device determines the optimal candidate picture from the plurality of candidate pictures based on the picture arrangement order corresponding to each first vector and the second probability corresponding to each first vector, and obtains the advertisement picture.

[0179] In the case where the number of the plurality of candidate pictures is n, n-1 rounds of comparison are needed to determine the optimal candidate picture from the plurality of candidate pictures.

[0180] In actual application, in the case where the second probability corresponding to the first vector (τ, v i ,t i ,v g ,t g ) is 1, it represents that v i is better than v g .

[0181] In an embodiment, the electronic device generates at least one group of first vectors corresponding to the plurality of candidate pictures based on the first advertisement language, each two candidate pictures, and the corresponding text information; inputs each generated first vector into the first model to obtain a second probability corresponding to each first vector; and determines an advertisement picture from the plurality of candidate pictures based on the second probability corresponding to each first vector. Since the second probability represents a probability that a first candidate picture is better than a second candidate picture, and the first candidate picture is before the second candidate picture, the electronic device can determine the optimal candidate picture from the plurality of candidate pictures based on the picture arrangement order corresponding to each first vector and the second probability corresponding to each first vector, and thus put the optimal candidate picture into an advertisement.

[0182] Figure 9 A schematic diagram of the advertisement picture selection method provided by the embodiment of the present application, Figure 10An implementation flowchart of an advertisement picture selection method provided by an embodiment of the present application is shown in FIG. 9. Figure 10 As shown in FIG. 9, the advertisement picture selection method includes the following steps.

[0183] Step 901: Obtain advertisement data of an advertisement; the advertisement data includes an advertisement position identifier, an advertisement phrase, a picture, and a click rate.

[0184] Step 902: From the pictures corresponding to the obtained advertisement, determine at least one second picture set in which the advertisement position identifier and the advertisement phrase are the same; each second picture set includes one picture corresponding to an advertisement.

[0185] Among them, Figure 9 The sampler in the above formula is used to determine the advertisement data of different advertisements with the same advertisement phrase on the same advertisement position.

[0186] Step 903: Based on the click rate corresponding to the picture, sort the pictures in each second picture set to obtain a corresponding third picture set.

[0187] Step 904: Based on the arrangement order of the pictures in each third picture set and the corresponding advertisement phrase, generate a first sample set; the first sample set includes at least one group of sample vectors.

[0188] Step 905: Randomly mix the samples in the first sample set, and take at least one group of sample vectors from the first sample set.

[0189] Here, the electronic device takes at least one group of first sample vectors from the first sample set in order.

[0190] In order to avoid overfitting of the picture analysis model, before each training, randomly mix each group of samples in the first sample set, and take at least one group of sample vectors from the mixed sample set.

[0191] Among them, the tensorflow.random.shuffle function of tensorflow can be used to mix the samples, so as to realize data randomization and avoid overfitting.

[0192] Step 906: Input at least one group of sample vectors into the picture analysis model to obtain a first probability corresponding to each group of sample vectors.

[0193] Step 907: Based on the first probability corresponding to each group of sample vectors in the at least one group of sample vectors, calculate a loss value corresponding to each group of sample vectors in the at least one group of sample vectors.

[0194] Step 908: Update the weight parameters of the picture analysis model according to the calculated loss value.

[0195] Step 909: output the trained picture analysis model to obtain a first model.

[0196] Step 910: obtain a first advertisement language corresponding to a to-be-launched advertisement and a plurality of candidate pictures.

[0197] Step 911: identify text information used in each candidate picture in the plurality of candidate pictures.

[0198] Step 912: based on the first advertisement language, each two candidate pictures, and the corresponding text information, generate at least one group of first vectors corresponding to the plurality of candidate pictures.

[0199] Step 913: input each generated group of first vectors into the first model to obtain a second probability corresponding to each group of first vectors; the second probability represents a probability that a first candidate picture is better than a second candidate picture; the first candidate picture is before the second candidate picture.

[0200] Step 914: based on the picture arrangement order corresponding to each group of first vectors and the second probability corresponding to each group of first vectors, determine an advertisement picture from the plurality of candidate pictures.

[0201] Step 915: based on the first advertisement language and the determined advertisement picture, generate a picture-text advertisement.

[0202] To implement the picture analysis model training method of the embodiments of the present application, the embodiments of the present application further provide an electronic device, as shown in the accompanying drawings, which comprises: Figure 11

[0203] The training unit 1101 is configured to input at least one group of sample vectors into a picture analysis model to obtain a first probability corresponding to each group of sample vectors; each group of sample vectors is generated based on a first advertisement and a second advertisement with the same advertisement language on the same advertisement position, and comprises an advertisement language, a first picture used by the first advertisement, first text used by the first picture, a second picture used by the second advertisement, and second text used by the second picture; the first probability represents a probability that the first picture is better than the second picture in the corresponding sample vector; and a click rate of the first advertisement is higher than a click rate of the second advertisement.

[0204] The calculation unit 1102 is configured to calculate a loss value corresponding to each group of sample vectors in the at least one group of sample vectors based on the first probability corresponding to each group of sample vectors.

[0205] The updating unit 1103 is configured to update a weight parameter of the picture analysis model according to the calculated loss value.

[0206] ​In some embodiments, the picture analysis model comprises a text processing network, an image processing network, an attention network, and a binary classification network; the training unit 1101 is specifically configured to:

[0207] input the advertisement phrase, the first text, and the second text into the text processing network to obtain a first global feature vector, a second global feature vector, a first text feature vector, and a second text feature vector; wherein the first global feature vector represents global features of the advertisement phrase and the first text, the second global feature vector represents global features of the advertisement phrase and the second text, the first text feature vector is obtained based on the first global feature vector, and the second text feature vector is obtained based on the second global feature vector;

[0208] input the first picture and the first global feature vector into the image processing network to obtain a first image feature vector, and input the second picture and the second global feature vector into the image processing network to obtain a second image feature vector;

[0209] input the first text feature vector and the first image feature vector into the attention network to obtain a first picture vector, and input the second text feature vector and the second image feature vector into the attention network to obtain a second picture vector;

[0210] input the first picture vector and the second picture vector into the binary classification network to obtain corresponding first probabilities.

[0211] In some embodiments, the text processing network comprises a text encoder and a feedforward neural network; the training unit 1101 is specifically configured to:

[0212] concatenate the advertisement phrase and the first text to obtain a first sentence, and concatenate the advertisement phrase and the second text to obtain a second sentence;

[0213] input the first sentence and the second sentence into the text encoder to obtain the first global feature vector and the second global feature vector;

[0214] input the first global feature vector and the second global feature vector into the feedforward neural network to obtain the first text feature vector and the second text feature vector.

[0215] In some embodiments, the image processing network comprises a convolutional layer, a residual network, and a fully connected layer; the training unit 1101 is specifically configured to:

[0216] input a first picture set into the convolutional layer to obtain a first image tensor; the first picture set is composed of all first pictures or all second pictures in the at least one group of sample vectors;

[0217] inputting the first image tensor and the first text tensor into the residual network to obtain a second image tensor; the first text tensor is generated by all first global feature vectors or all second global feature vectors;

[0218] inputting the second image tensor into a full connection layer to obtain an image feature vector corresponding to each picture in the first picture set; the image feature vector is a first image feature vector or a second image feature vector.

[0219] In some embodiments, the residual network comprises at least one residual block, each residual block comprises at least one residual unit, and each residual unit comprises a convolution network, a first convolution layer, a first full connection layer, a second full connection layer and a normalization layer; the training unit 1101 is specifically configured to:

[0220] inputting the first image tensor into the convolution network to obtain a third image tensor;

[0221] inputting the first image tensor into the first convolution layer to obtain a fourth image tensor;

[0222] adding the third image tensor and the fourth image tensor to obtain a fifth image tensor;

[0223] inputting the first text tensor into the first full connection layer to obtain a first parameter tensor; the first parameter tensor represents a first association strength between each picture in the first picture set and the corresponding text;

[0224] inputting the first text tensor into the second full connection layer to obtain a second parameter tensor; the second parameter tensor represents a second association strength between each picture in the first picture set and the corresponding text;

[0225] inputting the fifth image tensor, the first parameter tensor and the second parameter tensor into the normalization layer to obtain the second image tensor.

[0226] In some embodiments, the training unit 1101 is specifically configured to:

[0227] inputting the first picture vector and the second picture vector into the binary classification network, and using a set activation function to normalize the difference between the first product and the second product to obtain a corresponding first probability; the first product represents the product between the first weight parameter and the first picture vector; the second product represents the product between the second weight parameter and the second picture vector.

[0228] In some embodiments, the electronic device further comprises:

[0229] an acquisition unit configured to acquire advertisement data of an advertisement; the advertisement data comprises an advertisement position identifier, an advertisement text, a picture and a click rate;

[0230] determining unit configured to determine at least one second picture set in which the ad position identifier and the ad language are the same from the pictures corresponding to the obtained ads; each of the second picture sets includes one picture corresponding to an ad;

[0231] sorting unit configured to sort the pictures in each of the second picture sets based on the click rate corresponding to the pictures to obtain a corresponding third picture set;

[0232] generating unit configured to generate a first sample set based on the arrangement order of the pictures in each of the third picture sets and the corresponding ad language; the first sample set includes the at least one sample vector.

[0233] In actual application, each unit included in the electronic device can be implemented by a processor in the electronic device. Of course, the processor needs to run a program stored in a memory to implement the functions of the above-mentioned program modules.

[0234] It should be noted that: the electronic device provided in the above embodiments is only used as an example to illustrate the division of the above-mentioned program modules when training the picture analysis model. In actual application, the above-mentioned processing can be completed by different program modules according to needs, that is, the internal structure of the electronic device is divided into different program modules to complete all or part of the above-described processing. In addition, the electronic device provided in the above embodiments and the picture analysis model training method embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be described here.

[0235] To implement the ad picture selection method of the embodiments of the present application, the embodiments of the present application further provide an electronic device, as shown in Figure 12 The electronic device includes:

[0236] The generating unit 1201 is configured to generate at least one first vector corresponding to a plurality of candidate pictures based on the first ad language, each of the two candidate pictures and the corresponding text information;

[0237] The processing unit 1202 is configured to input each of the generated first vectors into a first model to obtain a second probability corresponding to each of the first vectors;

[0238] The determining unit 1203 is configured to determine an ad picture from the plurality of candidate pictures based on the arrangement order of the pictures corresponding to each of the first vectors and the second probability corresponding to each of the first vectors; wherein,

[0239] The first model is a picture analysis model trained by using the picture analysis model training method described above; the second probability represents the probability that the first candidate picture is better than the second candidate picture; and the first candidate picture is before the second candidate picture.

[0240] In actual application, each unit included in the electronic device can be implemented by a processor in the electronic device. Of course, the processor needs to run a program stored in the memory to implement the functions of the above-mentioned program modules.

[0241] It should be noted that the electronic device provided in the above embodiments only takes the division of the above-mentioned program modules as an example when selecting an advertisement picture. In actual application, the above-mentioned processing can be completed by different program modules according to needs, that is, the internal structure of the electronic device is divided into different program modules to complete all or part of the processing described above. In addition, the electronic device and the advertisement picture selection method provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be described here.

[0242] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of the present application, the embodiments of the present application further provide an electronic device. Figure 13 The hardware component structure of the electronic device of the embodiments of the present application is shown in FIG. 1. Figure 13 As shown in FIG. 1, the electronic device includes:

[0243] The communication interface 110 can interact with other devices such as network devices and the like.

[0244] The processor 120 is connected with the communication interface 110 to realize information interaction with other devices, and is used to run a computer program to execute the picture analysis model training method and / or the advertisement picture selection method provided in one or more technical solutions. The computer program is stored on the memory 130.

[0245] Of course, in actual application, each component in the electronic device is coupled together through a bus system 140. It can be understood that the bus system 140 is used to realize the connection and communication between the components. The bus system 140 includes not only a data bus, but also a power bus, a control bus and a status signal bus. However, in order to clearly illustrate, all kinds of buses are marked as the bus system 140 in the Figure 13

[0246] The memory 130 in the embodiments of the present application is used to store various types of data to support the operation of the electronic device. Examples of these data include any computer programs used to operate on the electronic device.

[0247] ​It can be understood that the memory 130 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM can be used, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), sync link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memory 130 described in the embodiments of the present application is intended to include, but not limited to, these and any other suitable type of memory.

[0248] The method disclosed in the embodiments of the present application can be applied in the processor 120 or implemented by the processor 120. The processor 120 can be an integrated circuit chip having a processing capability of signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 120 or the instruction in the form of software. The processor 120 described above can be a general processor, a DSP, or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The processor 120 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiments of the present application, the execution can be directly completed by the hardware decoding processor or by the combination of hardware and software modules in the decoding processor. The software module can be located in the storage medium, which is located in the memory 130. The processor 120 reads the program in the memory 130 and combines the hardware to complete the steps of the above method.

[0249] The processor 120 implements the corresponding flow in each method of the embodiments of the present application when executing the program. For brevity, it will not be repeated here.

[0250] In the exemplary embodiments, the embodiments of the present application also provide a storage medium, i.e. a computer storage medium, specifically a computer readable storage medium, such as the memory 130 storing the computer program, which can be executed by the processor 120 to complete the steps of the above method. The computer readable storage medium can be FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM, etc.

[0251] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, another division mode can be used, such as: a plurality of units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed components can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0252] The units described as separate parts above can or can not be physically separate, and the parts shown as units can or can not be physical units, that is, can be located in one place or distributed on multiple network units; part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0253] In addition, each functional unit in each embodiment of the present application can be integrated into one processing module, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

[0254] Those skilled in the art can understand that all or part of the steps of the above method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program executes the steps including the above method embodiments when executed; and the foregoing storage medium includes mobile storage device, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and various storage medium capable of storing program codes.

[0255] It should be noted that the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0256] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for training an image analysis model, characterized in that, include: At least one set of sample vectors is input into an image analysis model to obtain a first probability corresponding to each set of sample vectors. Each set of sample vectors is generated based on a first and a second advertisement with the same advertising slogan in the same ad slot, including the advertising slogan, a first image used in the first advertisement, a first text used in the first image, a second image used in the second advertisement, and a second text used in the second image. The first probability represents the probability that the first image is superior to the second image in the corresponding sample vector. The click-through rate (CTR) of the first advertisement is higher than that of the second advertisement. The character lengths of both the first and second texts are less than or equal to a first set threshold. The image analysis model includes a text processing network, an image processing network, an attention network, and a binary classification network. Based on the first probability corresponding to each group of sample vectors in the at least one group of sample vectors, the loss value corresponding to each group of sample vectors in the at least one group of sample vectors is calculated; The weight parameters of the image analysis model are updated based on the calculated loss value; When inputting at least one set of sample vectors into an image analysis model to obtain the first probability corresponding to each set of sample vectors, the method includes: The advertising slogan, the first text, and the second text are input into a text processing network to obtain a first global feature vector, a second global feature vector, a first text feature vector, and a second text feature vector. The first global feature vector represents the global features of the advertising slogan and the first text, and the second global feature vector represents the global features of the advertising slogan and the second text. The first text feature vector is obtained based on the first global feature vector, and the second text feature vector is obtained based on the second global feature vector. The first image and the first global feature vector are input into an image processing network to obtain a first image feature vector, and the second image and the second global feature vector are input into an image processing network to obtain a second image feature vector. The first text feature vector and the first image feature vector are input into an attention network to obtain a first image vector, and the second text feature vector and the second image feature vector are input into an attention network to obtain a second image vector. The first image vector and the second image vector are input into a binary classification network to obtain the corresponding first probability.

2. The method according to claim 1, characterized in that, The text processing network includes a text encoder and a feedforward neural network; the step of inputting the advertising slogan, the first text, and the second text into the text processing network to obtain a first global feature vector, a second global feature vector, a first text feature vector, and a second text feature vector includes: By concatenating the advertising slogan and the first text, we obtain the first statement; and by concatenating the advertising slogan and the second text, we obtain the second statement. The first and second statements are input into the text encoder to obtain the first global feature vector and the second global feature vector. The first global feature vector and the second global feature vector are input into the feedforward neural network to obtain the first text feature vector and the second text feature vector.

3. The method according to claim 1, characterized in that, The image processing network includes convolutional layers, residual networks, and fully connected layers; the method for inputting a first image and a first global feature vector into the image processing network to obtain a first image feature vector, and inputting a second image and a second global feature vector into the image processing network to obtain a second image feature vector, includes: The first set of images is input into a convolutional layer to obtain a first image tensor; the first set of images consists of all the first images or all the second images in the at least one set of sample vectors. The first image tensor and the first text tensor are input into the residual network to obtain the second image tensor; the first text tensor is generated by all first global feature vectors or all second global feature vectors. The second image tensor is input into the fully connected layer to obtain the image feature vector corresponding to each image in the first image set; wherein the image feature vector is either the first image feature vector or the second image feature vector.

4. The method according to claim 3, characterized in that, The residual network includes at least one residual block, each residual block includes at least one residual unit, and each residual unit includes a convolutional network, a first convolutional layer, a first fully connected layer, a second fully connected layer, and a normalization layer; the method of inputting a first image tensor and a first text tensor into the residual unit to obtain a second image tensor includes: The first image tensor is input into the convolutional network to obtain the third image tensor; The first image tensor is input into the first convolutional layer to obtain the fourth image tensor; Add the third image tensor to the fourth image tensor to obtain the fifth image tensor; The first text tensor is input into the first fully connected layer to obtain the first parameter tensor; the first parameter tensor represents the first association strength between each image in the first image set and the corresponding text. The first text tensor is input into the second fully connected layer to obtain the second parameter tensor; the second parameter tensor represents the second association strength between each image in the first image set and the corresponding text. The fifth image tensor, the first parameter tensor, and the second parameter tensor are input into the normalization layer to obtain the second image tensor.

5. The method according to claim 1, characterized in that, When inputting the first image vector and the second image vector into a binary classification network to obtain the corresponding first probability, the method includes: The first image vector and the second image vector are input into a binary classification network. The difference between the first product and the second product is normalized using a set activation function to obtain the corresponding first probability. The first product represents the product between the first weight parameter and the first image vector, and the second product represents the product between the second weight parameter and the second image vector.

6. The method according to any one of claims 1 to 5, characterized in that, Before inputting at least one set of sample vectors into the image analysis model to obtain the first probability corresponding to each set of sample vectors, the method further includes: Obtain advertising data for the advertisement; the advertising data includes ad placement identifier, ad copy, image, and click-through rate; From the images corresponding to the obtained advertisements, identify at least one second image set where both the advertisement placement identifier and the advertisement text are the same; each second image set includes one image corresponding to the advertisement; Based on the click-through rate of each image, the images in each of the second image sets are sorted to obtain the corresponding third image set; Based on the arrangement order of the images in each of the third image sets and the corresponding advertising slogans, a first sample set is generated; wherein the first sample set includes the at least one set of sample vectors.

7. A method for selecting advertising images, characterized in that, include: Based on the first advertising slogan, every two candidate images and corresponding text information, generate at least one first vector corresponding to multiple candidate images; Each generated first vector is input into the first model to obtain the second probability corresponding to each first vector. Based on the image arrangement order corresponding to each group of first vectors and the second probability corresponding to each group of first vectors, an advertising image is determined from the plurality of candidate images; wherein, The first model is an image analysis model trained using the image analysis model training method as described in any one of claims 1 to 6; the second probability represents the probability that the first candidate image is better than the second candidate image; the first candidate image is before the second candidate image; the image analysis model includes a text processing network, an image processing network, an attention network, and a binary classification network.

8. An electronic device, characterized in that, include: A training unit is used to input at least one set of sample vectors into an image analysis model to obtain a first probability corresponding to each set of sample vectors. Each set of sample vectors is generated based on a first advertisement and a second advertisement with the same advertising slogan in the same ad slot, including the advertising slogan, a first image used in the first advertisement, a first text used in the first image, a second image used in the second advertisement, and a second text used in the second image. The first probability represents the probability that the first image is superior to the second image in the corresponding sample vector. The click-through rate (CTR) of the first advertisement is higher than that of the second advertisement. The character lengths of both the first and second texts are less than or equal to a first set threshold. The image analysis model includes a text processing network, an image processing network, an attention network, and a binary classification network. The calculation unit is used to calculate the loss value corresponding to each group of sample vectors in the at least one group of sample vectors based on the first probability corresponding to each group of sample vectors in the at least one group of sample vectors; An update unit is used to update the weight parameters of the image analysis model based on the calculated loss value; The training unit is specifically used to input the advertising slogan, the first text, and the second text into a text processing network to obtain a first global feature vector, a second global feature vector, a first text feature vector, and a second text feature vector; wherein, the first global feature vector represents the global features of the advertising slogan and the first text, the second global feature vector represents the global features of the advertising slogan and the second text, the first text feature vector is obtained based on the first global feature vector, and the second text feature vector is obtained based on the second global feature vector; input the first image and the first global feature vector into an image processing network to obtain a first image feature vector, and input the second image and the second global feature vector into an image processing network to obtain a second image feature vector; input the first text feature vector and the first image feature vector into an attention network to obtain a first image vector, and input the second text feature vector and the second image feature vector into an attention network to obtain a second image vector; input the first image vector and the second image vector into a binary classification network to obtain the corresponding first probability.

9. An electronic device, characterized in that, include: The generation unit is used to generate at least one first vector corresponding to multiple candidate images based on the first advertising slogan, every two candidate images and the corresponding text information; The processing unit is used to input each generated first vector into the first model to obtain the second probability corresponding to each first vector. The determining unit is used to determine the advertising image from the plurality of candidate images based on the second probability corresponding to each group of first vectors; wherein, The first model is an image analysis model trained using the image analysis model training method as described in any one of claims 1 to 6; the second probability represents the probability that the first candidate image is better than the second candidate image; the first candidate image is before the second candidate image; the image analysis model includes a text processing network, an image processing network, an attention network, and a binary classification network.

10. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor runs the computer program, it performs at least one of the following: The steps of the image analysis model training method according to any one of claims 1 to 6; Perform the steps of the advertising image selection method as described in claim 7.

11. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it performs at least one of the following: The steps of the image analysis model training method according to any one of claims 1 to 6; The steps of the advertising image selection method according to claim 7.

Images