Multi-dimensional scoring model training method, system, equipment and medium

Abstract

The invention is suitable for the technical field of computers, and provides a multi-dimensional scoring model training method, which comprises the following steps: inputting text data into a text graph model to obtain a training image; inputting the text data and the training image into a pre-training scoring model to obtain text embedding and image embedding; obtaining final image embedding based on text embedding and image embedding; based on the final image embedding, obtaining a prediction score of the training image; multi-dimensional score data of the training image is acquired, a score training loss function is generated based on the multi-dimensional score data, the prediction score and a preset parameter, and the preset parameter is a parameter specified by a user and represents the intervention degree of the user on the score; and training the pre-training scoring model based on the scoring training loss function to obtain a multi-dimensional scoring model. According to the method, the multi-dimensional score prediction of the training image can be realized, and the generated multi-dimensional score model can output the multi-dimensional comprehensive score based on the user opinion and the expert opinion.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to a multi-dimensional scoring model training method, system, device and medium. Background Art

[0002] The text graph model automatically generates images from text descriptions, allowing the text content to be intuitively presented in a visual form. It is not only highly automated and accurate, but also highly scalable and customizable, and has broad application prospects. In addition, evaluating the generated text graph is an indispensable part of the text graph model design process. Through effective evaluation methods, problems in the text graph model can be discovered in a timely manner, such as inaccurate semantic understanding, unreasonable generated image structure, and insufficient picture details, thereby guiding the further optimization and improvement of the model.

[0003] In the prior art, two methods, Inception Score or CLIP, are usually used to evaluate the generated text graphs. Inception Score is an indicator to measure the realism of the generated model. Inception Net trained on the ImageNet dataset is usually used to classify the fake images generated by the evaluated text graph model. Fréchet Inception Distance (FID) embeds the image into the feature space and stops at a specific layer. It then assumes that the feature vector follows a Gaussian distribution and calculates the Fréchet distance between the distributions of the real and generated data. CLIP (Contrastive Language-Image Pre-training) is a model from OpenAI that can evaluate the semantic similarity between text and images. The score based on CLIP can be used as a metric to evaluate the performance of the text graph model by calculating the similarity between the text description and the generated image.

[0004] The above two methods for evaluating text images usually have the following defects: (1) Subjectivity of manual evaluation. Although manual evaluation can capture details and semantic relevance that machine evaluation may ignore, the evaluation results are easily affected by the subjective factors of the evaluators, resulting in inconsistency and bias in the results. (2) Limitation of evaluation dimensions. Existing automated evaluation indicators such as IS and FID often focus on the overall quality of the image or the similarity of a specific dimension, but it is difficult to comprehensively evaluate the performance of the image in multiple aspects such as details, style, and semantic relevance. Summary of the invention

[0005] The embodiments of the present application provide a multi-dimensional scoring model training method, system, device and medium, which can solve one of the above-mentioned prior art problems.

[0006] In a first aspect, an embodiment of the present application provides a multi-dimensional scoring model training method, comprising:

[0007] Input text data into the text-image model to obtain training images;

[0008] Inputting the text data and the training image into a pre-trained scoring model to obtain text embedding and image embedding;

[0009] Based on the text embedding and the image embedding, obtaining a final image embedding;

[0010] Based on the final image embedding, obtaining a predicted score for the training image;

[0011] Acquire multi-dimensional scoring data of the training image, and generate a scoring training loss function based on the multi-dimensional scoring data, the predicted score and preset parameters, wherein the preset parameters are parameters specified by the user and represent the degree of user intervention in the scoring;

[0012] Based on the scoring training loss function, the pre-trained scoring model is trained to obtain a multi-dimensional scoring model.

[0013] Furthermore, the pre-trained scoring model includes a text encoder and an image encoder;

[0014] The step of inputting the text data and the training image into a pre-trained scoring model to obtain text embedding and image embedding comprises:

[0015] Inputting the text data into the text encoder to obtain text embedding;

[0016] The training image is input into the image encoder to obtain image embedding, wherein the image embedding includes an initial image embedding and multiple hidden layer outputs, wherein the initial image embedding is the image embedding output of the image encoder, and the hidden layer output is the output of each hidden layer of the image encoder.

[0017] Furthermore, the text encoder includes a text feature extraction module, a text feature fusion module and a text feature output module;

[0018] The text feature module is used to extract text features, the text feature fusion module is used to fuse multiple text features, and the text feature output module is used to output text embedding;

[0019] The image encoder includes an image feature extraction module, an attention mechanism layer, an image feature fusion module and an image feature output module;

[0020] The image feature extraction module is used to extract the basic features of the image, the attention mechanism layer is used to extract the color features of the image, the image feature fusion module is used to fuse the basic features and the color features, and the image feature output module is used to output the initial image embedding;

[0021] The image feature extraction module, the attention mechanism layer and the image feature fusion module each include multiple hidden layers.

[0022] Furthermore, the calculation formula for the attention mechanism layer to extract the color features of the image is as follows:

[0023]

[0024] Wherein, T represents the attention of color features, X represents the color features of the image input into the attention mechanism layer, and μ(X) represents the average value of the color features input into the attention mechanism layer.

[0025] Furthermore, obtaining a final image embedding based on the text embedding and the image embedding includes:

[0026] Performing a similarity operation on the text embedding and the initial image embedding to obtain text-image similarity;

[0027] A final image embedding is generated based on the text-image similarity and a plurality of the hidden layer outputs.

[0028] Furthermore, obtaining a predicted score of the training image based on the final image embedding includes:

[0029] The final image is embedded into a multi-level evaluation network, and the predicted score of the training image is obtained through each hidden layer in the multi-level evaluation network.

[0030] Furthermore, the calculation formula of the scoring training loss function is as follows:

[0031]

[0032] Among them, α and β are two preset parameters used to indicate the degree of user intervention in the scoring of the training images, and p i represents the predicted score, y i Represents multi-dimensional rating data.

[0033] In a second aspect, the embodiment of the present application provides a multi-dimensional scoring model training system, including:

[0034] The first processing module is used to input text data into the text-generated graph model to obtain a training image;

[0035] A second processing module: used for inputting the text data and the training image into a pre-trained scoring model to obtain text embedding and image embedding;

[0036] A third processing module: used for obtaining a final image embedding based on the text embedding and the image embedding;

[0037] A fourth processing module: configured to obtain a predicted score of the training image based on the final image embedding;

[0038] A fifth processing module: used to obtain multi-dimensional scoring data of the training image, and generate a scoring training loss function based on the multi-dimensional scoring data, the predicted score and preset parameters, wherein the preset parameters are parameters specified by the user, indicating the degree of user intervention in the scoring;

[0039] The sixth processing module is used to train the pre-trained scoring model based on the scoring training loss function to obtain a multi-dimensional scoring model.

[0040] In a third aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned multi-dimensional scoring model training method when executing the computer program.

[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, including a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the multi-dimensional scoring model training method described above is implemented.

[0042] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0043] The present application discloses a multi-dimensional scoring model training method for a Vincent graph. By embedding the final image, the finally generated multi-dimensional scoring model can evaluate the training image from multiple angles, thereby realizing multi-dimensional scoring prediction for the training image. In addition, the preset parameters and multi-dimensional scoring data in the scoring training loss function make the final trained multi-dimensional scoring model output a comprehensive score consisting of expert opinions and user opinions, thereby avoiding the drawback that the existing scoring model can only represent expert opinions and eliminating the influence of expert subjective opinions on the Vincent graph scoring results. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0045] Figure 1 It is a flowchart of a multi-dimensional scoring model training method provided by one embodiment of the present invention;

[0046] Figure 2 is a schematic diagram of the structure of a multi-level evaluation network provided by an embodiment of the present invention;

[0047] Figure 3 It is a structural diagram of a multi-dimensional scoring model training system provided by an embodiment of the present invention;

[0048] Figure 4 It is a schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0050] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0051] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0052] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0053] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0054] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0055] See also Figure 1 As shown, the present invention is a multi-dimensional scoring model training method, comprising the following steps:

[0056] S100, inputting text data into a text-generated image model to obtain a training image;

[0057] The multi-dimensional scoring model of the present application is to score the Vincent graph generated by the Vincent graph model. Specifically, by judging whether the Vincent graph contains the correct number of subjects, for example, if the text data is two dogs, then the number of dogs in the generated Vincent graph must be two, judging whether the Vincent graph is deformed, judging whether the Vincent graph contains meaningless details, judging whether the style of the Vincent graph is aesthetic, etc. Based on the above judgment process, the multi-dimensional scoring model finally outputs a comprehensive score for quantifying the quality of the Vincent graph and then evaluating the performance of the Vincent graph model. Therefore, in this embodiment, the text data is input into the Vincent graph model, the Vincent graph is output by the Vincent graph model, and the Vincent graph is used as a training image for the multi-dimensional scoring model to train and generate the multi-dimensional scoring model.

[0058] S200, inputting the text data and the training image into a pre-trained scoring model to obtain text embedding and image embedding;

[0059] In one embodiment, the pre-trained scoring model includes a text encoder and an image encoder, wherein the above step S200 includes:

[0060] Inputting the text data into the text encoder to obtain text embedding;

[0061] The training image is input into the image encoder to obtain image embedding, wherein the image embedding includes an initial image embedding and multiple hidden layer outputs, wherein the initial image embedding is the image embedding output of the image encoder, and the hidden layer output is the output of each hidden layer of the image encoder.

[0062] In this embodiment, a multidimensional graph scoring model is generated by training a pre-trained scoring model. Specifically, the pre-trained scoring model includes an image encoder and a text encoder, which are used to learn scoring in multiple dimensions, so that the multi-dimensional scoring model finally generated can generate a comprehensive score for evaluating text graphs.

[0063] In some of the embodiments, the text encoder includes a text feature extraction module, a text feature fusion module and a text feature output module;

[0064] The text feature module is used to extract text features, the text feature fusion module is used to fuse multiple text features, and the text feature output module is used to output text embedding;

[0065] The image encoder includes an image feature extraction module, an attention mechanism layer, an image feature fusion module and an image feature output module;

[0066] The image feature extraction module is used to extract the basic features of the image, the attention mechanism layer is used to extract the color features of the image, the image feature fusion module is used to fuse the basic features and the color features, and the image feature output module is used to output the initial image embedding;

[0067] The image feature extraction module, the attention mechanism layer and the image feature fusion module each include multiple hidden layers.

[0068] In this embodiment, by clearly dividing the text encoder and the image encoder into multiple modules according to their functions, the text and image processing processes are clarified and modularized, thereby improving the readability and maintainability of the encoder. At the same time, it is convenient for developers to optimize or replace each module according to specific needs, thereby enhancing the flexibility and scalability of the encoder.

[0069] In this embodiment, the text feature extraction module includes a first text convolution layer and a second text convolution layer, wherein the first text convolution layer is used to preliminarily extract text features, and the second text convolution layer is used to further refine the text feature representation based on the text features extracted by the first text convolution layer to improve the performance of text feature extraction. In some preferred embodiments, the first text convolution layer includes a 5×5 convolution layer, a batch normalization layer and a relu activation function, and the second text convolution layer includes a 3×3 convolution layer, a batch normalization layer and a linear layer.

[0070] In this embodiment, the text feature fusion module is repeatedly stacked three times, and the output after each processing by the text feature fusion module is used as the input of the next iteration to gradually deepen and refine the text feature representation. In some preferred embodiments, the text feature fusion module includes a 1×1 convolution layer, a 2×2 average pooling layer, a batch normalization layer and a relu activation function.

[0071] In this embodiment, the text feature output module converts the final output of the text feature fusion module into the final output format required by the encoder, that is, outputs the text output features of the final layer. In some preferred embodiments, the text feature output module is a linear layer.

[0072] In this embodiment, the image encoder includes an image convolution layer and a pooling layer through an image feature extraction module, wherein the image convolution layer is used to preliminarily extract image features, and the pooling layer is used to downsample based on the image features extracted by the image convolution layer to reduce the data dimension while retaining important features of the image and generating basic features. In addition, the image encoder also includes an attention mechanism layer for enhancing the color features of the image. The basic features and color features are fused through the image feature fusion module. The image feature output module converts the final output of the image feature fusion module into the final output format required by the encoder, that is, outputs the image output features of the final layer. In some preferred embodiments, the image convolution layer includes a 3×3 convolution layer, a batch normalization layer and a relu activation function, the pooling layer is a 2×2 average pooling layer, and the image feature output module is a linear layer.

[0073] In this embodiment, the image feature fusion module is repeatedly stacked four times, and the output after each processing by the image feature fusion module is used as the input of the next iteration to gradually deepen and refine the image feature representation. In some preferred embodiments, the image feature fusion module includes a 5×5 convolution layer, a 2×2 maximum pooling layer, a batch normalization layer, a 1×1 convolution and an activation function.

[0074] In this embodiment, in the process of training the pre-trained scoring model to generate a multidimensional scoring model, the output of each hidden layer of the image encoder is extracted, and the final image embedding is calculated through the output of the hidden layer of the image encoder, the initial image embedding of the final layer output of the image encoder, and the text embedding of the output of the text encoder. The predicted score is obtained based on the final image embedding, and the pre-trained scoring model is trained using a scoring training loss function based on the predicted score.

[0075] In some embodiments, the calculation formula for the attention mechanism layer to extract the color features of the image is as follows:

[0076]

[0077] Wherein, T represents the attention of color features, X represents the color features of the image input into the attention mechanism layer, and μ(X) represents the average value of the color features input into the attention mechanism layer.

[0078] In this embodiment, the attention of the color feature is calculated by the above calculation formula, thereby enhancing or suppressing the color feature in the image.

[0079] S300, obtaining a final image embedding based on the text embedding and the image embedding;

[0080] In some embodiments, the step S300 includes:

[0081] The obtaining a final image embedding based on the text embedding and the image embedding comprises:

[0082] Performing a similarity operation on the text embedding and the initial image embedding to obtain text-image similarity;

[0083] A final image embedding is generated based on the text-image similarity and a plurality of the hidden layer outputs.

[0084] In this embodiment, the number of hidden layers in the image encoder is multiple, that is, the number of generated hidden layer outputs is multiple, such as Figure 2 As shown, the hidden layer output of each hidden layer is recorded as custom_attributes[i], where 0≤i≤n, n represents the number of hidden layers. In addition, the initial image embedding output of the final layer of the image encoder is recorded as origin_image_embedding, and the text embedding output of the final layer of the text encoder is recorded as text_embedding.

[0085] In this embodiment, the similarity between the text embedding and the initial image embedding is calculated by a dot product operation. Specifically, the dot product operation is a vector product, which can calculate the similarity of two vectors in direction, that is, if the direction of the text data and the image generated based on the text data is similar, the dot product value of the text embedding and the initial image embedding will be relatively large, indicating that the image and the text have a high similarity in content or semantics. In this embodiment, the text image similarity that measures the similarity between the initial image embedding and the text embedding is recorded as text_image_similarity.

[0086] In this embodiment, in order to construct a rich and comprehensive image embedding representation, information of multiple dimensions is integrated into the final image embedding. Specifically, a hidden layer output table custom_attributes is set, where custom_attributes is a list containing all hidden layer outputs, each of which represents a feature vector output by the image encoder at a certain hidden layer or an attribute after specific processing. These hidden layer outputs may capture different aspects of the image, such as color, texture, shape or higher-level semantic features. By using these hidden layer outputs as combined elements of the final image embedding, the characteristics of the image are more comprehensively described.

[0087] In one embodiment, the final image embedding is recorded as image_embedding, where the text image similarity is the first element of the final image embedding, and the text image similarity is combined with multiple hidden layer outputs to form a composite vector, then image_embedding = [text_image_similarity, custom_attributes[0], custom_attributes[1], ..., custom_attributes[n]]. Through the above combination, image_embedding not only contains the association information between the image and the text, but also integrates the multi-level and multi-dimensional features of the image itself, providing support for the training of the subsequent pre-trained scoring model.

[0088] S400, obtaining a predicted score of the training image based on the final image embedding;

[0089] In this embodiment, by extracting the output results of each hidden layer in the image encoder and calculating the text image similarity between the initial image embedding of the final layer of the image encoder and the text embedding of the text encoder, the two are combined to form a final image embedding as the input of the multi-level evaluation network, so that the multi-level evaluation network can evaluate the training image from multiple angles, thereby realizing multi-dimensional score prediction of the training image.

[0090] In one embodiment, the step S400 includes:

[0091] The final image is embedded into a multi-level evaluation network, and the predicted score of the training image is obtained through each hidden layer in the multi-level evaluation network.

[0092] In this embodiment, the final image generated in step S300 is embedded into a multi-level evaluation network, wherein the multi-level evaluation network is a neural network model, the neural network model includes an input layer, an output layer and multiple hidden layers, each layer is composed of multiple neurons, and the neural network model is obtained after training according to a preset network model. Specifically, image_embedding is used as the input of the multi-level evaluation network, that is, the feature vector of the training image is passed to the first layer of the multi-level evaluation network, and then each layer of the multi-level evaluation network further processes and transforms the feature vector, and extracts and abstracts higher-level information layer by layer through weighted summation, application of activation functions and other steps. Based on the above steps, multiple iterations are performed, and the output layer is continued, and the final output layer generates the final predicted score. Through the above method, the multi-level evaluation network can learn the complex relationship between image_embedding and predicted scores, and then provide data support for the training of subsequent pre-trained scoring models.

[0093] S500, obtaining multi-dimensional scoring data of the training image, and generating a scoring training loss function based on the multi-dimensional scoring data, the predicted score and preset parameters, wherein the preset parameters are parameters specified by the user and represent the degree of user intervention in the scoring;

[0094] In some embodiments, the calculation formula of the scoring training loss function is as follows:

[0095]

[0096] Among them, α and β are two preset parameters used to indicate the degree of user intervention in the scoring of the training images, and p i represents the predicted score, y i Represents multi-dimensional rating data.

[0097] In this embodiment, the comprehensiveness and accuracy of the final evaluation score of the training image can be improved by obtaining multi-dimensional scoring data of the training image. In one embodiment, the multi-dimensional scoring data is obtained by an expert scoring method. Specifically, according to predefined scoring criteria, multiple experts score the training images from multiple dimensions. These dimensions may include but are not limited to image clarity, color reproduction, composition and creativity, content accuracy, and technical defects. The scoring results of all experts are summarized and analyzed to obtain multi-dimensional scoring data.

[0098] In this embodiment, the above-mentioned scoring training loss function is provided with two preset parameters α and β, which are used to represent the degree of user intervention in the scoring, and are usually hyperparameters specified by the user, and 0<α<1, 0<β<1. It emphasizes the degree of user intervention in the final scoring. At the same time, the multi-dimensional scoring data is determined by the expert scoring. Therefore, the comprehensive score finally output by the multi-dimensional scoring model trained based on the above-mentioned scoring training loss function consists of two parts: expert opinion and user opinion, thereby avoiding the disadvantage that the existing scoring model can only represent expert opinions, and eliminating the influence of expert subjective opinions on the scoring results of the text diagram.

[0099] S600: Based on the scoring training loss function, the pre-trained scoring model is trained to generate a multi-dimensional scoring model.

[0100] In this embodiment, multiple text data are input into a text-image model to generate multiple training images, the multiple training images and multiple text data are preprocessed to generate a training set, the training set is input into a pre-trained scoring model for training, and at the same time, the scoring loss function value generated by the scoring training loss function is used as a feedback signal to guide the pre-trained scoring model to continuously optimize its internal parameters and generation strategy, and generate a multidimensional scoring model through multiple rounds of iterative training. Specifically, when the training times of the pre-trained scoring model reaches a threshold or the scoring loss function value of the pre-trained scoring model is less than the error threshold, the multidimensional scoring model is obtained.

[0101] See also Figure 3 As shown, the present invention also provides a multi-dimensional scoring model training system, the system comprising:

[0102] The first processing module 301 is used to input text data into the text-image model to obtain a training image;

[0103] The second processing module 302 is used to input the text data and the training image into a pre-trained scoring model to obtain text embedding and image embedding;

[0104] The third processing module 303 is used to obtain a final image embedding based on the text embedding and the image embedding;

[0105] The fourth processing module 304 is used to obtain a predicted score of the training image based on the final image embedding;

[0106] The fifth processing module 305 is used to obtain the multi-dimensional scoring data of the training image, and generate a scoring training loss function based on the multi-dimensional scoring data, the predicted score and preset parameters, wherein the preset parameters are parameters specified by the user and represent the degree of user intervention in the scoring;

[0107] The sixth processing module 306 is used to train the pre-trained scoring model based on the scoring training loss function to obtain a multi-dimensional scoring model.

[0108] It is understandable that if Figure 1 The contents of the multi-dimensional scoring model training method embodiment shown in the figure are applicable to the multi-dimensional scoring model training system embodiment. The functions specifically implemented by the multi-dimensional scoring model training system embodiment are similar to those in the embodiment shown in the figure. Figure 1 The multi-dimensional scoring model training method embodiment shown is the same as that shown in FIG. Figure 1 The beneficial effects achieved by the multi-dimensional scoring model training method embodiment shown are also the same.

[0109] It should be noted that the information interaction, execution process and other contents between the above-mentioned systems are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0110] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0111] See also Figure 4 As shown, an embodiment of the present invention further provides a computer device 4, comprising: a memory 402 and a processor 401 and a computer program 403 stored in the memory 402. When the computer program 403 is executed on the processor 401, a multi-dimensional scoring model training method as described in any one of the above methods is implemented.

[0112] The computer device 4 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 4 may include, but is not limited to, a processor 401 and a memory 402. Those skilled in the art will appreciate that Figure 4It is only an example of the computer device 4 and does not constitute a limitation on the computer device 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0113] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0114] In some embodiments, the memory 402 may be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 402 may also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 4. Further, the memory 402 may also include both an internal storage unit and an external storage device of the computer device 4. The memory 402 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 402 may also be used to temporarily store data that has been output or is to be output.

[0115] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the multi-dimensional scoring model training method as described in any one of the above methods is implemented.

[0116] In this embodiment, if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0117] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A multi-dimensional scoring model training method, characterized in that: include: Input text data into the text-image model to obtain training images; Inputting the text data and the training image into a pre-trained scoring model to obtain text embedding and image embedding; Based on the text embedding and the image embedding, obtaining a final image embedding; Based on the final image embedding, obtaining a predicted score for the training image; Acquire multi-dimensional scoring data of the training image, and generate a scoring training loss function based on the multi-dimensional scoring data, the predicted score and preset parameters, wherein the preset parameters are parameters specified by the user and represent the degree of user intervention in the scoring; Based on the scoring training loss function, the pre-trained scoring model is trained to obtain a multi-dimensional scoring model.

2. The method according to claim 1, characterized in that The pre-trained scoring model includes a text encoder and an image encoder; The step of inputting the text data and the training image into a pre-trained scoring model to obtain text embedding and image embedding comprises: Inputting the text data into the text encoder to obtain text embedding; The training image is input into the image encoder to obtain image embedding, wherein the image embedding includes an initial image embedding and multiple hidden layer outputs, wherein the initial image embedding is the image embedding output of the image encoder, and the hidden layer output is the output of each hidden layer of the image encoder.

3. The method according to claim 2, characterized in that The text encoder includes a text feature extraction module, a text feature fusion module and a text feature output module; The text feature module is used to extract text features, the text feature fusion module is used to fuse multiple text features, and the text feature output module is used to output text embedding; The image encoder includes an image feature extraction module, an attention mechanism layer, an image feature fusion module and an image feature output module; The image feature extraction module is used to extract the basic features of the image, the attention mechanism layer is used to extract the color features of the image, the image feature fusion module is used to fuse the basic features and the color features, and the image feature output module is used to output the initial image embedding; The image feature extraction module, the attention mechanism layer and the image feature fusion module each include multiple hidden layers.

4. The method according to claim 3, characterized in that The calculation formula of the attention mechanism layer for extracting the color features of the image is as follows: Wherein, T represents the attention of color features, X represents the color features of the image input into the attention mechanism layer, and μ(X) represents the average value of the color features input into the attention mechanism layer.

5. The method according to claim 2, characterized in that The obtaining a final image embedding based on the text embedding and the image embedding comprises: Performing a similarity operation on the text embedding and the initial image embedding to obtain text-image similarity; A final image embedding is generated based on the text-image similarity and a plurality of the hidden layer outputs.

6. The method according to claim 1, characterized in that The obtaining, based on the final image embedding, a predicted score of the training image, comprises: The final image is embedded into a multi-level evaluation network, and the predicted score of the training image is obtained through each hidden layer in the multi-level evaluation network.

7. The method according to claim 1, characterized in that The calculation formula of the scoring training loss function is as follows: Among them, α and β are two preset parameters used to indicate the degree of user intervention in the scoring of the training images, and p i represents the predicted score, y i Represents multi-dimensional rating data.

8. A multi-dimensional scoring model training system, characterized in that: include: The first processing module is used to input text data into the text-generated graph model to obtain a training image; A second processing module: used for inputting the text data and the training image into a pre-trained scoring model to obtain text embedding and image embedding; A third processing module: used for obtaining a final image embedding based on the text embedding and the image embedding; A fourth processing module: configured to obtain a predicted score of the training image based on the final image embedding; A fifth processing module: used to obtain multi-dimensional scoring data of the training image, and generate a scoring training loss function based on the multi-dimensional scoring data, the predicted score and preset parameters, wherein the preset parameters are parameters specified by the user, indicating the degree of user intervention in the scoring; The sixth processing module is used to train the pre-trained scoring model based on the scoring training loss function to obtain a multi-dimensional scoring model.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images