Model training method and device, electronic equipment and computer storage medium

Abstract

Embodiments of the present application provide a model training method and device, electronic equipment and computer storage medium, wherein the model training method comprises: obtaining pre-training sample data, the pre-training sample data comprising multi-modal data; pre-training an encoder in a neural network model using the pre-training sample data to obtain a pre-trained encoder; obtaining a feature representation output by the pre-trained encoder after processing the pre-training sample data, and a pre-training reference sample corresponding to the feature representation; and pre-training a decoder in the neural network model using the feature representation and the pre-training reference sample. The encoder and the decoder are trained in stages, thereby improving the training effect and efficiency of the model.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of artificial intelligence, and in particular to a model training method and device, electronic equipment and computer storage medium. BACKGROUND

[0002] With the development of technology, the way of information dissemination on the Internet is more inclined to images, videos and other more intuitive ways. Multimodal data is increasingly applied to various fields of information dissemination and storage. Data can be divided into text, images, videos, speech, etc. according to the carrier type, and the so-called multimodal data is data of multiple carrier types. Although multimodal data is more intuitive, in some scenarios, many information needs to be manually configured by users according to multimodal data in the form of text. For example, in the e-commerce scenario, the selling points of goods, common questions, etc. need to be filled in manually according to the images, videos or text descriptions of the goods; for example, in the live streaming scenario, the live streaming theme, keyword description, etc. need to be filled in manually by users according to the content of the images or videos, which consumes a lot of manual time cost. Taking the selling points of goods as an example, the images, text descriptions, etc. of the goods usually reveal the selling points of the goods, and only need to intelligently identify and extract the images and text to obtain the selling points of the goods. If a neural network model is used to intelligently fill in, manual annotation is required during model training. Since the selling points of the goods are extracted from the multimodal data such as images and text descriptions, the images and text descriptions need to be annotated respectively, resulting in high cost of manual annotation. Therefore, for the neural network model processing multimodal data, the model training efficiency is low and the model training effect is poor. SUMMARY

[0003] Therefore, embodiments of the present application provide a model training method and device, electronic equipment and computer storage medium to solve some or all of the above problems.

[0004] According to a first aspect of embodiments of the present application, a model training method is provided, comprising: obtaining pre-training sample data, the pre-training sample data comprising multimodal data; pre-training an encoder in a neural network model using the pre-training sample data to obtain a pre-trained encoder; obtaining a feature representation output by the pre-trained encoder after processing the pre-training sample data, and a pre-training reference sample corresponding to the feature representation; and pre-training a decoder in the neural network model using the feature representation and the pre-training reference sample.

[0005] According to a second aspect of the embodiment of the present application, a model training apparatus is provided, comprising: a sample module configured to obtain pre-training sample data, the pre-training sample data comprising multi-modal data; an encoder module configured to pre-train an encoder in a neural network model using the pre-training sample data, to obtain a pre-trained encoder; a feature representation module configured to obtain a feature representation output by the pre-trained encoder after processing the pre-training sample data, and a pre-training reference sample corresponding to the feature representation; and a decoder module configured to pre-train a decoder in the neural network model using the feature representation and the pre-training reference sample.

[0006] According to a third aspect of the embodiment of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, the processor, the memory, and the communication interface being in communication with each other through the communication bus; the memory being configured to store at least one executable instruction, the executable instruction causing the processor to perform operations corresponding to the model training method of the first aspect.

[0007] According to a fourth aspect of the embodiment of the present application, a computer storage medium is provided, having a computer program stored thereon, the program being executed by a processor to implement the model training method of the first aspect.

[0008] The model training method, apparatus, electronic device, and computer storage medium provided by the embodiment of the present application obtain pre-training sample data, the pre-training sample data comprising multi-modal data; pre-train an encoder in a neural network model using the pre-training sample data, to obtain a pre-trained encoder; obtain a feature representation output by the pre-trained encoder after processing the pre-training sample data, and a pre-training reference sample corresponding to the feature representation; and pre-train a decoder in the neural network model using the feature representation and the pre-training reference sample. The encoder is pre-trained first, and then the feature representation output by the pre-trained encoder is used to train the decoder. The encoder and the decoder are trained in stages, which can make the model training converge more easily, improve the model training effect, and reduce the labor cost by not requiring manual labeling, thereby improving the training efficiency of the model. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0010] Figure 1 A scene schematic diagram of a model training method provided by the embodiment of the present application;

[0011] Figure 2 A flowchart of a model training method provided for Embodiment One of the present application is shown in

[0012] Figure 3 An encoder effect schematic diagram provided for Embodiment One of the present application is shown in

[0013] Figure 4 A decoder effect schematic diagram provided for Embodiment One of the present application is shown in

[0014] Figure 5 An application scenario schematic diagram of a neural network model provided for Embodiment One of the present application is shown in

[0015] Figure 6 A structural block diagram of a model training apparatus provided for Embodiment Two of the present application is shown in

[0016] Figure 7 A structural schematic diagram of an electronic device provided for Embodiment Three of the present application is shown in DETAILED DESCRIPTION

[0017] In order to make the personnel in the art better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art should belong to the scope of protection of the embodiments of the present application.

[0018] The specific implementation of the embodiments of the present application will be further described below in conjunction with the drawings of the embodiments of the present application.

[0019] Embodiment One

[0020] Embodiment One of the present application provides a model training method. In order to facilitate understanding, the application scenario of the model training method is described, as shown in Figure 1 Figure 1 A scenario schematic diagram of a model training method provided for Embodiment One of the present application is shown in Figure 1 The scenario shown includes a model training apparatus 101. The model training apparatus 101 can be an apparatus for executing the model training method provided in Embodiment One of the present application.

[0021] The model training apparatus 101 can be a terminal device such as a notebook computer or a desktop computer, or the model training apparatus 101 can be a server, etc. As Figure 1 ​As shown, the model training apparatus 101 can obtain pre-training sample data including multi-modal data, pre-train an encoder in the neural network model using the pre-training sample data, and then pre-train a decoder in the neural network model using feature representations output by the pre-trained encoder and corresponding pre-training reference samples.

[0022] In combination Figure 1 As shown in the scenario, the model training method provided by Embodiment One of the present application is described in detail. It needs to be noted that, Figure 1 is only one application scenario of the model training method provided by Embodiment One of the present application, and does not mean that the model training method must be applied to Figure 1 As shown in the scenario, refer to Figure 2 As shown, Figure 2 a flowchart of a model training method provided by Embodiment One of the present application. The method comprises the following steps:

[0023] Step 201, obtaining pre-training sample data.

[0024] The pre-training sample data includes multi-modal data. In the present application, the multi-modal data can include data of multiple carrier types or the multi-modal data can include multi-modal data, for example, the multi-modal data can include at least two types of data among text, image, video, and voice.

[0025] Step 202, pre-training an encoder in the neural network model using the pre-training sample data to obtain a pre-trained encoder.

[0026] The neural network model includes an encoder and a decoder, and the encoder can be a denoising autoencoder. In step 202, the encoder is pre-trained. The encoder can be used to extract features of input data and output feature representations. In the present application, the data output by the encoder is defined as feature representations. It needs to be noted that the pre-training sample data can be multiple. The multiple pre-training sample data is input into the encoder, a loss function value is calculated according to the output feature representations, the parameters of the encoder are adjusted according to the loss function value, and the process is repeated until the loss function value is less than or equal to a preset function value.

[0027] Optionally, in an embodiment of the present application, the encoder in the neural network model is pre-trained using pre-training sample data to obtain a pre-trained encoder, comprising: adding noise to the data of at least one modality in the pre-training sample data to obtain pre-training sample data containing noise; and inputting the pre-training sample data containing noise into the encoder to pre-train the encoder. Adding noise to the pre-training sample data can be to cover part of the data. Further, adding noise to the data of at least one modality in the pre-training sample data to obtain pre-training sample data containing noise comprises: covering the data of at least one modality in the pre-training sample data to obtain pre-training sample data containing noise.

[0028] As shown in Figure 3 , Figure 3 An encoder effect diagram provided for Embodiment One of the present application, Figure 3 In the embodiment, the pre-training sample data includes text data and image data, the text data is divided into six parts (i.e. six phrases), which are represented by X1-X6 respectively, and the image data is divided into four parts, which are represented by Y1-Y4 respectively. Noise can be added to one type of data or to both types of data. Figure 3 In the embodiment, taking adding noise to the text data as an example, a phrase-based masked language model (PMLM) can be used to introduce a phrase structure tree to extract phrases in the pre-training sample data, and the phrases are covered with X2 and X3 being covered, and then X1, X4, X5, X6, and Y1-Y4 are input into the encoder to train the encoder. Because part of the data is covered, the semantics of the pre-training sample data becomes imperfect, and the image data can be used to predict the missing text data, which can improve the learning ability of the encoder.

[0029] Step 203, obtaining the feature representation output by the pre-trained encoder after processing the pre-training sample data, and the pre-training reference sample corresponding to the feature representation.

[0030] It should be noted that the decoder is used to analyze the feature representation output by the encoder to obtain the output of the decoder, which can include text content. The pre-training reference sample is the output of the decoder, which can also be said to be the expected data. The pre-training reference sample corresponding to the feature representation is the expected output of the decoder after inputting the feature representation into the decoder.

[0031] Step 204, pre-training the decoder in the neural network model using the feature representation and the pre-training reference sample.

[0032] It should be noted that the pre-training sample data for pre-training the encoder and the decoder can be the same or different. Alternatively, in an embodiment of the present application, the decoder in the neural network model is pre-trained using the feature representation and the pre-training reference sample, comprising: adding noise to the pre-training reference sample to obtain a pre-training reference sample containing noise; and inputting the pre-training reference sample containing noise and the corresponding feature representation into the decoder to pre-train the decoder. Adding noise to the pre-training reference sample can improve the learning ability of the decoder and enhance the effect of the decoder. Further, the pre-training reference sample containing noise and the corresponding feature representation are input into the decoder to pre-train the decoder, comprising: inputting the feature representation into the decoder to obtain the output of the corresponding decoder, comparing the output of the decoder with the pre-training reference sample, and adjusting the parameters of the decoder according to the comparison result on the premise that the parameters of the pre-trained encoder are fixed. Because the encoder has been pre-trained, the parameters of the encoder are fixed, and only the parameters of the decoder are adjusted. Moreover, the model convergence can be accelerated and the consistency between the pre-training and fine-tuning stages can be ensured.

[0033] As shown in Figure 4 , Figure 4 a decoder effect diagram for the first embodiment of the present application is provided, Figure 4 In the decoder effect diagram, the pre-training sample data is input into the encoder to output the feature representation, the feature representation is divided into six phrases, which are represented by Z1-Z6 respectively, and correspondingly, the reference data is also divided into six phrases, which are represented by C1-C6 respectively. C2 and C3 can be covered, Z1-Z6 are input into the decoder, C2 and C3 are predicted, and the output of the decoder is obtained to adjust the parameters of the decoder.

[0034] It should be noted that, alternatively, a Masked Region Classification with Linguistic Clues (MRC) model can be used to add noise to the pre-training reference sample. Two implementation methods are listed here to illustrate how to add noise to the pre-training reference sample.

[0035] Alternatively, in the first implementation method, the pre-training reference sample is processed by adding noise to obtain a pre-training reference sample containing noise, comprising: dividing the pre-training reference sample into at least two phrases, rearranging the at least two phrases, and obtaining the pre-training reference sample containing noise.

[0036] Alternatively, in the second implementation method, the pre-training reference sample is processed by adding noise to obtain a pre-training reference sample containing noise, comprising: performing phrase deletion or phrase covering on the pre-training reference sample, and obtaining the pre-training reference sample containing noise.

[0037] Optionally, after steps 201-204, the decoder can also be further trained according to different decoding tasks to achieve different decoding tasks. For example, after pre-training the decoder in the neural network model with the feature representation and the pre-training reference sample, the method further includes: obtaining model training sample data, inputting the model training sample data into the pre-trained encoder to obtain the feature representation; inputting the feature representation and the model reference sample corresponding to the decoding task into the pre-trained decoder, and training the pre-trained decoder. After further training the decoder according to the specific decoding task, the decoder can solve different problems, for example, it can automatically generate product selling points, push search results, push question and answer information, etc. Here, three specific examples are listed for illustration:

[0038] Optionally, in the first example, the neural network model is used to generate product selling points, the model training sample data includes multi-modal data of a product, and the model reference sample includes selling point data of the product; the pre-trained decoder is trained by inputting the feature representation and the model reference sample corresponding to the decoding task into the pre-trained decoder. The multi-modal data of the product is input into the encoder to obtain the feature representation of the corresponding product, and the feature representation of the product and the selling point data of the corresponding product are input into the decoder to train the decoder. After training the decoder, the trained neural network model can be obtained, and the selling point data of the product can be automatically generated by inputting the multi-modal data of the product into the neural network model.

[0039] Optionally, in the second example, the neural network model is used for intelligent search, the model training sample data includes multi-modal data for search, and the model reference sample includes search results; the pre-trained decoder is trained by inputting the feature representation and the model reference sample corresponding to the decoding task into the pre-trained decoder. The multi-modal data for search is input into the encoder to obtain the corresponding feature representation, and the feature representation and the corresponding search results are input into the decoder to train the decoder. After training the decoder, the trained neural network model can be obtained, and the search results can be automatically obtained by inputting the multi-modal data for search into the neural network model.

[0040] Optionally, in a third example, the neural network model is used for intelligent question answering, the model training sample data includes multi-modal data for asking questions, and the model reference sample includes question and answer data; the pre-trained decoder is inputted with the feature representation and the model reference sample corresponding to the decoding task, and the pre-trained decoder is trained, including: inputting the feature representation and the corresponding question and answer data into the pre-trained decoder, and training the decoder. The multi-modal data for question and answer is inputted into the encoder, and the corresponding feature representation can be obtained. The feature representation and the corresponding question and answer data are inputted into the decoder, and the decoder can be trained. After the training of the decoder is completed, the trained neural network model can be obtained. After the multi-modal data for question and answer is inputted into the neural network model, the question and answer data can be obtained, that is, according to the multi-modal data inputted by the user, the related questions and answers can be automatically pushed.

[0041] Based on the above three examples, after the pre-training of the encoder and the decoder is completed, the decoder can be further trained according to the decoding task. After the training is completed, the trained neural network model can be used for different decoding tasks. Here, one specific application scenario is listed for illustration. As shown in the following figure, Figure 5 It should be noted that the terminal device 501 can access the network and be connected to the cloud 502 through the network. In this application, the network includes a local area network (English: Local Area Network, LAN), a wide area network (English: Wide Area Network, WAN), a mobile communication network; such as the World Wide Web (English: World Wide Web, WWW), a Long Term Evolution (English: Long Term Evolution, LTE) network, a 2G network (English: 2th Generation Mobile Network), a 3G network (English: 3th Generation Mobile Network), a 5G network (English: 5th Generation Mobile Network), etc. Of course, this is only an example and does not limit the application. The cloud 502 can include Figure 1 As shown in the following figure, the model training device 101 can be a server, a relay device, a device-to-device (English: Device-to-Device, D2D) device, etc.

[0042] The user 503 inputs multi-modal data on the terminal device 501, the terminal device 501 transmits the multi-modal data to the cloud 502, the cloud 502 processes the multi-modal data by using the trained neural network model, specifically, the multi-modal data is feature-extracted by using an encoder to obtain corresponding feature representation, and then the feature representation is analyzed by using a decoder to obtain the output of the corresponding model, the cloud 502 returns the output of the model to the terminal device 501, and the user can view the output of the model on the terminal device 501. For example, the user inputs a picture and text of a commodity on the terminal device 501, and the terminal device 501 can show the selling points of the commodity to the user by interacting with the cloud 502; for another example, the user inputs multi-modal data for searching on the terminal device 501, and the terminal device 501 can show the search results to the user by interacting with the cloud 502; for another example, the user inputs multi-modal data for question and answer on the terminal device 501, and the terminal device 501 can show the question and answer data to the user by interacting with the cloud 502, that is, automatically push the question and answer. Of course, the above is only an example.

[0043] The model training method provided in the embodiment of the application obtains pre-training sample data, the pre-training sample data includes multi-modal data; the encoder in the neural network model is pre-trained by using the pre-training sample data, and a pre-trained encoder is obtained; the feature representation output by the pre-trained encoder after processing the pre-training sample data and the pre-training reference sample corresponding to the feature representation are obtained; and the decoder in the neural network model is pre-trained by using the feature representation and the pre-training reference sample. The encoder is pre-trained first, and then the feature representation output by the pre-trained encoder is used to train the decoder. The encoder and the decoder are trained in stages, which can make the model training converge more easily, improve the model training effect, and reduce the labor cost by not requiring manual labeling, thereby improving the training efficiency of the model.

[0044] Embodiment two

[0045] Based on the method described in the above embodiment one, the embodiment two of the application provides a model training device for executing the method described in the above embodiment one, referring to the model training device 60 shown in the above embodiment one, the model training device 60 includes: Figure 6

[0046] The sample module 601 is configured to obtain pre-training sample data, and the pre-training sample data includes multi-modal data.

[0047] The encoder module 602 is configured to pre-train an encoder in a neural network model by using pre-training sample data, and obtain a pre-trained encoder.

[0048] ​The feature representation module 603 is configured to acquire the feature representation output by the pre-trained encoder after processing the pre-training sample data, and a pre-training reference sample corresponding to the feature representation.

[0049] The decoder module 604 is configured to pre-train a decoder in the neural network model by using the feature representation and the pre-training reference sample.

[0050] Optionally, in an embodiment of the present application, the encoder module 602 is configured to perform noise addition processing on the data of at least one modality in the pre-training sample data to obtain pre-training sample data containing noise; and input the pre-training sample data containing noise into the encoder to pre-train the encoder and obtain the pre-trained encoder.

[0051] Optionally, in an embodiment of the present application, the encoder module 602 is configured to perform covering processing on the data of at least one modality in the pre-training sample data to obtain pre-training sample data containing noise.

[0052] Optionally, in an embodiment of the present application, the decoder module 604 is configured to perform noise addition processing on the pre-training reference sample to obtain pre-training reference sample containing noise; and input the pre-training reference sample containing noise and the corresponding feature representation into the decoder to pre-train the decoder.

[0053] Optionally, in an embodiment of the present application, the decoder module 604 is configured to input the feature representation into the decoder to obtain the output of the corresponding decoder, compare the output of the decoder with the pre-training reference sample, and adjust the parameters of the decoder on the premise that the parameters of the pre-trained encoder are fixed according to the comparison result.

[0054] Optionally, in an embodiment of the present application, the decoder module 604 is configured to split the pre-training reference sample into at least two phrases, shuffle the order of the at least two phrases, and obtain the pre-training reference sample containing noise.

[0055] Optionally, in an embodiment of the present application, the decoder module 604 is configured to perform phrase deletion or phrase covering on the pre-training reference sample, and obtain the pre-training reference sample containing noise.

[0056] Optionally, in an embodiment of the present application, as shown in Figure 6 The model training apparatus 60 further includes a training module 605 configured to acquire multi-modal data, input the multi-modal data into the pre-trained encoder to obtain a feature representation, input the feature representation and a model reference sample corresponding to a decoding task into the pre-trained decoder, and train the pre-trained decoder.

[0057] Optionally, in an embodiment of the present application, the multi-modal data comprises multi-modal data of a commodity, and the model reference sample comprises selling point data of the commodity; the training module 605 is configured to input the feature representation and the corresponding selling point data of the commodity into the pre-trained decoder, and train the decoder.

[0058] Optionally, in an embodiment of the present application, the multi-modal data comprises multi-modal data for searching, and the model reference sample comprises search results; the training module 605 is configured to input the feature representation and the corresponding search results into the pre-trained decoder, and train the decoder.

[0059] Optionally, in an embodiment of the present application, the multi-modal data comprises multi-modal data for asking questions, and the model reference sample comprises question and answer data; the training module 605 is configured to input the feature representation and the corresponding question and answer data into the pre-trained decoder, and train the decoder.

[0060] The model training apparatus provided in the embodiments of the present application acquires pre-training sample data, the pre-training sample data comprising multi-modal data; pre-trains an encoder in a neural network model by using the pre-training sample data, to obtain a pre-trained encoder; acquires a feature representation output by the pre-trained encoder after processing the pre-training sample data, and a pre-training reference sample corresponding to the feature representation; and pre-trains a decoder in the neural network model by using the feature representation and the pre-training reference sample. The encoder is pre-trained first, and then the decoder is trained by using the feature representation output by the pre-trained encoder. The encoder and the decoder are trained in stages, which can make the model training converge more easily, improve the model training effect, and reduce the labor cost by not requiring manual labeling, thereby improving the model training efficiency.

[0061] Embodiment Three

[0062] Based on the method described in Embodiment One, Embodiment Three of the present application provides an electronic device for executing the method described in Embodiment One, which refers to Figure 7 , and Figure 7 is a structural schematic diagram of an electronic device provided in Embodiment Five of the present application. The specific implementation of the electronic device is not limited in the embodiments of the present application.

[0063] As Figure 7 shown, the electronic device can include a processor 702, a communications interface 704, a memory 706, and a communications bus 708.

[0064] Among them:

[0065] The processor 702, the communication interface 704, and the memory 706 communicate with each other through a communication bus 708.

[0066] The communication interface 704 is configured to communicate with other electronic devices, such as terminal devices or servers.

[0067] The processor 702 is configured to execute the program 710, and specifically can execute the related steps in the above method embodiments.

[0068] Specifically, the program 710 can include program code including computer operation instructions.

[0069] The processor 702 can be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the electronic device can be processors of the same type, such as one or more CPUs; or can be processors of different types, such as one or more CPUs and one or more ASICs.

[0070] The memory 706 is configured to store the program 710. The memory 706 can include a high-speed RAM memory, and can also include a non-volatile memory such as a disk memory.

[0071] The program 710 can be specifically used to cause the processor 702 to execute any method in the foregoing embodiment one.

[0072] The specific implementation of each step in the program 710 can refer to the corresponding description in the corresponding steps and units of the model training method embodiments, which will not be described here. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the devices and modules described above can refer to the corresponding process description in the foregoing method embodiments, which will not be described here.

[0073] The electronic device provided in the embodiment of the present application acquires pre-training sample data, the pre-training sample data comprising multi-modal data; pre-trains an encoder in a neural network model using the pre-training sample data to obtain a pre-trained encoder; acquires feature representation output by the pre-trained encoder after processing the pre-training sample data, and a pre-training reference sample corresponding to the feature representation; and pre-trains a decoder in the neural network model using the feature representation and the pre-training reference sample. The encoder is pre-trained first, and then the decoder is trained using the feature representation output by the pre-trained encoder. The encoder and the decoder are trained in stages, which can make the model training converge more easily, improve the model training effect, and does not require manual labeling, reduces the labor cost, and improves the model training efficiency.

[0074] Embodiment four

[0075] Based on the method described in the above embodiment one, the fourth embodiment of the present application provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the embodiment one.

[0076] The computer storage medium provided in the embodiment of the present application acquires pre-training sample data, the pre-training sample data comprising multi-modal data; pre-trains an encoder in a neural network model using the pre-training sample data to obtain a pre-trained encoder; acquires feature representation output by the pre-trained encoder after processing the pre-training sample data, and a pre-training reference sample corresponding to the feature representation; and pre-trains a decoder in the neural network model using the feature representation and the pre-training reference sample. The encoder is pre-trained first, and then the decoder is trained using the feature representation output by the pre-trained encoder. The encoder and the decoder are trained in stages, which can make the model training converge more easily, improve the model training effect, and does not require manual labeling, reduces the labor cost, and improves the model training efficiency.

[0077] It should be noted that, according to the needs of implementation, each component / step described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or part of the operations of the components / steps can be combined into a new component / step, to achieve the purpose of the embodiments of the present application.

[0078] The above-described methods according to embodiments of the present application can be implemented in hardware, firmware, or software, or any combination thereof, and can be stored in a recording medium such as CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk, or be downloaded by network from a remote recording medium or non-transitory machine-readable medium originally stored in a remote recording medium or non-transitory machine-readable medium and to be stored in a local recording medium, so that such software implementing the methods described herein can be stored in a recording medium on a general-use computer, a special-purpose processor, or programmable or special-purpose hardware (such as ASIC or FPGA). It can be understood that the computer, processor, microprocessor controller, or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, processor, or hardware, the model training methods described herein are implemented. Furthermore, when a general-purpose computer accesses the code for implementing the model training methods shown herein, the execution of the code will convert the general-purpose computer into a special-purpose computer for executing the model training methods shown herein.

[0079] Those skilled in the art can realize that the units and method steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the present application.

[0080] The above embodiments are only used to illustrate but not limit the embodiments of the present application, and a person of ordinary skill in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application, therefore all equivalent technical solutions belong to the scope of the embodiments of the present application, and the patent protection scope of the embodiments of the present application should be defined by the claims.

Claims

1. A model training method comprising: obtaining pre-training sample data, the pre-training sample data comprising multi-modal data, wherein the multi-modal data comprises at least two of the following: text data, image data, video data, and voice data; pre-training an encoder in a neural network model using the pre-training sample data to obtain a pre-trained encoder; obtaining a feature representation output by the pre-trained encoder after processing the pre-training sample data, and a pre-training reference sample corresponding to the feature representation, wherein the pre-training reference sample is used to represent an expected output of a decoder after the feature representation is input into the decoder for analysis; pre-training the decoder using the feature representation and the pre-training reference sample; obtaining model training sample data, inputting the model training sample data into the pre-trained encoder to obtain a feature representation, inputting the feature representation and a model reference sample corresponding to a decoding task into the pre-trained decoder, training the pre-trained decoder to obtain a trained neural network model; inputting the multi-modal data into the trained neural network model to generate selling point data of a product and / or obtain search results and / or obtain question and answer data.

2. The method of claim 1, wherein, The pre-training of the encoder in the neural network model using the pre-training sample data to obtain a pre-trained encoder comprises: performing noise addition processing on at least one modality of data in the pre-training sample data to obtain pre-training sample data containing noise; inputting the pre-training sample data containing noise into the encoder to pre-train the encoder and obtain a pre-trained encoder.

3. The method of claim 2, wherein, The noise addition processing on at least one modality of data in the pre-training sample data to obtain pre-training sample data containing noise comprises: performing covering processing on at least one modality of data in the pre-training sample data to obtain the pre-training sample data containing noise.

4. The method of claim 1, wherein, The pre-training of the decoder in the neural network model using the feature representation and the pre-training reference sample comprises: performing noise addition processing on the pre-training reference sample to obtain pre-training reference sample containing noise; inputting the pre-training reference sample containing noise and the corresponding feature representation into the decoder to pre-train the decoder.

5. The method of claim 4, wherein, The inputting of the pre-training reference sample containing noise and the corresponding feature representation into the decoder to pre-train the decoder comprises: inputting the feature representation into the decoder to obtain an output of the corresponding decoder, comparing the output of the decoder with the pre-training reference sample, and adjusting parameters of the decoder on the premise that parameters of the pre-trained encoder are fixed according to a comparison result.

6. The method of claim 4, wherein, The noise addition processing on the pre-training reference sample to obtain pre-training reference sample containing noise comprises: segmenting the pre-training reference sample into at least two phrases, rearranging the at least two phrases, and obtaining the pre-training reference sample containing noise.

7. The method of claim 4, wherein, The adding noise processing of the pre-training reference sample obtains a pre-training reference sample containing noise, including: The pre-training reference sample is subjected to phrase deletion or phrase covering, and the pre-training reference sample containing noise is obtained.

8. The method of claim 1, wherein, The multi-model training sample data includes multi-modal data of a commodity, and the model reference sample includes selling point data of the commodity; The model reference sample corresponding to the decoding task is input into the pre-trained decoder, and the pre-trained decoder is trained, including: The feature representation and the corresponding selling point data of the commodity are input into the pre-trained decoder, and the decoder is trained.

9. The method of claim 1, wherein, The model training sample data includes multi-modal data for searching, and the model reference sample includes search results; The model reference sample corresponding to the decoding task is input into the pre-trained decoder, and the pre-trained decoder is trained, including: The feature representation and the corresponding search results are input into the pre-trained decoder, and the decoder is trained.

10. The method of claim 1, wherein, The model training sample data includes multi-modal data for asking questions, and the model reference sample includes question and answer data; The model reference sample corresponding to the decoding task is input into the pre-trained decoder, and the pre-trained decoder is trained, including: The feature representation and the corresponding question and answer data are input into the pre-trained decoder, and the decoder is trained.

11. A model training apparatus, comprising: a sample module configured to obtain pre-training sample data, the pre-training sample data comprising multi-modal data, wherein the multi-modal data at least includes two of the following: text data, image data, video data, and voice data; an encoder module configured to pre-train an encoder in a neural network model using the pre-training sample data, to obtain a pre-trained encoder; a feature representation module configured to obtain a feature representation output by the pre-trained encoder after processing the pre-training sample data, and a pre-training reference sample corresponding to the feature representation, wherein the pre-training reference sample is used to represent an expected output of a decoder after the feature representation is input into the decoder for analysis; a decoder module configured to pre-train a decoder in the neural network model using the feature representation and the pre-training reference sample; The apparatus is further configured to obtain model training sample data, input the model training sample data into the pre-trained encoder to obtain a feature representation, input the feature representation and a model reference sample corresponding to a decoding task into the pre-trained decoder to train the pre-trained decoder, and obtain a trained neural network model. The apparatus is further configured to input the multi-modal data into the trained neural network model to generate selling point data of a commodity and / or obtain search results and / or obtain question and answer data.

12. An electronic device comprising: a processor, a memory, a communication interface, and a communication bus, which complete communication with each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction makes the processor execute the operation corresponding to the model training method in any one of claims 1-10.

13. A computer storage medium, having stored thereon a computer program, which is executed by a processor to implement the model training method in any one of claims 1-10.

Images