Model training method and device, question and answer information processing method and device, equipment and medium

Abstract

The invention provides a model training method, relates to the technical field of artificial intelligence, in particular to the technical fields of multi-modal embedding, large models, retrieval enhancement generation, knowledge bases and the like, and can be applied to an intelligent question and answer processing scene. According to the specific implementation scheme, modal samples of different modals of a plurality of cross-modal sample pairs are input into a plurality of coding networks used for different modals in a to-be-trained embedding model, a plurality of modal sample embedding vectors are obtained, and each cross-modal sample pair comprises two modal samples of different modals; according to the multiple modal sample embedding vectors, determining at least one correlation feature; determining at least one piece of correlation loss information according to the at least one tag feature and the at least one correlation feature; and training the to-be-trained embedding model according to the multiple pieces of correlation loss information. The invention further provides a question and answer information processing method and device, electronic equipment and a storage medium.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to multimodal embedding, large models, retrieval augmented generation (RAG), and knowledge bases, and is applicable to intelligent question-answering scenarios. More specifically, this disclosure provides a model training method, a question-answering information processing method, apparatus, electronic device, and storage medium. Background Art

[0002] With the development of artificial intelligence technology, the scenarios for question answering based on multimodal data are increasing. Multimodal data can include data in various modalities such as images, text, and audio. Summary of the Invention

[0003] The present disclosure provides a model training method, a question-and-answer information processing method, an apparatus, a device, and a storage medium.

[0004] According to one aspect of the present disclosure, a model training method is provided, which includes: inputting modal samples of different modalities of multiple cross-modal sample pairs into multiple encoding networks for different modalities in an embedding model to be trained, respectively, to obtain multiple modal sample embedding vectors, wherein the cross-modal sample pairs include two modal samples of different modalities; determining at least one correlation feature based on the multiple modal sample embedding vectors, wherein the correlation feature includes a correlation sample value between the two modal sample embedding vectors of different modalities; determining at least one correlation loss information based on at least one label feature and at least one correlation feature, wherein the label feature includes a correlation label value between the two modal samples of different modalities; and training the embedding model to be trained based on the multiple correlation loss information.

[0005] According to another aspect of the present disclosure, a question-answering information processing method is provided, the method comprising: determining at least one target knowledge data from a plurality of knowledge data as context information for the initial question information based on initial question information; determining answer information for the initial question information based on the context information, wherein the indexes of the plurality of knowledge data are determined based on the embedding vectors of the plurality of knowledge data, the embedding vectors of the knowledge data are obtained by inputting the knowledge data into a target embedding model, and the target embedding model is obtained by training the embedding model to be trained according to the model training method provided by the present disclosure.

[0006] According to another aspect of the present disclosure, a model training device is provided, which includes: a first acquisition module, used to input modal samples of different modalities of multiple cross-modal sample pairs into multiple encoding networks for different modalities in an embedding model to be trained, respectively, to obtain multiple modal sample embedding vectors, wherein the cross-modal sample pairs include two modal samples of different modalities; a first determination module, used to determine at least one correlation feature based on the multiple modal sample embedding vectors, wherein the correlation feature includes a correlation sample value between the two modal sample embedding vectors of different modalities; a second determination module, used to determine at least one correlation loss information based on at least one label feature and at least one correlation feature, wherein the label feature includes a correlation label value between the two modal samples of different modalities; a training module, used to train the embedding model to be trained based on the multiple correlation loss information.

[0007] According to another aspect of the present disclosure, a question-answering information processing device is provided, which includes: a third determination module for determining at least one target knowledge data from a plurality of knowledge data according to initial question information as context information for the initial question information; a fourth determination module for determining answer information for the initial question information according to the context information, wherein the index of each of the plurality of knowledge data is determined according to the embedding vector of each of the plurality of knowledge data, the embedding vector of the knowledge data is obtained by inputting the knowledge data into a target embedding model, and the target embedding model is obtained by training the embedding model to be trained according to the model training device provided by the present disclosure.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided according to the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the method provided according to the present disclosure when executed by a processor.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is a flowchart of a model training method according to one embodiment of the present disclosure;

[0014] Figure 2 is a schematic diagram of an embedding model to be trained according to one embodiment of the present disclosure;

[0015] Figure 3 is a flowchart of a question-and-answer information processing method according to another embodiment of the present disclosure;

[0016] Figure 4 is a schematic diagram of determining indexes of multiple knowledge data according to an embodiment of the present disclosure;

[0017] Figure 5 is a schematic diagram of a question-and-answer information processing method according to an embodiment of the present disclosure;

[0018] Figure 6 is a block diagram of a model training apparatus according to one embodiment of the present disclosure;

[0019] Figure 7 is a block diagram of a question and answer information processing apparatus according to an embodiment of the present disclosure; and

[0020] Figure 8 It is a block diagram of an electronic device to which at least one of a model training method and a question-and-answer information processing method can be applied according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] In a multimodal, multilingual question-answering system based on retrieval-enhanced generation, multiple retrieval models can be used to retrieve passages in different modalities. However, multiple retrieval models significantly increase system complexity and deployment costs. Furthermore, for passages in a document containing data from different modalities, different embedding models must be used to create separate indexes, which increases system complexity and development and deployment costs.

[0023] Different embedding models have difficulty in uniformly processing data of different modalities, especially image data, audio data, and video data.

[0024] Therefore, in order to process data of different modalities at low cost and high efficiency, the present disclosure provides a model training method, which will be described below.

[0025] Figure 1 4 is a flowchart of a model training method according to an embodiment of the present disclosure.

[0026] like Figure 1 As shown, the method 100 may include operations S110 to S140.

[0027] In operation S110, modal samples of different modalities of the plurality of cross-modal sample pairs are respectively input into a plurality of encoding networks for different modalities in the embedding model to be trained to obtain a plurality of modal sample embedding vectors.

[0028] In an embodiment of the present disclosure, a cross-modal sample pair includes two modal samples of different modalities. For example, a cross-modal sample pair may include an image sample and a text sample. A cross-modal sample pair may include a text sample and an audio sample. A cross-modal sample pair may include an image sample and an audio sample.

[0029] In the disclosed embodiments, a modality sample can be input into an encoding network for that modality. The encoding network is used to embed the modality sample. For example, an image sample can be input into an image encoding network to obtain an image sample embedding vector.

[0030] In operation S120, at least one correlation feature is determined based on the multiple modality sample embedding vectors.

[0031] In an embodiment of the present disclosure, a correlation feature includes a correlation sample value between two modal sample embedding vectors of different modalities. For example, a correlation feature may include a correlation sample value between an image sample embedding vector and a text sample embedding vector. The correlation sample value may be determined, for example, based on the product of the modal sample embedding vectors of different modalities.

[0032] In operation S130 , at least one relevance loss information is determined based on the at least one tag feature and the at least one relevance feature.

[0033] In an embodiment of the present disclosure, a label feature includes a correlation label value between two modal samples of different modalities. For example, the label feature may include a correlation label value between an image sample and a text sample. The correlation label value may correspond to a correlation sample value. Correlation loss information may be determined based on the differences between multiple correlation label values ​​and corresponding multiple correlation sample values.

[0034] In operation S140 , the embedding model to be trained is trained according to the plurality of correlation loss information.

[0035] For example, according to multiple correlation loss information, the parameters of the embedding model to be trained are adjusted until the correlation loss information converges to obtain the target embedding model.

[0036] Through the disclosed embodiments, the embedding model to be trained includes encoding networks for different modalities, and the modal samples of different modalities in multiple cross-modal sample pairs are processed by the encoding networks for different modalities respectively, so that the modal sample embedding vectors can be efficiently extracted. Correlation features are determined based on the sample embedding vectors of different modalities, and correlation loss information is determined based on the correlation features, so that multiple encoding networks are uniformly trained using samples of different modalities, and data of different modalities can be mapped to the same vector space, so that the distance between the embedding vectors of modal samples with different modalities but the same semantics is effectively reduced, so that the functions of multiple embedding models can be realized by using one embedding model, which effectively reduces the complexity of the embedding model and reduces the cost of knowledge retrieval.

[0037] It can be understood that the above describes the method of the present disclosure, and the cross-modal sample pairs and the embedding model to be trained will be further described below.

[0038] Figure 2 is a schematic diagram of an embedding model to be trained according to an embodiment of the present disclosure.

[0039] like Figure 2 As shown, the embedding model to be trained can include an image encoding network IE21, a text encoding network TE22, and an audio encoding network AE23. The image encoding network IE21 can be built based on a vision transformer network or a residual network (ResNet). The text encoding network TE22 can be built based on a transformer network. The audio encoding network AE23 can be built based on a convolutional neural network (CNN).

[0040] In some embodiments, the plurality of cross-modal sample pairs include at least two of a first cross-modal sample pair, a second cross-modal sample pair, and a third cross-modal sample pair. The first cross-modal sample pair includes an image sample and a text sample, the second cross-modal sample pair includes a text sample and an audio sample, and the third cross-modal sample pair includes an image sample and an audio sample. The semantic similarity between the two modal samples of different modalities in the cross-modal sample pair is greater than or equal to a preset semantic similarity threshold. The first cross-modal sample pair can be K, the second cross-modal sample pair can be M, and the third cross-modal sample pair can be N. K, N, and M are all non-negative integers and at least two of K, N, and M are greater than or equal to 1.

[0041] For example, taking K=1 as an example, a first cross-modal sample pair may include: image sample i201 and text sample t201. Image sample i201 may be a frontal portrait of a Border Collie. Text sample t201 may be a "Border Collie ID photo." The semantic similarity between the image sample and text sample in this first cross-modal sample pair may be greater than a preset semantic similarity threshold. For another example, taking M=1 as an example, a second cross-modal sample pair may include: text sample t202 "A bright moon rises above the sea" and the corresponding audio sample a202. Audio sample a202 may be an audio recording of the sentence "A bright moon rises above the sea." For another example, taking N=1 as an example, a third cross-modal sample pair may include: image sample i203 and the corresponding audio sample a203. Image sample i203 may be a photo of a farmer working with a hoe under the sun. Audio sample a203 may be an audio recording of the sentence "A farmer wields a hoe in the field under the scorching sun."

[0042] Through the embodiments of the present disclosure, when at least two of the first cross-modal sample pair, the second cross-modal sample pair, and the third cross-modal sample pair are included, multiple modalities such as audio, image, and text can be covered so as to train multiple encoding networks for multiple modalities. The semantics of the two modal samples in the cross-modal sample pair are the same, and they can be used as positive samples, so that the embedding model reduces the distance between the respective embedding vectors of the two modal samples with the same semantics. The semantics between different cross-modal sample pairs can be different, and they can be used as negative samples, so that the distance between the respective embedding vectors of the two modal samples with different semantics is increased.

[0043] In some embodiments, the multiple modal sample embedding vectors include image sample embedding vectors, text sample embedding vectors, and audio sample embedding vectors. In some implementations of the above operation S110, the modal samples of the different modalities of the multiple cross-modal sample pairs are respectively input into the multiple encoding networks for different modalities in the embedding model to be trained, and the multiple modal sample embedding vectors are obtained, including: inputting K+N image samples, M+K text samples, and N+M audio samples into the image encoding network, text encoding network, and audio encoding network of the embedding model to be trained, respectively, to obtain K+N image sample embedding vectors, M+K text sample embedding vectors, and N+M audio sample embedding vectors.

[0044] For example, K+N image samples are input into the image coding network to obtain K+N image sample embedding vectors. Figure 2 As shown, the K+N image samples may include image sample i201 and image sample i203. Image sample i201 and image sample i203 are input into image coding network ie21 to obtain image sample embedding vector iemb201 and image sample embedding vector iemb203.

[0045] Input M+K text samples into the text encoding network to obtain M+K text sample embedding vectors. Figure 2 As shown, the M+K text samples may include text sample t201 and text sample t202. Text sample t201 and text sample t202 are input into the text encoding network te22 to obtain the text sample embedding vector temb201 and text sample embedding vector temb202.

[0046] Input N+M audio samples into the audio coding network to obtain N+M audio sample embedding vectors. Figure 2 As shown, the N+M audio samples include audio sample a202 and audio sample a203. Audio sample a202 and audio sample a203 are input into the audio coding network ae23 to obtain audio sample embedding vectors aemb202 and aemb203.

[0047] I understand. Figure 2 In the example shown, K = M = N = 1. However, the present disclosure is not limited thereto, and at least two of K, N, and M are integers greater than or equal to 1, as will be explained below.

[0048] In another example, taking the case where the multiple cross-modal sample pairs include one first cross-modal sample pair, one second cross-modal sample pair, and do not include the third cross-modal sample pair as an example, K=M=1, N=0. Taking the case where the multiple cross-modal sample pairs include one third cross-modal sample pair, one second cross-modal sample pair, and do not include the first cross-modal sample pair as an example, K=0, M=N=1. Taking the case where the multiple cross-modal sample pairs include one first cross-modal sample pair, one third cross-modal sample pair, and do not include the second cross-modal sample pair as an example, M=0, K=N=1. Thus, when at least two of K, N, and M are greater than or equal to 1, the samples used to train the embedding model can cover the three modalities of audio, image, and text, thereby achieving training of the embedding model at a lower cost.

[0049] It can be understood that the above describes some ways to obtain the modal sample embedding vector, and the following describes some ways to determine the correlation features.

[0050] In some embodiments of operation S220, determining at least one correlation feature based on multiple modality sample embedding vectors includes determining a correlation sample value based on a product result between two modality sample embedding vectors of different modalities and a training temperature parameter. For example, the correlation sample value logit_value can be determined using the following formula:

[0051] (Formula 1)

[0052] It can be the embedding vector of the first modality sample in the two modality sample embedding vectors of different modalities, It can be a transposed second modality sample embedding vector. The transposed second modality embedding vector can be obtained by transposing the second modality sample embedding vector in two modality sample embedding vectors of different modalities. t can be a training temperature parameter, which can adjust the convergence speed of training. In one example, t can be 1 or other preset values. The first modality sample embedding vector and the second modality sample embedding vector can be embedding vectors of any two different modalities among the image sample embedding vector, the text modality sample embedding vector and the audio modality embedding vector. Through the embodiment of the present disclosure, the correlation between the two modality sample embedding vectors of different modalities is determined, and the distance between the embedding vectors of different modality samples can be represented efficiently and at low cost, which helps to improve training efficiency.

[0053] It can be understood that the above describes some ways to determine the correlation sample values ​​in the correlation feature. The following will further illustrate the correlation feature of the present disclosure using multiple modalities including images, text and audio as examples.

[0054] In some embodiments, the at least one correlation feature includes a first correlation feature, a second correlation feature, and a third correlation feature.

[0055] The first correlation feature includes (K+N)×(M+K) first correlation sample values ​​between K+N image sample embedding vectors and M+K text sample embedding vectors. For example, in the case of K=M=1, the first correlation feature It can be implemented as:

[0056] (Formula 2)

[0057] The first correlation sample value can be determined based on the first product result between the text sample embedding vector and the transposed image sample embedding vector and the first training temperature parameter. The transposed image sample embedding vector is obtained by transposing the image sample embedding vector. For example, based on the above formula 1, the text sample embedding vector temb201 is multiplied with the transposed image sample embedding vector iemb201 to obtain the first product result, and the first product result is multiplied with the first training temperature parameter to obtain the first correlation sample value between the image sample embedding vector iemb201 and the text sample embedding vector temb201. It can be understood that the first correlation sample value between the image sample embedding vector iemb201 and the text sample embedding vector temb202 is determined , determine the first correlation sample value between the image sample embedding vector iemb203 and the text sample embedding vector temb201 and determining a first correlation sample value between the image sample embedding vector iemb203 and the text sample embedding vector temb202 In this way, the first correlation sample value between the image sample embedding vector iemb201 and the text sample embedding vector temb201 is determined. The method is the same or similar, and the present disclosure will not repeat it here.

[0058] The second correlation feature includes (M+K)×(N+M) second correlation sample values ​​between M+K text sample embedding vectors and N+M audio sample embedding vectors. For example, in the case of M=N=1, the second correlation feature It can be implemented as:

[0059] (Formula 3)

[0060] The second correlation sample value is determined based on the second product result between the audio sample embedding vector and the transposed text sample embedding vector and the second training temperature parameter. The transposed text sample embedding vector is obtained by transposing the text sample embedding vector. For example, based on the above formula 1, the text sample embedding vector temb202 is multiplied by the audio sample embedding vector aemb202 to obtain a second product result, and the second product result is multiplied by the second training temperature parameter to obtain the second correlation sample value between the text sample embedding vector temb202 and the audio sample embedding vector aemb202. It can be understood that the second correlation sample value between the text sample embedding vector temb202 and the audio sample embedding vector aemb203 is determined , determine the second correlation sample value between the text sample embedding vector temb201 and the audio sample embedding vector aemb202 And determine the second correlation sample value between the text sample embedding vector temb201 and the audio sample embedding vector aemb203 In this way, the second correlation sample value between the text sample embedding vector temb202 and the audio sample embedding vector aemb202 is determined. The method is the same or similar, and the present disclosure will not repeat it here.

[0061] The third correlation feature includes (K+N)×(N+M) third correlation sample values ​​between K+N image sample embedding vectors and N+M audio sample embedding vectors. For example, in the case of M=N=1, the third correlation feature It can be implemented as:

[0062] (Formula 4)

[0063] A third correlation sample value is determined based on a third multiplication result between the image sample embedding vector and the transposed audio sample embedding vector and a third training temperature parameter. The transposed audio sample embedding vector is obtained by transposing the audio sample embedding vector. For example, based on the above formula 1, the image sample embedding vector iemb203 is multiplied by the transposed audio sample embedding vector aemb203 to obtain a third multiplication result, and the third multiplication result is multiplied by the third training temperature parameter to obtain a third correlation sample value between the audio sample embedding vector aemb203 and the image sample embedding vector iemb203. It can be understood that the third correlation sample value between the audio sample embedding vector aemb203 and the image sample embedding vector iemb201 is determined , determine the third correlation sample value between the audio sample embedding vector aemb202 and the image sample embedding vector iemb203 and determining a third correlation sample value between the audio sample embedding vector aemb202 and the image sample embedding vector iemb201 In this way, the third correlation sample value between the audio sample embedding vector temb203 and the image sample embedding vector iemb203 is determined. The method is the same or similar, and the present disclosure will not repeat it here.

[0064] It can be understood that, as mentioned above, K = M = N = 1. Therefore, (K + N) × (M + K) = (M + K) × (N + M) = (K + N) × (N + M) = 2 × 2.

[0065] It can be understood that the above describes the method for determining correlation features, and the following describes the label features.

[0066] In some embodiments, the label feature can be determined based on the label of the cross-modal sample pair. The label of the cross-modal sample pair can indicate that the semantic similarity between the two modal samples of different modalities in the cross-modal sample pair is greater than or equal to a preset semantic similarity threshold. For at least one correlation feature, at least one label feature can be determined based on the label of the cross-modal sample pair. The at least one label includes at least one of a first label feature, a second label feature, and a third label feature.

[0067] In some embodiments, the first label feature includes (K+N)×(M+K) first correlation label values ​​between K+N image samples and M+K text samples. For example, in the case of K=M=1, the first label feature It can be implemented as:

[0068] (Formula 5)

[0069] First label feature It may include a first correlation label value between the text sample t201 and the image sample i201 As mentioned above, the text sample t201 and the image sample i201 are modal samples of two different modalities in the first cross-modal sample pair, and have high semantic similarity. The first correlation label value Can be 1. First tag feature It can also include the first correlation label value between the text sample t202 and the image sample i201 , the first correlation label value between the text sample t201 and the image sample i203 and the first correlation label value between the text sample t202 and the image sample i203 The text sample t201 and the image sample i203 are modality samples in different cross-modal sample pairs, and the first correlation label value is It can be 0. Text sample t202, image sample i201 and image sample i203 are modality samples in different cross-modality sample pairs, and the first correlation label value is and the first relevance label value Both can be 0.

[0070] The second label feature includes (M+K)×(N+M) second correlation label values ​​between M+K text samples and N+M audio samples. For example, in the case of M=N=1, the second label feature It can be implemented as:

[0071] (Formula 6)

[0072] Second label feature The second correlation tag value between the text sample t202 and the audio sample a202 may be included As mentioned above, the text sample t202 and the audio sample a202 are modal samples of two different modalities in the second cross-modal sample pair, and have a high semantic similarity. The second correlation label value Can be 1. Second tag feature It can also include a second correlation label value between the text sample t202 and the audio sample a203 , the second correlation label value between the text sample t201 and the audio sample a202 and the second correlation label value between the text sample t201 and the audio sample i203 The text sample t202 and the audio sample a203 are modality samples in different cross-modal sample pairs, and the second correlation label value is It can be 0. Text sample t201, audio sample a202 and audio sample a203 are modality samples in different cross-modality sample pairs, and the second correlation label value is and the second relevance label value Both can be 0.

[0073] The third label feature includes (K+N)×(N+M) third correlation label values ​​between K+N image samples and N+M audio samples. For example, in the case of K=N=1, the third label feature It can be implemented as:

[0074] (Formula 7)

[0075] Third label feature A third correlation label value may be included between the image sample i 203 and the audio sample a 203 As mentioned above, the image sample t203 and the audio sample a203 are modal samples of two different modalities in the third cross-modal sample pair, and have high semantic similarity. The third correlation label value Can be 1. The third tag feature It can also include a third correlation label value between the image sample i203 and the audio sample a201 , the third correlation label value between image sample i201 and audio sample a202 and the third correlation label value between image sample i201 and audio sample a203 The image sample i201 and the audio sample a203 are modality samples in different cross-modal sample pairs, and the third correlation label value is It can be 0. Audio sample a202, image sample i201 and image sample i203 are modality samples in different cross-modality sample pairs, and the third correlation label value is and the third correlation label value Both can be 0.

[0076] It can be understood that the above describes each value in the tag feature and the correlation feature of the present disclosure, and the following describes the sub-features of different dimensions in the relativity feature and the tag feature.

[0077] In some embodiments, the correlation feature includes at least one first-dimensional correlation sample sub-feature and at least one second-dimensional correlation sample sub-feature, the first-dimensional correlation sample sub-feature includes at least one correlation sample value between the first modality sample embedding vector and at least one second modality sample embedding vector, and the second-dimensional correlation sample sub-feature includes at least one correlation sample value between the second modality sample embedding vector and at least one first modality sample embedding vector.

[0078] For example, the first dimension can be a row dimension, and the second dimension can be a column dimension. For example, along the row direction, the first correlation feature The first dimension correlation sample sub-features can be included [ 11, ] and the first dimension correlation sample sub-features[ 31, ]. The first dimension correlation sample sub-features[ 11, ] includes two first correlation sample values ​​between the image sample embedding vector iemb201 and the text sample embedding vector temb201 and the text sample embedding vector temb202. The first dimension correlation sample sub-feature [ 31, ] includes two first correlation sample values ​​between the image sample embedding vector iemb203 and the text sample embedding vector temb201 and the text sample embedding vector temb202. Along the column direction, the first correlation feature The second dimension correlation sample sub-features can be included [ 11, ] and the second dimension correlation sample sub-features[ 12, ]. Second dimension correlation sample sub-features[ 11, ] includes two first correlation sample values ​​between the text sample embedding vector temb201 and the image sample embedding vector iemb201 and the image sample embedding vector iemb203. The second dimension correlation sample sub-feature [ 12, ] includes two first correlation sample values ​​between the text sample embedding vector temb202 and the image sample embedding vector iemb201 and the image sample embedding vector iemb203.

[0079] It can be understood that for the first correlation feature, the first modality sample embedding vector can be an image sample embedding vector, and the second modality sample embedding vector can be a text sample embedding vector. The description of the first dimension correlation sample sub-feature of the second correlation feature and the first dimension correlation sample sub-feature of the third correlation feature is the same or similar to the description of the first dimension correlation sub-feature of the first correlation feature, and the present disclosure will not repeat it here. The description of the second dimension correlation sample sub-feature of the second correlation feature and the second dimension correlation sample sub-feature of the third correlation feature is the same or similar to the description of the second dimension correlation sub-feature of the first correlation feature, and the present disclosure will not repeat it here.

[0080] In some embodiments, the label feature includes at least one first dimension correlation label sub-feature and at least one second dimension correlation label sub-feature, the first dimension correlation label sub-feature includes at least one correlation label value between a first modality sample and at least one second modality sample, and the second dimension correlation label sub-feature includes at least one correlation label value between a second modality sample and at least one first modality sample.

[0081] For example, the first dimension can be a row dimension, and the second dimension can be a column dimension. For example, along the row direction, the first correlation label feature Can include first dimension relevance label sub-features [ 11, ] and the first dimension correlation label sub-feature[ 31, ]. The first dimension relevance label sub-feature[ 11, ] includes two first correlation label values ​​between image sample i201 and text sample t201 and text sample t202. The first dimension correlation label sub-feature [ 31, ] includes two first correlation label values ​​between image sample i203 and text sample t201 and text sample t202. Along the column direction, the first correlation label feature A second dimension relevance label sub-feature can be included [ 11, ] and the second dimension correlation label sub-feature[ 12, ]. The second dimension correlation label sub-feature[ 11, ] includes two first correlation label values ​​between the text sample t201 and the image sample i201 and the image sample embedding vector i203. The second dimension correlation label sub-feature [ 12, ] includes two first correlation label values ​​between the text sample t202 and the image samples i201 and i203.

[0082] It can be understood that, for the first label feature, the first modality sample can be an image sample, and the second modality sample can be a text sample. The description of the first dimension correlation label sub-feature of the second label feature and the first dimension correlation label sub-feature of the third label feature is the same or similar to the description of the first dimension correlation label sub-feature of the first label feature, and the present disclosure will not repeat them here. The description of the second dimension correlation label sub-feature of the second label feature and the second dimension correlation label sub-feature of the third label feature is the same or similar to the description of the second dimension correlation label sub-feature of the first label feature, and the present disclosure will not repeat them here.

[0083] It can be understood that the above describes the correlation features and different dimensional sub-features of the label features of the present disclosure, and the following describes the method for determining the correlation loss information of the present disclosure.

[0084] In some embodiments, in some implementations of the above operation S230, determining at least one correlation loss information based on at least one label feature and at least one correlation feature includes: determining multiple first-dimensional correlation loss information based on at least one first-dimensional correlation label sub-feature of each of the multiple label features and at least one first-dimensional correlation sample sub-feature of each of the multiple correlation features. For example, taking the first label feature shown in Formula 5 and the first correlation feature shown in Formula 2 as an example, based on the first-dimensional correlation label sub-feature [ 11, ] and the first dimension correlation sample sub-features[ 11, ], a first function result can be determined based on the cross-entropy loss function as first-dimensional correlation loss information. It is understood that normalization can be performed during the determination of the function result. For another example, for the multiple correlation features and label features shown in Formulas 2 to 7 above, three first-dimensional correlation loss information can be determined.

[0085] In some embodiments, in some implementations of the above operation S230, determining at least one correlation loss information based on at least one label feature and at least one correlation feature further includes: determining multiple second-dimensional correlation loss information based on at least one second-dimensional correlation label sub-feature of each of the multiple label features and at least one second-dimensional correlation sample sub-feature of each of the multiple correlation features. For example, taking the first label feature shown in Formula 5 and the first correlation feature shown in Formula 2 as an example, based on the second-dimensional correlation label sub-feature [ 11, ] and the second dimension correlation sample sub-features[ 11, ], the second function result can be determined based on the cross entropy loss function as the second dimension correlation loss information. For another example, for the multiple correlation features and label features shown in Formulas 2 to 7 above, three second dimension correlation loss information can be determined. Through the embodiment of the present disclosure, the correlation loss information along the first dimension and along the second dimension is determined, the difference between modal samples of different modalities can be determined based on different dimensions, and the embedding model can be trained efficiently and comprehensively.

[0086] It can be understood that some methods for determining correlation loss information are described above, and some methods for training the model to be trained will be described below.

[0087] In some embodiments, in some implementations of the above operation S140, training the embedding model to be trained according to multiple correlation loss information includes: determining target fusion loss information according to multiple first-dimensional correlation loss information and multiple second-dimensional correlation loss information.

[0088] At least one relative entropy loss information is determined based on at least one third cross-modal sample pair. For example, using the third cross-modal sample pair comprising image sample i203 and audio sample a203 as an example, relative entropy loss information can be determined based on the image sample embedding vector iemb203 and the audio sample embedding vector aemb203 and the relative entropy loss function.

[0089] The first weighted fusion loss information is obtained by weighting and fusing multiple first-dimensional correlation loss information and at least one relative entropy loss information. For example, the first weighted fusion loss information can be obtained by the following formula: :

[0090] (Formula 8)

[0091] It can be the first dimension correlation sample sub-feature in the first correlation feature, for example, the first dimension correlation sample sub-feature in the first correlation feature [ 11, ]. It can be the first dimension correlation label sub-feature in the first label feature, for example, the first dimension correlation label sub-feature in the first label feature [ 11, ]. It can be the first dimension correlation sample sub-feature in the second correlation feature, such as the first dimension correlation sample sub-feature in the above second correlation feature [ , ]. It can be the first dimension correlation label sub-feature in the second label feature, for example, the first dimension correlation label sub-feature in the above second label feature [ , ]. It can be the first dimension correlation sample sub-feature in the third correlation feature, such as the first dimension correlation sample sub-feature in the third correlation feature [ , ]. The first dimension correlation label sub-feature in the third label feature, for example, the first dimension correlation sample sub-feature in the third label feature above [ , The above description of the correlation sample value in the correlation sample sub-feature and the description of the correlation label value in the correlation label sub-feature are also applicable to Formula 8, and will not be repeated here in this disclosure.

[0092] is the relative entropy loss information. in is the embedding vector of the image sample in the third cross-modal sample pair. middle, is the embedding vector of the audio sample in the third cross-modal sample pair. is the cross entropy loss function. is the relative entropy loss function, also known as the KL divergence function. is a weight, for example, 0.01. Through the embodiments of the present disclosure, correlation loss information along the first dimension is determined, the difference between modal samples of different modalities can be determined based on the first dimension, and relative entropy loss information is determined based on a third cross-modal sample pair. The distance between the embedding vectors of semantically identical image samples and audio samples can be further reduced to train the embedding model efficiently and comprehensively.

[0093] The second weighted fusion loss information is obtained by weighting and fusing multiple second-dimensional correlation loss information and at least one relative entropy loss information. For example, the second weighted fusion loss information can be obtained by the following formula: :

[0094] (Formula 9)

[0095] It can be the second dimension correlation sample sub-feature in the first correlation feature, for example, the second dimension correlation sample sub-feature in the first correlation feature [ 11, ]. It can be the second dimension correlation label sub-feature in the first label feature, such as the second dimension correlation sample sub-feature in the first label feature [ 11, ]. It can be the second dimension correlation sample sub-feature in the second correlation feature, such as the second dimension correlation sample sub-feature in the above second correlation feature [ twenty two, ]. It can be the second dimension correlation label sub-feature in the second label feature, such as the second dimension correlation sample sub-feature in the above second label feature [ twenty two, ]. It can be the second dimension correlation sample sub-feature in the third correlation feature, for example, the second dimension correlation sample sub-feature in the third correlation feature [ 33, ]. It can be the second dimension correlation label sub-feature in the third label feature, such as the second dimension correlation sample sub-feature in the third label feature above [ 33, The above description of the correlation sample value in the correlation sample sub-feature and the description of the correlation label value in the correlation label sub-feature are also applicable to Formula 9, and will not be repeated here in this disclosure.

[0096] is the relative entropy loss information. in is the embedding vector of the image sample in the third cross-modal sample pair. middle, is the embedding vector of the audio sample in the third cross-modal sample pair. is the cross entropy loss function. is the relative entropy loss function, also known as the KL divergence function. is a weight, for example, 0.01. Through the disclosed embodiment, correlation loss information along the second dimension is determined, and the difference between modal samples of different modalities can be determined based on the second dimension to further reduce the distance between the embedding vectors of semantically identical image samples and audio samples, thereby efficiently and comprehensively training the embedding model.

[0097] The target fusion loss information is determined based on the first weighted fusion loss information and the second weighted fusion loss information. For example, based on the number of correlation loss information (e.g., 6), the target fusion loss information can be obtained by averaging the sum of the first weighted fusion loss information and the second weighted fusion loss information. The number of correlation loss information can be determined by the number of modalities (3) and the dimension of the correlation feature (2). The target fusion loss information can be determined by the following formula :

[0098] (Formula 10)

[0099] It can be the first weighted fusion loss information mentioned above. It can be the second weighted fusion loss information mentioned above.

[0100] Through the embodiments of the present disclosure, the relative entropy loss information is determined based on the third cross-modal sample pair, which can further reduce the distance between the embedding vectors of semantically identical image samples and the embedding vectors of audio samples, thereby enabling the embedding model to more efficiently and accurately embed samples of different modalities.

[0101] In some embodiments of operation S140, training the embedding model to be trained based on the plurality of correlation loss information further includes: training the embedding model to be trained based on the target fusion loss information. For example, the parameters of the model to be trained may be adjusted based on the loss information calculated in Formula 10.

[0102] It can be understood that the above describes the method of the present disclosure. When K is greater than 1, M is greater than 1, or N is greater than 1, the correlation loss information can be averaged based on the number of modal samples. For example, the cross entropy loss function can be used to determine multiple first function results between multiple first-dimensional correlation sample sub-features of the first correlation feature and multiple first-dimensional correlation label sub-features of the first label feature. The multiple first function results are added together to obtain a sum result. The sum result is divided by K or K+M+N to obtain the first-dimensional correlation loss information for the first correlation feature.

[0103] It can also be understood that in order to further efficiently map audio samples and image samples to the same vector space, the to-be-trained model of the present disclosure can be pre-trained based on pre-trained cross-modal sample pairs, which will be further explained below.

[0104] In some other embodiments, in some other implementation manners of the above operation S110, when the distance between two modality sample embedding vectors determined by the to-be-trained embedding model according to cross-modal sample pairs is greater than or equal to a preset distance threshold, the pre-trained modality samples of different modalities of each of the multiple pre-trained cross-modal sample pairs in the multiple cross-modal sample pairs are respectively input into the to-be-trained embedding model to obtain multiple pre-trained modality sample embedding vectors.

[0105] The multiple pre-trained cross-modal sample pairs include at least two of a first pre-trained cross-modal sample pair, a second pre-trained cross-modal sample pair, and a third pre-trained cross-modal sample pair. The first pre-trained cross-modal sample pair includes a pre-trained image sample and a pre-trained text sample. The second pre-trained cross-modal sample pair includes a pre-trained text sample and a pre-trained audio sample. The third pre-trained cross-modal sample pair includes a pre-trained image sample and a pre-trained audio sample.

[0106] The pre-trained image sample and the pre-trained audio sample are respectively generated by the pre-trained text sample. The pre-trained text sample includes at least one character. The pre-trained image sample includes at least one sample image block, and the graphics in the sample image block are consistent with the character. The pre-trained audio sample includes at least one sample audio segment, and the sample audio segment is consistent with the pronunciation of the character.

[0107] For example, taking the pre-trained text sample being "Hoeing the crops under the midday sun" as an example, an image with a white background and black characters can be generated as the pre-trained image sample. This pre-trained image sample includes 5 sample image blocks, which respectively have graphics consistent with the characters "hoe", "crops", "sun", "midday", and "under". A pre-trained audio sample can also be generated, which includes 5 sample audio segments, respectively consistent with the pronunciations of the characters "hoe", "crops", "sun", "midday", and "under". Based on this pre-trained text sample, this pre-trained image sample, and this pre-trained audio sample, performing the above operation S110 can obtain a pre-trained text sample embedding vector, a pre-trained image sample embedding vector, and a pre-trained audio sample embedding vector. Through the embodiments of the present disclosure, image samples and audio samples are generated based on the text sample, which can effectively pre-train the embedding model, improve the embedding processing efficiency and accuracy of the encoding network for actually collected real-scene image samples and real audio samples, and effectively reduce the model training cost.

[0108] Next, the above operations S120 and S130 can be performed to obtain multiple first-dimension loss information and multiple second-dimension loss information.

[0109] In some other embodiments of the above operation S140, at least one first pre-trained relative entropy loss information is determined based on at least one second pre-trained cross-modal sample pair. At least one second pre-trained relative entropy loss information is determined based on at least one third pre-trained cross-modal sample pair. It can be understood that the manner of determining the first pre-trained relative entropy loss information based on the second pre-trained cross-modal sample pair and the manner of determining the second pre-trained relative entropy loss information based on the third pre-trained cross-modal sample pair are the same or similar to the manner of determining the relative entropy loss information based on the above-mentioned third cross-modal sample pair, and the present disclosure will not repeat them here.

[0110] Determine the first pre-training fusion loss information based on the multiple first dimension loss information, at least one first pre-training relative entropy loss information, and at least one second pre-training relative entropy loss information. For example, the first pre-training fusion loss information can be determined by the following formula :

[0111] (Formula 11)

[0112] It can be the first dimension correlation sample sub-feature in the first correlation feature, for example, the first dimension correlation sample sub-feature in the first correlation feature [ 11, ]. It can be the first dimension correlation label sub-feature in the first label feature, for example, the first dimension correlation label sub-feature in the first label feature [ 11, ]. It can be the first dimension correlation sample sub-feature in the second correlation feature, such as the first dimension correlation sample sub-feature in the above second correlation feature [ , ]. It can be the first dimension correlation label sub-feature in the second label feature, for example, the first dimension correlation label sub-feature in the above second label feature [ , ]. It can be the first dimension correlation sample sub-feature in the third correlation feature, such as the first dimension correlation sample sub-feature in the third correlation feature [ , ]. The first dimension correlation label sub-feature in the third label feature, for example, the first dimension correlation sample sub-feature in the third label feature above [ , The above description of the correlation sample value in the correlation sample sub-feature and the description of the correlation label value in the correlation label sub-feature are also applicable to Formula 11, and will not be repeated here in this disclosure.

[0113] is the first pre-training relative entropy loss information. in is the embedding vector of the pre-trained audio sample in the second pre-trained cross-modal sample pair. in is the embedding vector of the pre-trained text sample in the second pre-trained cross-modal sample pair. is the second pre-training relative entropy loss information. in is the embedding vector of the pre-trained audio sample in the third pre-trained cross-modal sample pair. in is the embedding vector of the pre-trained image sample in the third pre-trained cross-modal sample pair. and They are respectively the weights for the first pre-training relative entropy loss information and the weights for the second pre-training relative entropy loss information, for example, both can be 0.01. Through the embodiment of the present disclosure, the pre-training correlation loss information along the first dimension is determined, the difference between modal samples of different modalities can be determined based on the first dimension, and the relative entropy loss information is determined based on the third pre-training cross-modal sample pair and the second pre-training cross-modal sample pair. The distance between the embedding vectors of the image samples with the same semantics and the embedding vectors of the audio samples can be further reduced, and the distance between the embedding vectors of the audio sample state with the same semantics and the embedding vectors of the text samples can also be further reduced, so as to pre-train the embedding model efficiently and comprehensively.

[0114] The second pre-training fusion loss information is determined based on the plurality of second dimension loss information, at least one first pre-training relative entropy loss information and at least one second pre-training relative entropy loss information. The first pre-training fusion loss information can be determined by the following formula :

[0115] (Formula 12)

[0116] It can be the second dimension correlation sample sub-feature in the first correlation feature, for example, the second dimension correlation sample sub-feature in the first correlation feature [ 11, ]. It can be the second dimension correlation label sub-feature in the first label feature, such as the second dimension correlation sample sub-feature in the first label feature [ 11, ]. It can be the second dimension correlation sample sub-feature in the second correlation feature, such as the second dimension correlation sample sub-feature in the above second correlation feature [ twenty two, ]. It can be the second dimension correlation label sub-feature in the second label feature, such as the second dimension correlation sample sub-feature in the above second label feature [ twenty two, ]. It can be the second dimension correlation sample sub-feature in the third correlation feature, for example, the second dimension correlation sample sub-feature in the third correlation feature [ 33, ]. It can be the second dimension correlation label sub-feature in the third label feature, such as the second dimension correlation sample sub-feature in the third label feature above [ 33, The above description of the correlation sample value in the correlation sample sub-feature and the description of the correlation label value in the correlation label sub-feature are also applicable to Formula 12, and will not be repeated in this disclosure.

[0117] is the first pre-training relative entropy loss information. in is the embedding vector of the pre-trained audio sample in the second pre-trained cross-modal sample pair. in is the embedding vector of the pre-trained text sample in the second pre-trained cross-modal sample pair. is the second pre-training relative entropy loss information. in is the embedding vector of the pre-trained audio sample in the third pre-trained cross-modal sample pair. in is the embedding vector of the pre-trained image sample in the third pre-trained cross-modal sample pair. and They are respectively the weights for the first pre-training relative entropy loss information and the weights for the second pre-training relative entropy loss information, for example, both can be 0.01. Through the embodiment of the present disclosure, the pre-training correlation loss information along the second dimension is determined, the difference between modal samples of different modalities can be determined based on the second dimension, and the relative entropy loss information is determined based on the third pre-training cross-modal sample pair and the second pre-training cross-modal sample pair. The distance between the embedding vectors of the image samples with the same semantics and the embedding vectors of the audio samples can be further reduced, and the distance between the embedding vectors of the audio sample state with the same semantics and the embedding vectors of the text samples can also be further reduced, so as to pre-train the embedding model efficiently and comprehensively.

[0118] According to the first pre-training fusion loss information and the second pre-training fusion loss information, the embedding model to be trained is trained. The target fusion loss information can be determined by the following formula :

[0119] (Formula 13)

[0120] The loss information can be fused for the first pre-training mentioned above. The loss information can be fused for the second pre-training mentioned above.

[0121] It can be understood that after multiple pre-trainings based on multiple batches of pre-trained cross-modal sample pairs, a pre-trained embedding model can be obtained. Through the embodiments of the present disclosure, relative entropy loss information is determined based on the second pre-trained cross-modal sample and the third pre-trained cross-modal sample pair, respectively, which can further reduce the distance between the embedding vectors of semantically identical image samples and the embedding vectors of audio samples, and can also further reduce the distance between the embedding vectors of semantically identical audio samples and the embedding vectors of text samples, thereby enabling the embedding model to more efficiently and accurately embed samples of different modalities.

[0122] Next, the trained embedding model can be further trained based on the above formulas 8 to 10 to obtain the target embedding model.

[0123] It can be understood that the above describes the model training method of the present invention, and the following describes the question and answer information processing method of the present invention.

[0124] Figure 3 is a flowchart of a question and answer information processing method according to another embodiment of the present disclosure.

[0125] like Figure 3 As shown, the method 300 may include operations S310 to S320.

[0126] In operation S310 , at least one target knowledge data is determined from a plurality of knowledge data according to initial question information as context information for the initial question information.

[0127] In the embodiment of the present disclosure, the initial question information may be question information in various modalities such as image, text, audio, and video.

[0128] In the disclosed embodiment, the indexes of the plurality of knowledge data are determined based on the embedding vectors of the plurality of knowledge data. The embedding vectors of the knowledge data are obtained by inputting the knowledge data into a target embedding model. The target embedding model is obtained by training the target embedding model according to the model training method provided in the present disclosure. For example, the target embedding model can be obtained by training the target embedding model according to the method 100 described above.

[0129] In the embodiment of the present disclosure, the knowledge data may be data in various modalities such as images, text, and audio.

[0130] In operation S320, answer information for the initial question information is determined based on the context information.

[0131] In an embodiment of the present disclosure, a large model may be used to determine new types of answer information for an initial question.

[0132] Through the embodiments of the present disclosure, the embedding vectors of knowledge data of different modalities are determined by a unified target embedding model, which can quickly determine knowledge data of various modalities that are semantically similar to the question information, thereby providing comprehensive and rich contextual information for determining the answer information.

[0133] It can be understood that the above describes the problem information processing method of the present disclosure, and the method of the present disclosure will be further described below.

[0134] Figure 4 FIG. 4 is a schematic diagram of determining indexes of a plurality of knowledge data according to an embodiment of the present disclosure.

[0135] like Figure 4 As shown, multiple documents in a document collection doc40 can be parsed to extract paragraphs para40, images image40, and charts chart40 from the documents. Chart chart40 can be further parsed to determine the chart description text ct40 for chart chart40. Using the target embedding model, the paragraph embedding vector pemb40 for paragraph para40, the image embedding vector iemb40 for image image40, the chart embedding vector cemb40 for chart chart40, and the description text embedding vector ctemb40 for the chart description text ct40 can be determined as indexes for the paragraph para40, image image40, chart chart40, and chart description text ct40, respectively. The indexes for paragraph para40 and chart description text ct40 can be stored in a text index library til40. The indexes for image image40 and chart chart40 can be stored in a multimodal index library mil40.

[0136] like Figure 4As shown, the target embedding model can be used to determine the first audio embedding vector aemb41 of the first audio a41 as the index of the first audio a41. The index of the first audio a41 can be stored in the multimodal index library mil40.

[0137] like Figure 4 As shown, the second audio a42 in the video v40 can be extracted. The video v40 is frame extracted to obtain the video frame vf40. Using the target embedding model, the second audio embedding vector aemb42 of the second audio a42 and the video frame embedding vector vfemb40 of the video frame vf40 can be determined as the index of the second audio a42 and the index of the video frame vf40 respectively. The index of the second audio a42 can be stored in the multimodal index library mil40. The index of the video frame vf40 can be stored in the multimodal index library mil40. It can be understood that the index of the text can be stored in the text index library til40. The indexes of audio, image, and video frames can be stored in the multimodal index library mil40.

[0138] It can be understood that the above describes the method for establishing the index of the knowledge data of the present disclosure, and the method of the present disclosure will be further described below.

[0139] Figure 5 It is a schematic diagram of a question-and-answer information processing method according to an embodiment of the present disclosure.

[0140] In some embodiments of operation S310, determining at least one target knowledge data from the plurality of knowledge data based on the initial question information includes: determining the target question information using a large model based on the initial question information; inputting the initial question information and the target question information into a target embedding model to obtain an initial question embedding vector and a target question embedding vector; and determining at least one target knowledge data from the plurality of knowledge data based on the initial question embedding vector and the target question embedding vector.

[0141] like Figure 5 As shown, the initial question information q50 can be rewritten using the large model to obtain the target question information q51. Next, it can be determined whether a search based on the question information is necessary. For example, if the initial question information is a mathematical calculation problem, it can be determined that a search based on the question information is not necessary. The initial question information and the target question information can be provided to the multimodal large model mlm50. For another example, if the initial question information is a data generation problem, it can be determined that a search based on the question information is necessary. The following example uses the need to search based on the initial question information q50 and the target question information q51 as an example.

[0142] The above-mentioned target embedding model can be used to determine the initial question embedding vector of the initial question information q50 and the target question embedding vector of the target question information q51. The initial question embedding vector and the target question embedding vector are spliced ​​together to obtain a splicing vector. The splicing vector is used to search in the text index library til50 and the multimodal index library mil50 to obtain one or more target indexes. The knowledge data corresponding to the target index is used as the target knowledge data kd50 as the context information of the initial question information q50. Next, the above-mentioned operation S320 can be performed to provide the context information to the multimodal large model mlm50 to obtain the answer information ans50 for the initial question information q50.

[0143] It can be understood that the above describes the method of the present disclosure, and the following will describe the device of the present disclosure.

[0144] Figure 6 is a block diagram of a model training apparatus according to an embodiment of the present disclosure.

[0145] like Figure 6 As shown, the apparatus 600 may include a first obtaining module 610 , a first determining module 620 , a second determining module 630 and a training module 640 .

[0146] The first obtaining module 610 is configured to input modal samples of different modalities of a plurality of cross-modal sample pairs into a plurality of encoding networks for different modalities in the embedding model to be trained, thereby obtaining a plurality of modal sample embedding vectors. A cross-modal sample pair includes two modal samples of different modalities.

[0147] The first determining module 620 is configured to determine at least one correlation feature based on the multiple modality sample embedding vectors. The correlation feature includes a correlation sample value between two modality sample embedding vectors of different modalities.

[0148] The second determining module 630 is configured to determine at least one piece of correlation loss information based on at least one label feature and at least one correlation feature. The label feature includes a correlation label value between two modality samples of different modalities.

[0149] The training module 640 is used to train the embedding model to be trained according to the multiple correlation loss information.

[0150] In some embodiments, the multiple cross-modal sample pairs include at least two of a first cross-modal sample pair, a second cross-modal sample pair, and a third cross-modal sample pair, the first cross-modal sample pair including an image sample and a text sample, the second cross-modal sample pair including a text sample and an audio sample, and the third cross-modal sample pair including an image sample and an audio sample. The semantic similarity between the two modal samples of different modalities in the cross-modal sample pair is greater than or equal to a preset semantic similarity threshold. The multiple modal sample embedding vectors include an image sample embedding vector, a text sample embedding vector, and an audio sample embedding vector.

[0151] In some embodiments, the multiple encoding networks for different modalities include an image encoding network, a text encoding network, and an audio encoding network. The number of first cross-modal sample pairs is K, the number of second cross-modal sample pairs is M, and the number of third cross-modal sample pairs is N. K, N, and M are all non-negative integers, and at least two of K, N, and M are greater than or equal to 1. The first acquisition module 610 includes: a first acquisition submodule, which is used to input K+N image samples, M+K text samples, and N+M audio samples into the image encoding network, text encoding network, and audio encoding network of the embedding model to be trained, respectively, to obtain K+N image sample embedding vectors, M+K text sample embedding vectors, and N+M audio sample embedding vectors.

[0152] In some embodiments, the first obtaining submodule includes: a first obtaining unit for inputting K+N image samples into an image coding network to obtain K+N image sample embedding vectors; a second obtaining unit for inputting M+K text samples into a text coding network to obtain M+K text sample embedding vectors; and a third obtaining unit for inputting N+M audio samples into an audio coding network to obtain N+M audio sample embedding vectors.

[0153] In some embodiments, the first determination module 620 includes: a first determination submodule, configured to determine a correlation sample value based on a product result between two modality sample embedding vectors of different modalities and a training temperature parameter.

[0154] In some embodiments, at least one label feature includes a first label feature, a second label feature, and a third label feature, the first label feature includes (K+N)×(M+K) first correlation label values ​​between K+N image samples and M+K text samples, the second label feature includes (M+K)×(N+M) second correlation label values ​​between M+K text samples and N+M audio samples, and the third label feature includes (K+N)×(N+M) third correlation label values ​​between K+N image samples and N+M audio samples, K, N, and M are all non-negative integers and at least two of K, N, and M are greater than or equal to 1. At least one correlation feature includes a first correlation feature, a second correlation feature and a third correlation feature, the first correlation feature includes (K+N)×(M+K) first correlation sample values ​​between K+N image sample embedding vectors and M+K text sample embedding vectors, the second correlation feature includes (M+K)×(N+M) second correlation sample values ​​between M+K text sample embedding vectors and N+M audio sample embedding vectors, and the third correlation feature includes (K+N)×(N+M) third correlation sample values ​​between K+N image sample embedding vectors and N+M audio sample embedding vectors.

[0155] In some embodiments, the first determination submodule includes at least one of the following units: a first determination unit, configured to determine a first correlation sample value based on a first multiplication result between the text sample embedding vector and the transposed image sample embedding vector and a first training temperature parameter. The transposed image sample embedding vector is obtained by transposing the image sample embedding vector. A second determination unit, configured to determine a second correlation sample value based on a second multiplication result between the audio sample embedding vector and the transposed text sample embedding vector and a second training temperature parameter. The transposed text sample embedding vector is obtained by transposing the text sample embedding vector. A third determination unit, configured to determine a third correlation sample value based on a third multiplication result between the image sample embedding vector and the transposed audio sample embedding vector and a third training temperature parameter. The transposed audio sample embedding vector is obtained by transposing the audio sample embedding vector.

[0156] In some embodiments, the correlation feature includes at least one first-dimension correlation sample sub-feature and at least one second-dimension correlation sample sub-feature, wherein the first-dimension correlation sample sub-feature includes at least one correlation sample value between a first modality sample embedding vector and at least one second modality sample embedding vector, and the second-dimension correlation sample sub-feature includes at least one correlation sample value between a second modality sample embedding vector and at least one first modality sample embedding vector. The label feature includes at least one first-dimension correlation label sub-feature and at least one second-dimension correlation label sub-feature, wherein the first-dimension correlation label sub-feature includes at least one correlation label value between a first modality sample and at least one second modality sample, and the second-dimension correlation label sub-feature includes at least one correlation label value between a second modality sample and at least one first modality sample.

[0157] In some embodiments, the second determination module 630 includes: a second determination submodule for determining a plurality of first-dimensional correlation loss information based on at least one first-dimensional correlation label subfeature of each of the plurality of label features and at least one first-dimensional correlation sample subfeature of each of the plurality of correlation features; and a third determination submodule for determining a plurality of second-dimensional correlation loss information based on at least one second-dimensional correlation label subfeature of each of the plurality of label features and at least one second-dimensional correlation sample subfeature of each of the plurality of correlation features.

[0158] In some embodiments, the training module 640 includes: a fourth determination submodule for determining target fusion loss information based on the plurality of first-dimensional correlation loss information and the plurality of second-dimensional correlation loss information; and a training submodule for training the embedding model to be trained based on the target fusion loss information.

[0159] In some embodiments, the fourth determination submodule includes: a fourth determination unit for determining at least one relative entropy loss information based on at least one third cross-modal sample pair; a first fusion unit for weightedly fusing multiple first-dimensional correlation loss information and at least one relative entropy loss information to obtain first weighted fusion loss information; a second fusion unit for weightedly fusing multiple second-dimensional correlation loss information and at least one relative entropy loss information to obtain second weighted fusion loss information; and a fifth determination unit for determining target fusion loss information based on the first weighted fusion loss information and the second weighted fusion loss information.

[0160] In some embodiments, the first obtaining module 610 includes: a second obtaining submodule for inputting pre-trained modal samples of different modalities of multiple pre-trained cross-modal sample pairs in the multiple cross-modal sample pairs into the embedding model to be trained, respectively, to obtain multiple pre-trained modal sample embedding vectors, when the distance between the two modal sample embedding vectors determined by the embedding model to be trained based on the cross-modal sample pair is greater than or equal to a preset distance threshold. The multiple pre-trained cross-modal sample pairs include at least two of a first pre-trained cross-modal sample pair, a second pre-trained cross-modal sample pair, and a third pre-trained cross-modal sample pair, the first pre-trained cross-modal sample pair including a pre-trained image sample and a pre-trained text sample, the second pre-trained cross-modal sample pair including a pre-trained text sample and a pre-trained audio sample, and the third pre-trained cross-modal sample pair including a pre-trained image sample and a pre-trained audio sample. The pre-trained image samples and pre-trained audio samples are respectively generated from pre-trained text samples. The pre-trained text samples include at least one character. The pre-trained image samples include at least one sample image block. The graphics in the sample image block are consistent with the character. The pre-trained audio samples include at least one sample audio segment. The sample audio segment is consistent with the pronunciation of the character.

[0161] In some embodiments, the training module 640 includes: a sixth determination submodule for determining at least one first pre-training relative entropy loss information based on at least one second pre-training cross-modal sample pair. A seventh determination submodule for determining at least one second pre-training relative entropy loss information based on at least one third pre-training cross-modal sample pair. An eighth determination submodule for determining first pre-training fusion loss information based on multiple first-dimensional loss information, at least one first pre-training relative entropy loss information, and at least one second pre-training relative entropy loss information. A ninth determination submodule for determining second pre-training fusion loss information based on multiple second-dimensional loss information, at least one first pre-training relative entropy loss information, and at least one second pre-training relative entropy loss information. A second training submodule for training the embedding model to be trained based on the first pre-training fusion loss information and the second pre-training fusion loss information.

[0162] Figure 7 It is a block diagram of a question and answer information processing apparatus according to an embodiment of the present disclosure.

[0163] like Figure 7 As shown, the apparatus 700 may include a third determination module 710 and a fourth determination module 720 .

[0164] The third determining module 710 is configured to determine, according to the initial question information, at least one target knowledge data from the plurality of knowledge data as context information for the initial question information.

[0165] The fourth determining module 720 is configured to determine answer information for the initial question information based on the context information.

[0166] In some embodiments, the index of each of the multiple knowledge data is determined based on the embedding vector of each of the multiple knowledge data. The embedding vector of the knowledge data is obtained by inputting the knowledge data into the target embedding model. The target embedding model is obtained by training the embedding model to be trained according to the model training device provided by the present disclosure.

[0167] In some embodiments, the third determination module 710 includes: a tenth determination submodule for determining target question information using the large model based on the initial question information; a third acquisition submodule for inputting the initial question information and the target question information into the target embedding model to obtain an initial question embedding vector and a target question embedding vector; and an eleventh determination submodule for determining at least one target knowledge data from the plurality of knowledge data based on the initial question embedding vector and the target question embedding vector.

[0168] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0169] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0170] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0171] like Figure 8As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. Computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to bus 804.

[0172] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0173] The computing unit 801 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as at least one of the model training method and the question-and-answer information processing method. For example, in some embodiments, at least one of the model training method and the question-and-answer information processing method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into RAM 803 and executed by computing unit 801, one or more steps of at least one of the model training method and the question-and-answer information processing method described above may be performed. Alternatively, in other embodiments, computing unit 801 may be configured to perform at least one of the model training method and the question-and-answer information processing method in any other suitable manner (e.g., via firmware).

[0174] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0175] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0176] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM) or flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0177] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) display or a liquid crystal display (LCD)) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0178] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0179] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0180] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0181] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A model training method, comprising: Inputting modality samples of different modalities of a plurality of cross-modal sample pairs into a plurality of encoding networks for different modalities in an embedding model to be trained, respectively, to obtain a plurality of modality sample embedding vectors, wherein the cross-modal sample pairs include two modality samples of different modalities; Determining at least one correlation feature based on the plurality of modality sample embedding vectors, wherein the correlation feature comprises a correlation sample value between two modality sample embedding vectors of different modalities; Determining at least one piece of correlation loss information based on at least one label feature and at least one of the correlation features, wherein the label feature includes a correlation label value between two modality samples of different modalities; The embedding model to be trained is trained according to the plurality of correlation loss information.

2. The method according to claim 1, wherein The plurality of cross-modal sample pairs include at least two of a first cross-modal sample pair, a second cross-modal sample pair, and a third cross-modal sample pair, wherein the first cross-modal sample pair includes an image sample and a text sample, the second cross-modal sample pair includes a text sample and an audio sample, and the third cross-modal sample pair includes an image sample and an audio sample. The semantic similarity between the two modal samples of different modalities in the cross-modal sample pair is greater than or equal to a preset semantic similarity threshold, The multiple modality sample embedding vectors include an image sample embedding vector, a text sample embedding vector, and an audio sample embedding vector.

3. The method according to claim 2, wherein: The multiple encoding networks for different modalities include an image encoding network, a text encoding network, and an audio encoding network, the number of the first cross-modal sample pairs is K, the number of the second cross-modal sample pairs is M, and the number of the third cross-modal sample pairs is N, K, N, and M are all non-negative integers and at least two of K, N, and M are greater than or equal to 1, The method of inputting modal samples of different modalities of the plurality of cross-modal sample pairs into the plurality of encoding networks for different modalities in the embedding model to be trained to obtain the plurality of modal sample embedding vectors includes: Input K+N image samples, M+K text samples, and N+M audio samples into the image encoding network, text encoding network, and audio encoding network of the embedding model to be trained, respectively, to obtain K+N image sample embedding vectors, M+K text sample embedding vectors, and N+M audio sample embedding vectors.

4. The method according to claim 3, wherein: The step of inputting K+N image samples, M+K text samples, and N+M audio samples into the image coding network, text coding network, and audio coding network of the embedding model to be trained, respectively, to obtain K+N image sample embedding vectors, M+K text sample embedding vectors, and N+M audio sample embedding vectors comprises: Inputting K+N image samples into the image coding network to obtain K+N image sample embedding vectors; Inputting M+K text samples into the text encoding network to obtain M+K text sample embedding vectors; Inputting N+M of the audio samples into the audio coding network to obtain N+M audio sample embedding vectors.

5. The method according to claim 2, wherein: Determining at least one correlation feature based on the plurality of modality sample embedding vectors includes: The correlation sample value is determined according to the product result between the two modality sample embedding vectors of different modalities and the training temperature parameter.

6. The method according to claim 5, wherein: At least one of the label features includes a first label feature, a second label feature, and a third label feature, wherein the first label feature includes (K+N)×(M+K) first correlation label values ​​between K+N image samples and M+K text samples, the second label feature includes (M+K)×(N+M) second correlation label values ​​between M+K text samples and N+M audio samples, and the third label feature includes (K+N)×(N+M) third correlation label values ​​between K+N image samples and N+M audio samples, K, N, and M are all non-negative integers and at least two of K, N, and M are greater than or equal to 1. At least one of the correlation features includes a first correlation feature, a second correlation feature and a third correlation feature, the first correlation feature includes (K+N)×(M+K) first correlation sample values ​​between K+N image sample embedding vectors and M+K text sample embedding vectors, the second correlation feature includes (M+K)×(N+M) second correlation sample values ​​between M+K text sample embedding vectors and N+M audio sample embedding vectors, and the third correlation feature includes (K+N)×(N+M) third correlation sample values ​​between K+N image sample embedding vectors and N+M audio sample embedding vectors.

7. The method according to claim 6, wherein: Determining the correlation sample value according to the product result between the two modality sample embedding vectors of different modalities and the training temperature parameter includes at least one of the following operations: determining the first correlation sample value according to a first product result between the text sample embedding vector and the transposed image sample embedding vector and a first training temperature parameter, wherein the transposed image sample embedding vector is obtained by transposing the image sample embedding vector; determining the second correlation sample value based on a second product result between the audio sample embedding vector and the transposed text sample embedding vector and a second training temperature parameter, wherein the transposed text sample embedding vector is obtained by transposing the text sample embedding vector; Determine the third correlation sample value based on a third product result between the image sample embedding vector and the transposed audio sample embedding vector and a third training temperature parameter, wherein the transposed audio sample embedding vector is obtained by transposing the audio sample embedding vector.

8. The method according to claim 2, wherein: The correlation feature includes at least one first-dimensional correlation sample sub-feature and at least one second-dimensional correlation sample sub-feature, wherein the first-dimensional correlation sample sub-feature includes at least one correlation sample value between a first modality sample embedding vector and at least one second modality sample embedding vector, and the second-dimensional correlation sample sub-feature includes at least one correlation sample value between the second modality sample embedding vector and at least one first modality sample embedding vector. The label feature includes at least one first dimension correlation label sub-feature and at least one second dimension correlation label sub-feature, the first dimension correlation label sub-feature includes at least one correlation label value between a first modality sample and at least one second modality sample, and the second dimension correlation label sub-feature includes at least one correlation label value between the second modality sample and at least one first modality sample.

9. The method according to claim 8, wherein The determining, based on the at least one tag feature and the at least one correlation feature, at least one correlation loss information includes: determining a plurality of first-dimensional correlation loss information according to at least one first-dimensional correlation label sub-feature of each of the plurality of label features and at least one first-dimensional correlation sample sub-feature of each of the plurality of correlation features; A plurality of second-dimensional correlation loss information is determined according to at least one second-dimensional correlation label sub-feature of each of the plurality of label features and at least one second-dimensional correlation sample sub-feature of each of the plurality of correlation features.

10. The method according to claim 9, wherein: The training of the embedding model to be trained according to the plurality of correlation loss information includes: Determining target fusion loss information according to the plurality of first-dimensional correlation loss information and the plurality of second-dimensional correlation loss information; The embedding model to be trained is trained according to the target fusion loss information.

11. The method according to claim 10, wherein: The determining target fusion loss information according to the plurality of first-dimensional correlation loss information and the plurality of second-dimensional correlation loss information includes: Determining at least one relative entropy loss information according to at least one third cross-modal sample pair; Weightedly fusing the plurality of first-dimensional correlation loss information and the at least one relative entropy loss information to obtain first weighted fusion loss information; Weightedly fusing the plurality of second-dimensional correlation loss information and the at least one relative entropy loss information to obtain second weighted fusion loss information; The target fusion loss information is determined according to the first weighted fusion loss information and the second weighted fusion loss information.

12. The method according to claim 9, wherein The method of inputting modal samples of different modalities of the plurality of cross-modal sample pairs into the plurality of encoding networks for different modalities in the embedding model to be trained to obtain the plurality of modal sample embedding vectors includes: When the distance between the embedding vectors of two modal samples determined by the embedding model to be trained according to the cross-modal sample pair is greater than or equal to a preset distance threshold, pre-trained modal samples of different modalities of a plurality of pre-trained cross-modal sample pairs in the plurality of cross-modal sample pairs are respectively input into the embedding model to be trained to obtain a plurality of pre-trained modal sample embedding vectors. The plurality of pre-trained cross-modal sample pairs include at least two of a first pre-trained cross-modal sample pair, a second pre-trained cross-modal sample pair, and a third pre-trained cross-modal sample pair, wherein the first pre-trained cross-modal sample pair includes a pre-trained image sample and a pre-trained text sample, the second pre-trained cross-modal sample pair includes a pre-trained text sample and a pre-trained audio sample, and the third pre-trained cross-modal sample pair includes a pre-trained image sample and a pre-trained audio sample. The pre-trained image sample and the pre-trained audio sample are respectively generated from the pre-trained text sample, the pre-trained text sample includes at least one character, the pre-trained image sample includes at least one sample image block, the graphics in the sample image block are consistent with the character, and the pre-trained audio sample includes at least one sample audio segment, and the sample audio segment is consistent with the pronunciation of the character.

13. The method according to claim 12, wherein: The training of the embedding model to be trained according to the plurality of correlation loss information includes: Determining at least one first pre-trained relative entropy loss information according to at least one second pre-trained cross-modal sample pair; Determining at least one second pre-trained relative entropy loss information according to at least one third pre-trained cross-modal sample pair; Determine first pre-training fusion loss information according to the plurality of first dimension loss information, at least one first pre-training relative entropy loss information, and at least one second pre-training relative entropy loss information; Determine second pre-training fusion loss information according to the plurality of second dimension loss information, at least one first pre-training relative entropy loss information, and at least one second pre-training relative entropy loss information; The embedding model to be trained is trained according to the first pre-training fusion loss information and the second pre-training fusion loss information.

14. A method for processing question-answer information, comprising: determining, according to the initial question information, at least one target knowledge data from a plurality of knowledge data as context information for the initial question information; determining answer information for the initial question information based on the context information, In which, the index of each of the multiple knowledge data is determined based on the embedding vector of each of the multiple knowledge data, the embedding vector of the knowledge data is obtained by inputting the knowledge data into the target embedding model, and the target embedding model is obtained by training the embedding model to be trained according to the method described in any one of claims 1 to 13.

15. The method according to claim 14, wherein The step of determining at least one target knowledge data from a plurality of knowledge data according to the initial question information includes: Based on the initial problem information, use the big model to determine the target problem information; Inputting the initial question information and the target question information into the target embedding model to obtain an initial question embedding vector and a target question embedding vector; At least one target knowledge data is determined from the plurality of knowledge data according to the initial question embedding vector and the target question embedding vector.

16. A model training device comprising: A first obtaining module is configured to input modal samples of different modalities of a plurality of cross-modal sample pairs into a plurality of encoding networks for different modalities in an embedding model to be trained, respectively, to obtain a plurality of modal sample embedding vectors, wherein the cross-modal sample pairs include two modal samples of different modalities; a first determining module, configured to determine at least one correlation feature based on the plurality of modality sample embedding vectors, wherein the correlation feature comprises a correlation sample value between two modality sample embedding vectors of different modalities; a second determining module, configured to determine at least one piece of correlation loss information based on at least one label feature and at least one of the correlation features, wherein the label feature includes a correlation label value between two modality samples of different modalities; A training module is used to train the embedding model to be trained based on the multiple correlation loss information.

17. A question-answer information processing device, comprising: a third determining module, configured to determine, according to the initial question information, at least one target knowledge data from the plurality of knowledge data as context information for the initial question information; a fourth determining module, configured to determine answer information for the initial question information based on the context information, In which, the index of each of the multiple knowledge data is determined based on the embedding vector of each of the multiple knowledge data, and the embedding vector of the knowledge data is obtained by inputting the knowledge data into the target embedding model, and the target embedding model is obtained by training the embedding model to be trained by the device according to claim 16.

18. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 15.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 15.

20. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 15.

Images