Model data processing method and apparatus, electronic device, and computer readable medium

Abstract

The application discloses a model data processing method and device, and relates to the technical field of artificial intelligence. A specific implementation manner of the method comprises the following steps: acquiring data of at least two different modalities in a to-be-processed data set; performing feature extraction, splicing and / or superimposition on the data of the at least two different modalities to obtain a feature sequence; performing model mapping processing on the feature sequence to obtain multi-modal input data compatible with an autoregressive model; and inputting the multi-modal input data into the autoregressive model to obtain single-modal results output by the autoregressive model. The implementation manner improves the richness of the information quantity of model input data.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computers, in particular to the technical field of artificial intelligence, and especially to a model data processing method and device, an electronic device, a computer readable medium, and a computer program product. BACKGROUND

[0002] Existing dialogue system datasets are mostly single-modal datasets, and the corresponding dialogue generation system research is limited by the datasets. The models researched are also single-modal. However, in actual application scenarios, the questions raised by customers in the dialogue system are often multi-modal, such as text information + picture information. This is because with the widespread use of smartphones, customers are more willing to take a picture and send it to the customer service. A picture is worth a thousand words, and a picture is used to describe the problem in more detail. However, there is no existing technology that combines a multi-modal dataset with a pre-trained single-modal language model. SUMMARY

[0003] Embodiments of the present disclosure provide a model data processing method and device, an electronic device, a computer readable medium, and a computer program product.

[0004] In a first aspect, embodiments of the present disclosure provide a model data processing method, which includes: obtaining data of at least two different modalities in a to-be-processed dataset; performing feature extraction, splicing, and / or superimposition on the data of the at least two different modalities to obtain a feature sequence; performing model mapping processing on the feature sequence to obtain multi-modal input data compatible with an autoregressive model; inputting the multi-modal input data into the autoregressive model to obtain a single-modal result output by the autoregressive model.

[0005] In some embodiments, the data of the at least two different modalities includes to-be-processed text data and knowledge graph data; the feature extraction, splicing, and / or superimposition on the data of the at least two different modalities to obtain the feature sequence includes: collecting triadic text data corresponding to the knowledge graph data; performing word segmentation processing on the triadic text data and the to-be-processed text data respectively, and splicing all word segmentation results to obtain the feature sequence.

[0006] In some embodiments, the method further includes: determining total position data of each word segmentation result in all word segmentation results spliced in the feature sequence; the model mapping processing on the feature sequence to obtain the multi-modal input data compatible with the autoregressive model includes: performing vector mapping on the feature sequence to obtain a data feature vector corresponding to the feature sequence; performing vector mapping on the total position data to obtain a total position vector; and adding the data feature vector and the total position vector to obtain the multi-modal input data compatible with the autoregressive model.

[0007] In some embodiments, the method further comprises: performing classification processing on each of the segmented results of the spliced feature sequence to obtain total classification data; and performing model mapping processing on the feature sequence to obtain multi-modal input data compatible with the autoregressive model, comprising: performing vector mapping on the feature sequence compatible with the autoregressive model to obtain a data feature vector corresponding to the feature sequence; performing vector mapping on the total classification data compatible with the autoregressive model to obtain a total classification vector; and adding the data feature vector and the total classification vector to obtain the multi-modal input data compatible with the autoregressive model.

[0008] In some embodiments, the at least two different modalities of data further comprise: image data; and the feature sequence obtained by performing feature extraction on the at least two different modalities of data and splicing and / or superimposing the at least two different modalities of data further comprises: image features obtained by performing feature extraction on the image data; and the obtained feature sequence further superimposes the image features; and the model mapping processing performed on the feature sequence to obtain the multi-modal input data compatible with the autoregressive model comprises: performing vector mapping on all segmented results of the spliced feature sequence compatible with the autoregressive model to obtain data feature vectors corresponding to all segmented results of the spliced feature sequence; performing vector mapping on the image features in the feature sequence to obtain image feature vectors of the same dimension as an input layer of the autoregressive model; and adding the data feature vectors and the image feature vectors to obtain the multi-modal input data compatible with the autoregressive model.

[0009] In some embodiments, the method further comprises: determining total position data of each of the segmented results of the spliced feature sequence; performing classification processing on each of the segmented results of the spliced feature sequence to obtain total classification data; and performing model mapping processing on the feature sequence to obtain multi-modal input data compatible with the autoregressive model, further comprising: performing vector mapping on the total position data compatible with the autoregressive model to obtain a total position vector; performing vector mapping on the total classification data compatible with the autoregressive model to obtain a total classification vector; and the obtained multi-modal input data compatible with the autoregressive model further comprises the total position vector and the total classification vector.

[0010] In some embodiments, the at least two different modalities of data include: text data to be processed and image data; the feature extraction, splicing and / or superimposition of the at least two different modalities of data to obtain a feature sequence includes: performing word segmentation processing on the text data to be processed, and splicing all word segmentation results to obtain a word sequence; performing feature extraction on the image data to obtain image features; superimposing the word sequence and the image features to obtain the feature sequence; the model mapping processing of the feature sequence to obtain multi-modal input data compatible with the autoregressive model includes: performing vector mapping of the word sequence in the feature sequence to obtain a data feature vector of the word sequence compatible with the autoregressive model; performing vector mapping of the image features in the feature sequence to obtain an image feature vector of the same dimension as the input layer of the autoregressive model; and adding the data feature vector and the image feature vector to obtain multi-modal input data compatible with the autoregressive model.

[0011] In some embodiments, the method further includes: determining word position data of each word in the word sequence; performing classification processing on each word in the word sequence to obtain word classification data; and the model mapping processing of the feature sequence to obtain multi-modal input data compatible with the autoregressive model further includes: performing vector mapping of the word position data to obtain a word position vector compatible with the autoregressive model; performing vector mapping of the word classification data to obtain a word classification vector compatible with the autoregressive model; and the obtained multi-modal input data compatible with the autoregressive model further includes the word position vector and the word classification vector.

[0012] In a second aspect, embodiments of the present disclosure provide a model data processing apparatus, which includes: an acquisition unit configured to acquire at least two different modalities of data in a data set to be processed; an extraction unit configured to perform feature extraction, splicing and / or superimposition on the at least two different modalities of data to obtain a feature sequence; an obtaining unit configured to perform model mapping processing on the feature sequence to obtain multi-modal input data compatible with an autoregressive model; and a processing unit configured to input the multi-modal input data into the autoregressive model to obtain a single-modal result output by the autoregressive model.

[0013] In some embodiments, the at least two different modalities of data include: text data to be processed and knowledge graph data; and the extraction unit includes: a collection module configured to collect triadic text data corresponding to the knowledge graph data; a total word segmentation module configured to perform word segmentation processing on the triadic text data and the text data to be processed, respectively; and a total splicing module configured to splice all word segmentation results to obtain the feature sequence.

[0014] In some embodiments, the apparatus further comprises a total position unit configured to determine total position data of each of the spliced word segmentation results in the feature sequence; the obtaining unit comprises a data mapping module configured to perform vector mapping on the feature sequence to adapt to the autoregressive model to obtain a data feature vector corresponding to the feature sequence; a total position mapping module configured to perform vector mapping on the total position data to adapt to the autoregressive model to obtain a total position vector; and a number obtaining module configured to add the data feature vector and the total position vector to obtain the multi-modal input data adapted to the autoregressive model.

[0015] In some embodiments, the apparatus further comprises a total classification unit configured to perform classification processing on each of the spliced word segmentation results in the feature sequence to obtain total classification data; the obtaining unit comprises a data mapping module configured to perform vector mapping on the feature sequence to adapt to the autoregressive model to obtain a data feature vector corresponding to the feature sequence; a total classification mapping module configured to perform vector mapping on the total classification data to adapt to the autoregressive model to obtain a total classification vector; and a number classification obtaining module configured to add the data feature vector and the total classification vector to obtain the multi-modal input data adapted to the autoregressive model.

[0016] In some embodiments, the at least two different modalities of data further comprise image data; the extraction unit is further configured to perform feature extraction on the image data to obtain image features; the obtained feature sequence further superimposes the image features; the obtaining unit comprises a data mapping module configured to perform vector mapping on all the spliced word segmentation results in the feature sequence to adapt to the autoregressive model to obtain a data feature vector corresponding to all the spliced word segmentation results; an image mapping module configured to perform vector mapping on the image features in the feature sequence to adapt to the input layer of the autoregressive model to obtain an image feature vector; and a number image obtaining module configured to add the data feature vector and the image feature vector to obtain the multi-modal input data adapted to the autoregressive model.

[0017] In some embodiments, the apparatus further comprises a total position unit configured to determine total position data of each of the spliced word segmentation results in the feature sequence; a total classification unit configured to perform classification processing on each of the spliced word segmentation results in the feature sequence to obtain total classification data; and the obtaining unit further comprises a total position mapping module configured to perform vector mapping on the total position data to adapt to the autoregressive model to obtain a total position vector; a total classification mapping module configured to perform vector mapping on the total classification data to adapt to the autoregressive model to obtain a total classification vector; and the obtained multi-modal input data adapted to the autoregressive model further comprises the total position vector and the total classification vector.

[0018] In some embodiments, the at least two different modalities of data include: text data to be processed and image data; the extraction unit includes: a word segmentation module configured to perform word segmentation processing on the text data to be processed; a concatenation module configured to concatenate all word segmentation results to obtain a word segmentation sequence; an extraction module configured to perform feature extraction on the image data to obtain image features; and a superposition module configured to superimpose the word segmentation sequence and the image features to obtain a feature sequence; and the obtaining unit includes: a word segmentation mapping module configured to perform vector mapping on the word segmentation sequence in the feature sequence to obtain a data feature vector of the word segmentation sequence, the vector mapping being adapted to the self-recurrent model; an image mapping module configured to perform vector mapping on the image features in the feature sequence to obtain an image feature vector, the vector mapping being of the same dimension as an input layer of the self-recurrent model; and a word-image obtaining module configured to add the data feature vector and the image feature vector to obtain multi-modal input data adapted to the self-recurrent model.

[0019] In some embodiments, the apparatus further includes: a word positioning unit configured to determine word position data of each word in the word segmentation sequence; and a word classification unit configured to perform classification processing on each word in the word segmentation sequence to obtain word classification data; and the obtaining unit further includes: a word position mapping module configured to perform vector mapping on the word position data to obtain a word position vector, the vector mapping being adapted to the self-recurrent model; and a word classification mapping module configured to perform vector mapping on the word classification data to obtain a word classification vector, the vector mapping being adapted to the self-recurrent model; and the obtained multi-modal input data adapted to the self-recurrent model further includes the word position vector and the word classification vector.

[0020] In a third aspect, the embodiments of the present disclosure provide an electronic device, including: one or more processors; a storage device having one or more programs stored thereon; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the implementations of the first aspect.

[0021] In a fourth aspect, the embodiments of the present disclosure provide a computer readable medium having a computer program stored thereon, the program being executed by a processor to implement the method described in any of the implementations of the first aspect.

[0022] In a fifth aspect, the embodiments of the present disclosure provide a computer program product including a computer program, the computer program being executed by a processor to implement the method described in any of the implementations of the first aspect.

[0023] The model data processing method and device provided by the embodiment of the present disclosure first acquire data of at least two different modalities in a to-be-processed data set; secondly, the data of the at least two different modalities are spliced and / or superimposed after feature extraction to obtain a feature sequence; then, the feature sequence is subjected to model mapping processing to obtain multi-modal input data compatible with an autoregressive model; finally, the multi-modal input data is input into the autoregressive model to obtain single-modal results output by the autoregressive model. In this way, by inputting the multi-modal input data obtained by processing the at least two modalities of data in the to-be-processed data set into the autoregressive model, the autoregressive model can not only process single-modal data but also process multi-modal data, and the single-modal results output by the entire model are more accurate, thereby improving the model data processing capability. BRIEF DESCRIPTION OF DRAWINGS

[0024] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments made with reference to the accompanying drawings:

[0025] Figure 1 is an exemplary system architecture diagram to which an embodiment of the present disclosure can be applied;

[0026] Figure 2 is a flowchart of one embodiment of a model data processing method according to the present disclosure;

[0027] Figure 3 is a flowchart of another embodiment of a model data processing method according to the present disclosure;

[0028] Figure 4 is a data structure diagram in one practical scenario of the present disclosure;

[0029] Figure 5 is a structural schematic diagram of an embodiment of a model data processing device according to the present disclosure;

[0030] Figure 6 is a structural schematic diagram of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that, for the sake of description, only the parts related to the application are shown in the drawings.

[0032] It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings and embodiments.

[0033] Figure 1 An exemplary system architecture 100 to which the model data processing method of the present disclosure can be applied is shown.

[0034] As shown in Figure 1 The system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium to provide a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, and can generally include a wireless communication link, etc.

[0035] The terminal devices 101, 102, 103 interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as instant messaging tools, email clients, etc.

[0036] The terminal devices 101, 102, 103 can be hardware or software; when the terminal devices 101, 102, 103 are hardware, they can be user devices with communication and control functions, and the above user devices can communicate with the server 105. When the terminal devices 101, 102, 103 are software, they can be installed in the above user devices; the terminal devices 101, 102, 103 can be implemented as multiple software or software modules (such as software or software modules for providing distributed services), or as a single software or software module. No specific limitation is made herein.

[0037] The server 105 can be a server providing various services, such as a background server for model data processing to support a data set to be processed in a client question-answering system on the terminal devices 101, 102, 103. The background server can analyze and process relevant information to be input into an autoregressive model in the network, and feed back the processing result (such as a single-modal result output by the autoregressive model) to the terminal device.

[0038] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (such as software or software modules for providing distributed services), or as a single software or software module. No specific limitation is made herein.

[0039] It should be noted that the model data processing method provided by the embodiments of the present disclosure is generally executed by the server 105.

[0040] It should be understood that Figure 1The number of terminal devices, networks and servers in the system is only illustrative. Any number of terminal devices, networks and servers can be provided according to implementation needs.

[0041] As Figure 2 shown in FIG. 2, an embodiment of a model data processing method according to the present disclosure is shown, which includes the following steps:

[0042] Step 201, obtaining data of at least two different modalities in a to-be-processed data set.

[0043] In this embodiment, the to-be-processed data set includes data of multiple different modalities, and the data of multiple different modalities can be single-modality data with multiple different sources or single-modality data with different forms. In practice, single-modality data with multiple different sources refers to description data of the same thing collected by different sensors, such as audio data and image data of the same animal at the same time in video retrieval, wherein the audio data and image data are two different single-modality data. Single-modality data with different forms refers to data whose representation content can be known through the data, for example, in a task-oriented dialogue communication process, image data, text data, etc. sent by a user to a dialogue robot.

[0044] In this embodiment, the to-be-processed data set can include a preset data set for implementing construction or training of an autoregressive model; the to-be-processed data set can also include a collection of all single-modality data that needs to be processed by the autoregressive model after the autoregressive model is trained.

[0045] The autoregressive model is a statistical method for processing time series, which uses the performance of the same variable in previous periods to predict the performance of the variable in the current period, and assumes that they are in a linear relationship. For example, an autoregressive language model predicts the next word that may follow according to the context, or vice versa, predicts the previous word according to the context.

[0046] In actual application scenarios, such as a task-oriented dialogue communication process, some knowledge base is often needed to assist in effectively answering questions. For example, in an online customer service system of a certain mall, customers often need to inquire about some attributes of goods. Dialogue generation driven by knowledge graph information makes the content more accurate and the information contained more sufficient, which can meet the needs of customers in actual application scenarios. For the above scenario, the data of at least two different modalities can include any two or more of to-be-processed text data, image data and knowledge graph data.

[0047] In this embodiment, the execution subject of the model data processing method (such as the server 105 shown in FIG. 1) can obtain the to-be-processed data set through various means, such as real-time collection from a user terminal (such as the terminal device 101 shown in FIG. 1). Figure 1 In this embodiment, the execution subject of the model data processing method (such as the server 105 shown in FIG. 1) can obtain the to-be-processed data set through various means, such as real-time collection from a user terminal (such as the terminal device 101 shown in FIG. 1).Figure 1 The terminal device 101, 102, 103 shown collects a to-be-processed data set and extracts data of at least two different modalities from the to-be-processed data set. Alternatively, the to-be-processed data set is obtained from a local memory, and data of at least two different modalities is extracted from the to-be-processed data set.

[0048] In step 202, the data of at least two different modalities is spliced and / or superimposed after feature extraction to obtain a feature sequence.

[0049] In this embodiment, since the obtained data of at least two different modalities has respective standards, the features extracted can be spliced after feature extraction for each single-modality data that can have a unified form after data conversion to obtain a spliced sequence, for example, knowledge graph data in a knowledge base. The knowledge graph data exists in the form of a head entity, a relationship entity, and a tail entity triple. In the data processing process, the literal content corresponding to the head entity, the relationship entity, and the tail entity in the triple involved in the dialogue can be extracted to obtain triple text data. The triple text data and the text data have a unified form. Therefore, the extracted to-be-processed text data and the triple text data can be spliced.

[0050] Correspondingly, for each single-modality data that does not have a unified data form after data conversion, the features extracted can be superimposed after feature extraction. The superimposition in this embodiment is only a kind of information superposition mode, and the two still exist independently in different feature vector spaces. For example, image data and to-be-processed text data cannot be converted to a unified data form, but since they represent the same thing, the description of the text data on the same thing can be enriched through the image data form.

[0051] In some optional implementation modes of this embodiment, the data of at least two different modalities includes to-be-processed text data and knowledge graph data. The above-mentioned splicing and / or superimposition of the data of at least two different modalities after feature extraction to obtain a feature sequence includes: collecting triple text data corresponding to the knowledge graph data; performing word segmentation processing on the triple text data and the to-be-processed text data respectively, and splicing all word segmentation results to obtain a feature sequence.

[0052] Specifically, the word segmentation processing process of the triple text data and the to-be-processed text data is described in detail in the embodiment shown in the following. Figure 4 The embodiment shown is described in detail.

[0053] In this optional implementation mode, when the data of at least two different modalities includes to-be-processed text data and knowledge graph data, splicing the features extracted from the two can effectively enrich the data information.

[0054] Optionally, for the optional implementation manner above, the data of at least two different modalities can further include image data. The feature extraction on the data of at least two different modalities and the splicing and / or superimposition after the feature extraction can include: performing feature extraction on the image data to obtain image features, and the obtained feature sequence is further superimposed with the image features.

[0055] In this optional implementation manner, the image features can be superimposed after the splicing of all the word segmentation results, to obtain the feature sequence, thereby enriching the data information in the form other than the knowledge graph and the text.

[0056] In some optional implementation manners of this embodiment, the data of at least two different modalities include: to-be-processed text data and image data; the feature extraction on the data of at least two different modalities and the splicing and / or superimposition after the feature extraction can include:

[0057] performing word segmentation processing on the to-be-processed text data, and splicing all the word segmentation results to obtain a word segmentation sequence; performing feature extraction on the image data to obtain image features; and superimposing the word segmentation sequence and the image features to obtain the feature sequence.

[0058] In this embodiment, since the word segmentation sequence and the image features are different data in two different dimensional spaces, the superimposition of the two is only one way of superimposition of the two, and through this superimposition way, the word segmentation sequence and the image features can be quickly separated after the feature sequence is obtained.

[0059] In this optional implementation manner, when the data of at least two different modalities include to-be-processed text data and image data, the superimposition of the features extracted from the two can effectively enrich the data information.

[0060] In step 203, the model mapping processing is performed on the feature sequence to obtain multi-modal input data compatible with the autoregressive model.

[0061] In this embodiment, the model mapping processing on the feature sequence is to convert the feature sequence into data received by the autoregressive model, that is, the dimension of the converted data is the same as the input layer dimension of the autoregressive model, so that the converted data can be input into the autoregressive model, and the converted data can also be adjusted with the change of the parameters in the training process of the autoregressive model.

[0062] In this embodiment, based on the different contents of the data of at least two different modalities, the contents of the obtained feature sequence are not the same, and therefore the contents of the multi-modal input data are also not the same.

[0063] In some optional implementations of the embodiment, when the at least two different modalities of data include: to-be-processed text data and image data, the model mapping processing on the feature sequence to obtain the multi-modality input data compatible with the autoregressive model includes:

[0064] vector mapping is performed on the word segmentation sequence in the feature sequence to obtain a data feature vector of the word segmentation sequence, vector mapping of the same dimension as the input layer of the autoregressive model is performed on the image feature in the feature sequence to obtain an image feature vector, and the data feature vector and the image feature vector are added to obtain the multi-modality input data compatible with the autoregressive model.

[0065] In the embodiment, the vector mapping of different data compatible with the autoregressive model means that the different data is converted into a vector of the same dimension as the input layer of the autoregressive model, so that the input layer of the autoregressive model can receive the converted different vector. Of course, the means for converting different forms of data into a vector of the same dimension as the input layer of the autoregressive model can be different.

[0066] In the optional implementation, when the at least two different modalities of data include: to-be-processed text data and image data, the to-be-processed text data and the image feature are respectively mapped into vectors, which enriches the diversity of the content of the multi-modality input data.

[0067] Optionally, when the at least two different modalities of data include: to-be-processed text data and knowledge graph data, the model mapping processing on the feature sequence to obtain the multi-modality input data compatible with the autoregressive model includes:

[0068] vector mapping is performed on the feature sequence to obtain a data feature vector corresponding to the feature sequence, and the data feature vector is the obtained multi-modality input data compatible with the autoregressive model. It should be noted that in the optional implementation, the feature sequence is also the spliced all word segmentation results.

[0069] In some optional implementations of the embodiment, when the at least two different modalities of data include: to-be-processed text data, knowledge graph data and image data, the model mapping processing on the feature sequence to obtain the multi-modality input data compatible with the autoregressive model includes:

[0070] vector mapping is performed on the spliced all word segmentation results in the feature sequence to obtain a data feature vector corresponding to the spliced all word segmentation results, vector mapping of the same dimension as the input layer of the autoregressive model is performed on the image feature in the feature sequence to obtain an image feature vector, and the data feature vector and the image feature vector are added to obtain the multi-modality input data compatible with the autoregressive model.

[0071] In the optional implementation, when the at least two different modal data include: to-be-processed text data, knowledge graph data and image data, the to-be-processed text data and the knowledge graph data are subjected to word segmentation processing to obtain a plurality of word segmentation results, all the word segmentation results are spliced to obtain spliced word segmentation results, and the spliced word segmentation results and image features are subjected to vector mapping respectively, so that the content of the multi-modal input data is enriched through the plurality of different modal data.

[0072] Further, each feature in the feature sequence has a position and a type feature, so that the obtained feature sequence can be subjected to position annotation and type annotation.

[0073] In some optional implementations of the embodiment, when the at least two different modal data include: to-be-processed text data and knowledge graph data, the method further includes: determining total position data of each word segmentation result in the spliced all word segmentation results in the feature sequence.

[0074] The above model mapping processing of the feature sequence obtains multi-modal input data adapted to the autoregressive model, including: performing vector mapping of the feature sequence adapted to the autoregressive model to obtain a data feature vector corresponding to the feature sequence; performing vector mapping of the total position data adapted to the autoregressive model to obtain a total position vector; and adding the data feature vector and the total position vector to obtain the multi-modal input data adapted to the autoregressive model.

[0075] In the optional implementation, based on the position sequence of each word segmentation result in the spliced all word segmentation results, position data (such as code) can be allocated to each word segmentation result, and the total position data is a set including the position data of each word segmentation result in all the word segmentation results. Through the position data, the sequence value or information represented by each word segmentation result in the spliced all word segmentation results can be determined conveniently and quickly.

[0076] In the optional implementation, by determining the total position data of the word segmentation result, each word segmentation result in the feature sequence can be effectively positioned, and the position feature of the multi-modal input data is enriched.

[0077] In some optional implementations of the embodiment, when the data of at least two different modalities includes the text data to be processed and the knowledge graph data, the method further includes: performing classification processing on each of the all segmented word results spliced in the feature sequence to obtain total classification data. The model mapping processing on the feature sequence to obtain the multi-modal input data compatible with the autoregressive model includes: performing vector mapping on the feature sequence compatible with the autoregressive model to obtain a data feature vector corresponding to the feature sequence; performing vector mapping on the total classification data compatible with the autoregressive model to obtain a total classification vector; and adding the data feature vector and the total classification vector to obtain the multi-modal input data compatible with the autoregressive model.

[0078] In the optional implementation, based on the type of each of the all segmented word results, classification data (such as encoding) can be assigned to each of the segmented word results, and the total classification data is a set including the classification data of each of the all segmented word results. The type of each of the segmented word results in the all segmented word results can be determined conveniently and quickly through the classification data.

[0079] In the optional implementation, the type to which each of the segmented word results in the feature sequence belongs can be determined effectively through the total classification data of the segmented word results, and the type characteristics of the multi-modal input data are enriched.

[0080] Optionally, when the data of at least two different modalities includes the text data to be processed and the knowledge graph data, the method further includes: determining total position data of each of the all segmented word results spliced in the feature sequence; and performing classification processing on each of the all segmented word results spliced in the feature sequence to obtain total classification data. The model mapping processing on the feature sequence to obtain the multi-modal input data compatible with the autoregressive model includes: performing vector mapping on the feature sequence compatible with the autoregressive model to obtain a data feature vector corresponding to the feature sequence; performing vector mapping on the total position data compatible with the autoregressive model to obtain a total position vector; performing vector mapping on the total classification data compatible with the autoregressive model to obtain a total classification vector; and adding the data feature vector, the total position vector, and the total classification vector to obtain the multi-modal input data compatible with the autoregressive model.

[0081] In some optional implementations of the embodiment, when the data of at least two different modalities includes the text data to be processed and the image data, the method further includes: determining word position data of each word in the segmented word sequence; performing classification processing on each word in the segmented word sequence to obtain word classification data; and performing model mapping processing on the feature sequence to obtain the multi-modal input data compatible with the autoregressive model, including:

[0082] The word segmentation sequence in the feature sequence is subjected to vector mapping adapted to the autoregressive model to obtain a data feature vector of the word segmentation sequence; the image feature in the feature sequence is subjected to vector mapping of the same dimension as the input layer of the autoregressive model to obtain an image feature vector; the word position data is subjected to vector mapping adapted to the autoregressive model to obtain a word position vector; the word classification data is subjected to vector mapping adapted to the autoregressive model to obtain a word classification vector; and the data feature vector, the word position vector, the word classification vector, and the image feature vector are added to obtain multi-modal input data adapted to the autoregressive model.

[0083] In this optional implementation, by determining the word position data and the classification data of each word in the word segmentation sequence, the type to which each word in the word segmentation sequence belongs and the position of each word can be effectively determined, thereby enriching the position features and type features of the multi-modal input data.

[0084] In step 204, the multi-modal input data is input into the autoregressive model to obtain a single-modal result output by the autoregressive model.

[0085] In this embodiment, the autoregressive model can be an autoregressive model whose output data is single-modal data, such as a UniLM (Unified Language Model Pre-training for Natural Language Understanding and Generation) model, a GPT (Generative Pre-Training) model, or the like. The autoregressive model can be a newly constructed model (such as a model whose input is multi-modal data and whose output is single-modal data).

[0086] Alternatively, the autoregressive model can also be a pre-trained autoregressive model whose input and output are both single-modal data. For example, the autoregressive model is a language model whose input and output are both text data. For the language model, the applicable data set to be processed can be dialog records or articles, chapters, or paragraphs of data.

[0087] After the multi-modal input data is input into the autoregressive model, the autoregressive model is subjected to model training to obtain a trained autoregressive model.

[0088] For a question-and-answer autoregressive model, a user's question (with multi-modal data such as pictures and knowledge bases) can be input into the trained autoregressive model to obtain an answer to the user's question.

[0089] In an actual application scenario, the autoregressive model can adopt a Transformer autoregressive model structure, such as a UniLM or DialoGPT model. Through the method of the present disclosure, the autoregressive model can process not only traditional text data but also multi-modal data containing pictures, knowledge graphs, and text, so that the entire autoregressive model can have a better processing effect on multi-modal data sets to be processed.

[0090] The model data processing method provided by the embodiments of the present disclosure first acquires data of at least two different modalities in a data set to be processed; second, performs feature extraction, splicing, and / or superimposition on the data of the at least two different modalities to obtain a feature sequence; then, performs model mapping processing on the feature sequence to obtain multi-modal input data compatible with an autoregressive model; and finally, inputs the multi-modal input data into the autoregressive model to obtain a single-modality result output by the autoregressive model. In this way, by inputting the multi-modal input data obtained by processing the data of at least two modalities in the data set to be processed into the autoregressive model, the autoregressive model can not only process single-modality data but also process multi-modal data, and the single-modality result output by the entire model is more accurate, thereby improving the model data processing capability.

[0091] In another embodiment of the present disclosure, the data of the two different modalities include: text data to be processed, knowledge graph data, and image data, such as Figure 3 FIG. 3 shows a flow 300 of another embodiment of the model data processing method according to the present disclosure, which includes the following steps:

[0092] In step 301, text data to be processed, knowledge graph data, and image data in a data set to be processed are acquired.

[0093] As shown in FIG. 3, the data set to be processed can be dialogue data obtained by intercepting and processing from a dialogue system (for example, a customer service system of a certain shopping mall), which includes text data, image data, and knowledge graph data of dialogue content obtained from a pre-built knowledge graph. It should be noted that the text corresponding to the text data to be processed is composed of at least one sentence, and each sentence can be segmented by symbols. Figure 4

[0094] Figure 4 In the example shown in FIG. 3, the text data to be processed includes a question and an answer, the question is: Are there any board shoes?, in which board shoes in the question is a picture sent by a user at hand, and the answer is: Are these board shoes?, in which board shoes in the answer is another picture sent by an execution subject.

[0095] In step 302, triple text data corresponding to the knowledge graph data is collected.​

[0096] In this embodiment, the knowledge graph data is stored in the form of triples of head entity, relation entity and tail entity. After the knowledge graph data is extracted, the head entity, the relation entity and the tail entity in the knowledge graph data can be easily obtained.

[0097] Figure 4 In this embodiment, the knowledge graph data obtained by searching the knowledge graph includes: board shoes-material-canvas; board shoes-closing-lace; board shoes-upper-low. The triples of the triple text data are respectively: board shoes, material, canvas; board shoes, closing, lace; board shoes, upper, low.

[0098] In step 303, the triple text data and the to-be-processed text data are respectively subjected to word segmentation processing.

[0099] In this embodiment, the to-be-processed text corresponding to the to-be-processed text data and the triple text corresponding to the triple text data can be respectively subjected to word segmentation by using a traditional word segmentation tool.

[0100] Specifically, after the triple text is subjected to word segmentation, the following knowledge expressions subjected to word segmentation can be obtained, such as S11, S12, …, S1n, P11, P12, …, P1n, O11, O12, …, O1n, S21, S22, …, S2n, P21, P22, …, P2n, O21, O22, …, O2n, …, Sn1, Sn2, …, Snm, Pn1, Pn2, …, Pnm, On1, On2, …, Onm, where Snm represents the mth token in the head entity in the nth (n>0, m>0) triple, Pnm represents the mth token in the relation entity in the nth triple, and Onm represents the mth token in the tail entity in the nth triple.

[0101] Specifically, in this embodiment, each sentence in the context text corresponding to the to-be-processed text data is subjected to word segmentation to obtain Wn1, Wn2, Wn3, …, Wnm, where Wnm represents the mth token after word segmentation in the nth (n>0, m>0) sentence. In actual word segmentation, text symbols with special meanings, such as URL (Uniform Resource Location, Uniform Resource Locator) links, expression symbol text expressions and other special meaning texts, can be replaced by special tokens, so that the semantics of the sentence can be maintained to the greatest extent.

[0102] In step 304, all the word segmentation results are spliced to obtain all the spliced word segmentation results.

[0103] Referring to Figure 4As shown, a special symbol [CLS] is used as the beginning of the sentence, and the above knowledge graph data labels (referred to as knowledge labels) and the tokenization labels of each sentence in the to-be-processed text are sequentially spliced. Among them, special symbols [SEP] need to be added between the knowledge graph data and the to-be-processed text, and between the sentences to represent the interval of the labels of different contents.

[0104] Step 305, feature extraction is performed on the image data to obtain image features.

[0105] In this embodiment, for the pictures appearing in the to-be-processed data set, a pre-trained feature collection model (for example, ResNet50) can be used to perform feature extraction on the pictures.

[0106] Step 306, superimpose all the spliced tokenization results and image features to obtain a feature sequence.

[0107] In this embodiment, the superimposition of all the spliced tokenization results and image features is to facilitate obtaining a feature sequence with a unified whole, so as to perform subsequent processing on the entire feature sequence.

[0108] Step 307, determine the total position data of each tokenization result in the spliced tokenization results in the feature sequence.

[0109] In this embodiment, the total position data is assigned by position coding to each tokenization result in the feature sequence, as shown in Figure 4 The [CLS] position data at the beginning of the sentence is 0, the position data of all knowledge labels is 1, and the position data of the remaining non-knowledge labels is sequentially increased from 2, as shown in Figure 4 The total position data obtained in the above example is: 0 1 1 1 1 1 11 1 1 23 4 5 6 7 8 9 10 11 12.

[0110] Step 308, classify each tokenization result in the spliced tokenization results in the feature sequence to obtain total classification data.

[0111] In this embodiment, in order to distinguish different groups in the spliced labels, refer to Figure 4 As shown, several classification categories are specially introduced to represent different labels, the [CLS] category represents that the label is the beginning of the sentence, the [SEP] category represents that the label is used to separate knowledge and sentences, or sentences and sentences, the [SUB] category represents that the label belongs to the head entity in the knowledge triple, the [PRE] category represents that the label belongs to the relationship entity in the knowledge triple, and the [OBJ] category represents that the label belongs to the tail entity in the knowledge triple. The [SP1] category represents that the label belongs to the content spoken by role 1 in the dialogue, and the [SP2] category represents that the label belongs to the content spoken by role 2 in the dialogue.

[0112] Step 309, vector mapping of all segmented results in the feature sequence is performed to adapt to the autoregressive model, and a data feature vector corresponding to all segmented results is obtained.

[0113] The vector mapping in this embodiment refers to mapping a feature data to a feature vector adapted to the autoregressive model.

[0114] In this embodiment, the knowledge markers (Snm, Pnm, Onm) generated in step 303, the text marker Wnm, and [CLS] and [SEP] generated in step 304 are consistent in semantics, so the same vocabulary and corresponding vector mapping are used to obtain a data feature vector, and the dimension of the obtained data feature vector is the same as the input layer dimension of the autoregressive model.

[0115] In practice, the data feature vector can be obtained by the tf.nn.embedding_lookup(params, ids) function, which mainly selects the element corresponding to the index in the tensor, where params can be a tensor, an array, etc., and ids is the corresponding index value, which is also the vector corresponding to the autoregressive model.

[0116] Step 310, vector mapping of the image features in the feature sequence is performed to adapt to the input layer of the autoregressive model, and an image feature vector is obtained.

[0117] In this embodiment, for the image features in the feature sequence, a linear regression model (Linear) can be used to process the image features to map them to a specified dimension, which is the same as the input layer dimension of the autoregressive model.

[0118] It should be noted that the autoregressive model can be a language model (such as a question and answer model), and the input layer dimension of the autoregressive model is determined by the embedding (embedding layer) dimension, which is a hyperparameter, for example, it can be 300 or 512 dimensions.

[0119] Step 311, vector mapping of the total position data is performed to adapt to the autoregressive model, and a total position vector is obtained.

[0120] In this embodiment, the total position vector can be obtained by using the traditional Transformer triangular function position encoding method, so as to map the corresponding total position data to a continuous space of a specified dimension.

[0121] Step 312, vector mapping of the total classification data is performed to adapt to the autoregressive model, and a total classification vector is obtained.

[0122] In this embodiment, the seven different category labels obtained in step 308 are mapped to a continuous space of a specified dimension using a vocabulary and corresponding vector mapping.

[0123] In step 313, the data feature vector, the image feature vector, the total position vector, and the total classification vector are added to obtain multi-modal input data suitable for the autoregressive model.

[0124] In this embodiment, it can be known from the above processing that since the data feature vector, the image feature vector, the total position vector, and the total classification vector are all mapped to a specified dimension, adding the four vectors can obtain the input data of the autoregressive model.

[0125] In step 314, the multi-modal input data is input into the autoregressive model to obtain a single-modal result output by the autoregressive model.

[0126] In this embodiment, the autoregressive model can be an autoregressive language model based on a Transformer (a model structure composed of an encoder and a decoder). As shown in FIG. 6, the dialogue context text, the picture, and the knowledge information are processed to obtain multi-modal input data, which is input into the autoregressive language model. After training, the model can utilize multi-modal information and knowledge information when generating a response to a user question, so that the generated response is more fluent and accurate. Figure 4

[0127] In this embodiment, by effectively fusing the information of knowledge graph data, image data, and text data, the dialogue context information is fully expressed, and the model based on the autoregressive structure is trained end-to-end to generate a fluent and information-rich dialogue response.

[0128] ​The model data processing method provided in the embodiment can cover data of different modalities such as a knowledge graph, a text, and an image, and can effectively locate and classify data of different modalities of knowledge, thereby improving the accuracy of the multi-modal input data and making the single-modal result output by the autoregressive model more fluent and accurate.

[0129] Further referring to Figure 5 , as an implementation of the method shown in the above figures, the disclosure provides an embodiment of a model data processing device, which corresponds to the method embodiment shown in Figure 2 , and the device can be specifically applied to various electronic devices.

[0130] As shown in Figure 5 , the embodiment of the disclosure provides a model data processing device 500, which comprises an acquisition unit 501, an extraction unit 502, a obtaining unit 503, and a processing unit 504. The acquisition unit 501 can be configured to acquire at least two different modalities of data in a data set to be processed. The extraction unit 502 can be configured to splice and / or superimpose the at least two different modalities of data after feature extraction to obtain a feature sequence. The obtaining unit 503 can be configured to perform model mapping processing on the feature sequence to obtain multi-modal input data adapted to an autoregressive model. The processing unit 504 can be configured to input the multi-modal input data into the autoregressive model to obtain a single-modal result output by the autoregressive model.

[0131] In the embodiment, the specific processing of the acquisition unit 501, the extraction unit 502, the obtaining unit 503, and the processing unit 504 in the model data processing device 500 and the technical effects brought by the specific processing can be respectively referred to steps 201, 202, 203, and 204 in the corresponding embodiment. Figure 2 corresponding embodiment.

[0132] In some embodiments, the at least two different modalities of data described above include: the text data to be processed and the knowledge graph data; and the extraction unit 502 described above includes: a collection module (not shown in the figure), a total word segmentation module (not shown in the figure), and a total concatenation module (not shown in the figure). The collection module can be configured to collect the triple text data corresponding to the knowledge graph data. The total word segmentation module can be configured to perform word segmentation processing on the triple text data and the text data to be processed respectively. The total concatenation module can be configured to concatenate all the word segmentation results to obtain a feature sequence.

[0133] In some embodiments, the device 500 described above further includes a total positioning unit (not shown in the figure), which can be configured to determine total position data of each word segmentation result in all the word segmentation results concatenated in the feature sequence. The obtaining unit 503 described above includes a data mapping module (not shown in the figure), a total position mapping module (not shown in the figure), and a digit obtaining module (not shown in the figure). The data mapping module can be configured to perform vector mapping on the feature sequence to adapt to the autoregressive model, to obtain a data feature vector corresponding to the feature sequence. The total position mapping module can be configured to perform vector mapping on the total position data to adapt to the autoregressive model, to obtain a total position vector. The digit obtaining module can be configured to add the data feature vector and the total position vector to obtain a multi-modal input data adapted to the autoregressive model.

[0134] In some embodiments, the device 500 described above further includes a total classification unit (not shown in the figure), which can be configured to perform classification processing on each word segmentation result in all the word segmentation results concatenated in the feature sequence, to obtain total classification data. The obtaining unit 503 described above includes a data mapping module (not shown in the figure), a total classification mapping module (not shown in the figure), and a digit class obtaining module (not shown in the figure). The data mapping module can be configured to perform vector mapping on the feature sequence to adapt to the autoregressive model, to obtain a data feature vector corresponding to the feature sequence. The total classification mapping module can be configured to perform vector mapping on the total classification data to adapt to the autoregressive model, to obtain a total classification vector. The digit class obtaining module can be configured to add the data feature vector and the total classification vector to obtain a multi-modal input data adapted to the autoregressive model.

[0135] In some embodiments, the at least two different modalities of data described above further include: image data. The extraction unit 502 described above is further configured to perform feature extraction on the image data to obtain image features. In embodiments, the obtained feature sequence further superimposes the image features. The obtaining unit 503 described above includes: a data mapping module (not shown in the figure), an image mapping module (not shown in the figure), and an image mapping module (not shown in the figure). The data mapping module can be configured to perform vector mapping on all word segmentation results spliced in the feature sequence to adapt to the self-recurrent model to obtain data feature vectors corresponding to all word segmentation results spliced. The image mapping module can be configured to perform vector mapping on the image features in the feature sequence to the same dimension as the input layer of the self-recurrent model to obtain image feature vectors. The data and image obtaining module can be configured to add the data feature vectors and the image feature vectors to obtain multi-modal input data adapted to the self-recurrent model.

[0136] In some embodiments, the device 500 described above further includes: a total positioning unit (not shown in the figure) and a total classification unit (not shown in the figure). The total positioning unit can be configured to determine the total position data of each word segmentation result in all word segmentation results spliced in the feature sequence. The total classification unit can be configured to perform classification processing on each word segmentation result in all word segmentation results spliced in the feature sequence to obtain total classification data. The obtaining unit 503 described above further includes: a total position mapping module (not shown in the figure) and a total classification mapping module (not shown in the figure). The total position mapping module is configured to perform vector mapping on the total position data to adapt to the self-recurrent model to obtain a total position vector. The total classification mapping module is configured to perform vector mapping on the total classification data to adapt to the self-recurrent model to obtain a total classification vector. In this embodiment, the obtained multi-modal input data adapted to the self-recurrent model further includes the total position vector and the total classification vector.

[0137] In some embodiments, the at least two different modalities of data include: text data to be processed and image data. The extraction unit 502 includes: a word segmentation to be performed module (not shown in the figure), a concatenation to be performed module (not shown in the figure), an extraction module (not shown in the figure), and a superposition module (not shown in the figure). The word segmentation to be performed module can be configured to perform word segmentation processing on the text data to be processed. The concatenation to be performed module can be configured to concatenate all word segmentation results to obtain a word segmentation sequence. The extraction module can be configured to perform feature extraction on the image data to obtain image features. The superposition module can be configured to superimpose the word segmentation sequence and the image features to obtain a feature sequence. The obtaining unit 503 includes: a word segmentation mapping module (not shown in the figure), an image mapping module (not shown in the figure), and a word-image obtaining module (not shown in the figure). The word segmentation mapping module can be configured to perform vector mapping on the word segmentation sequence in the feature sequence to obtain a data feature vector of the word segmentation sequence, which is suitable for the autoregressive model. The image mapping module can be configured to perform vector mapping on the image features in the feature sequence to obtain an image feature vector, which has the same dimension as the input layer of the autoregressive model. The word-image obtaining module can be configured to add the data feature vector and the image feature vector to obtain multi-modal input data suitable for the autoregressive model.

[0138] In some embodiments, the device 500 further includes: a word positioning unit (not shown in the figure) and a word classification unit (not shown in the figure). The word positioning unit is configured to determine word position data of each word in the word segmentation sequence. The word classification unit is configured to perform classification processing on each word in the word segmentation sequence to obtain word classification data. The obtaining unit 503 further includes: a word position mapping module (not shown in the figure) and a word classification mapping module (not shown in the figure). The word position mapping module can be configured to perform vector mapping on the word position data to obtain a word position vector, which is suitable for the autoregressive model. The word classification mapping module can be configured to perform vector mapping on the word classification data to obtain a word classification vector, which is suitable for the autoregressive model. The multi-modal input data suitable for the autoregressive model obtained in this embodiment further includes the word position vector and the word classification vector.

[0139] Reference is made below to Figure 6 which shows a structural schematic diagram of an electronic device 600 suitable for implementing embodiments of the present disclosure.

[0140] As Figure 6As shown, the electronic device 600 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or loaded into a random access memory (RAM) 603 from a storage device 608. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0141] In general, the following devices can be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 can allow the electronic device 600 to communicate with other devices wirelessly or wired to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it is understood that all of the shown devices are not required to be implemented or present. More or fewer devices can alternatively be implemented or present. Figure 6 Each block shown in the flowcharts can represent a device or multiple devices as needed.

[0142] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product including a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-described functions defined in the methods of embodiments of the present disclosure are performed.

[0143] It should be noted that the computer-readable medium of the embodiments of the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the embodiments of the present disclosure, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, an RF (Radio Frequency) or the like, or any suitable combination of the above.

[0144] The computer-readable medium described above can be contained in the server described above; or can exist separately and not be assembled into the server. The computer-readable medium described above carries one or more programs, which, when executed by the server, cause the server to: acquire data of at least two different modalities in a data set to be processed; perform feature extraction, splicing and / or superimposition on the data of at least two different modalities to obtain a feature sequence; perform model mapping processing on the feature sequence to obtain multi-modal input data adapted to an autoregressive model; input the multi-modal input data into the autoregressive model to obtain a single-modal result output by the autoregressive model.

[0145] Computer program code for carrying out operations of embodiments of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0146] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0147] The units described in the embodiments of the present disclosure can be implemented by hardware, software, or a combination of hardware and software. The units described can be arranged in a processor, for example, can be described as: a processor including an obtaining unit, an extracting unit, a deriving unit, and a processing unit. In some cases, the names of the units do not constitute a limitation on the units themselves, for example, the obtaining unit can also be described as a unit configured to obtain data of at least two different modalities in the data set to be processed.

[0148] The above description is merely that of the preferred embodiments of the present disclosure and a description of the technical principles of the present disclosure. It should be understood by those skilled in the art that the inventive scope of the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or equivalent features without departing from the above inventive concept. For example, the technical solutions formed by the mutual replacement of the above features and the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) form the technical solutions.

Claims

1. A model data processing method, the method comprising: acquiring data of at least two different modalities in a data set to be processed; performing feature extraction, splicing and / or superimposition on the data of the at least two different modalities to obtain a feature sequence; performing model mapping processing on the feature sequence to obtain multi-modal input data compatible with an autoregressive model, comprising: performing vector mapping on the feature sequence compatible with the autoregressive model to obtain a data feature vector corresponding to the feature sequence; performing vector mapping on total classification data compatible with the autoregressive model to obtain a total classification vector; and adding the data feature vector and the total classification vector to obtain multi-modal input data compatible with the autoregressive model; the total classification data is obtained by classifying each word segmentation result in all word segmentation results spliced in the feature sequence; inputting the multi-modal input data into the autoregressive model to obtain a single-modal result output by the autoregressive model.

2. The method of claim 1, wherein, The data of the at least two different modalities comprises text data to be processed and knowledge graph data; The feature extraction, splicing and / or superimposition on the data of the at least two different modalities to obtain a feature sequence comprises: collecting triadic text data corresponding to the knowledge graph data; performing word segmentation processing on the triadic text data and the text data to be processed respectively, and splicing all word segmentation results to obtain a feature sequence.

3. The method of claim 2, further comprising: determining total position data of each word segmentation result in all word segmentation results spliced in the feature sequence; The model mapping processing on the feature sequence to obtain multi-modal input data compatible with an autoregressive model comprises: performing vector mapping on the feature sequence compatible with the autoregressive model to obtain a data feature vector corresponding to the feature sequence; performing vector mapping on the total position data compatible with the autoregressive model to obtain a total position vector; adding the data feature vector and the total position vector to obtain multi-modal input data compatible with the autoregressive model.

4. The method of claim 2, wherein, The data of the at least two different modalities further comprises image data; The feature extraction, splicing and / or superimposition on the data of the at least two different modalities to obtain a feature sequence further comprises: performing feature extraction on the image data to obtain image features; the obtained feature sequence is further superimposed with the image features; The model mapping processing on the feature sequence to obtain multi-modal input data compatible with an autoregressive model comprises: performing vector mapping on all word segmentation results spliced in the feature sequence compatible with the autoregressive model to obtain a data feature vector corresponding to all word segmentation results spliced; performing vector mapping on the image features in the feature sequence to obtain an image feature vector with the same dimension as the input layer of the autoregressive model; adding the data feature vector and the image feature vector to obtain multi-modal input data compatible with the autoregressive model.

5. The method of claim 4, further comprising: determining total position data of each word segmentation result in all word segmentation results spliced in the feature sequence; The total classification data is obtained by performing classification processing on each of all the segmented results of the feature sequence; The model mapping processing on the feature sequence to obtain the multi-modal input data compatible with the autoregressive model further includes: The total position vector is obtained by performing vector mapping on the total position data compatible with the autoregressive model; The total classification vector is obtained by performing vector mapping on the total classification data compatible with the autoregressive model; The obtained multi-modal input data compatible with the autoregressive model further includes the total position vector and the total classification vector.

6. The method of claim 1, wherein, The at least two different modal data includes: text data to be processed and image data; The feature sequence is obtained by performing feature extraction, splicing and / or superimposition on the at least two different modal data, including: The segmented sequence is obtained by performing segmentation processing on the text data to be processed and splicing all the segmented results; The image feature is obtained by performing feature extraction on the image data; The feature sequence is obtained by superimposing the segmented sequence and the image feature; The model mapping processing on the feature sequence to obtain the multi-modal input data compatible with the autoregressive model includes: The data feature vector of the segmented sequence in the feature sequence is obtained by performing vector mapping compatible with the autoregressive model; The image feature vector is obtained by performing vector mapping of the same dimension as the input layer of the autoregressive model on the image feature in the feature sequence; The data feature vector and the image feature vector are added to obtain the multi-modal input data compatible with the autoregressive model.

7. The method of claim 6, further comprising: determining word position data of each word in the segmented sequence; performing classification processing on each word in the segmented sequence to obtain word classification data; The model mapping processing on the feature sequence to obtain the multi-modal input data compatible with the autoregressive model further includes: The word position vector is obtained by performing vector mapping on the word position data compatible with the autoregressive model; The word classification vector is obtained by performing vector mapping on the word classification data compatible with the autoregressive model; The obtained multi-modal input data compatible with the autoregressive model further includes the word position vector and the word classification vector.

8. A model data processing apparatus, comprising: an acquisition unit configured to acquire at least two different modal data in a data set to be processed; an extraction unit configured to perform feature extraction, splicing and / or superimposition on the at least two different modal data to obtain a feature sequence; The obtaining unit is configured to perform model mapping processing on the feature sequence to obtain multi-modal input data adapted to the autoregressive model, including: performing vector mapping on the feature sequence to obtain a data feature vector corresponding to the feature sequence, the vector mapping being adapted to the autoregressive model; performing vector mapping on total classification data to obtain a total classification vector, the vector mapping being adapted to the autoregressive model; and adding the data feature vector and the total classification vector to obtain the multi-modal input data adapted to the autoregressive model, the total classification data being obtained by performing classification processing on each word segmentation result in all word segmentation results spliced in the feature sequence; The processing unit is configured to input the multi-modal input data into the autoregressive model to obtain a single-modal result output by the autoregressive model. 9.An electronic device, comprising: one or more processors; a memory device having stored thereon one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-7.

10. A computer readable medium having stored thereon a computer program, wherein, The program is executed by the processor to implement the method according to any one of claims 1-7. 11.A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-7.

Images