Model data processing method, device, electronic device, computer-readable medium and computer program

Abstract

The present disclosure discloses a model data processing method and apparatus, which relates to the technical field of artificial intelligence. One specific embodiment of the method includes the steps of: acquiring data of at least two different modalities in a dataset to be processed; performing feature extraction on the at least two different modalities and then obtaining a feature sequence by concatenation and / or superposition; performing a model mapping process on the feature sequence to obtain multimodal input data that fits an autoregressive model; and inputting the multimodal input data into the autoregressive model to obtain a single-modal result output from the autoregressive model. This embodiment enhances the information richness of the model input data.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This disclosure claims priority to a Chinese patent application filed on January 21, 2021, bearing application number 202110080903.5, and entitled "Model data processing method, apparatus, electronic device, and computer-readable medium," the entire text of which is incorporated herein by reference.

[0002] The present disclosure relates to the field of computer technology, specifically to the field of artificial intelligence, in particular to a method, an apparatus, an electronic device, a computer-readable medium and a computer program for processing model data To Regarding. [Background technology]

[0003] The datasets of existing dialogue systems are often unimodal datasets, and the research on corresponding dialogue generation systems is limited by the dataset, and the models studied are also unimodal. However, in actual application scenarios, the problems submitted by clients in dialogue systems are often multimodal, consisting of text information and image information. This is because, with the widespread application of smartphones, clients often take photos casually and send them to customer service, hoping to explain the problem in more detail through the photos, but there is no example in the prior art of combining a multimodal dataset with a pre-trained unimodal language model. Summary of the Invention

[0004] The embodiments of the present disclosure include a model data processing method, a model data processing device, an electronic device, a computer-readable medium, and a computer program. of provide.

[0005] In a first aspect, an embodiment of the present disclosure provides a model data processing method including the steps of acquiring at least two different modal data in a dataset to be processed, performing feature extraction on the at least two different modal data and then acquiring a feature sequence by concat and / or superposition, performing a model mapping process on the feature sequence to acquire multi-modal input data that fits an autoregressive model, and inputting the multi-modal input data into the autoregressive model to acquire a single-modal result output from the autoregressive model.

[0006] In some embodiments, the data of at least two different modalities includes text data to be processed and knowledge graph data, and the step of obtaining a feature sequence by concatenation and / or superposition after performing feature extraction on the data of at least two different modalities includes a step of collecting ternary text data corresponding to the knowledge graph data, and a step of performing word segmentation processing on the ternary text data and the text data to be processed, respectively, and concatenating all the word segmentation results to obtain a feature sequence.

[0007] In some embodiments, the above model data processing method further comprises: The method further includes a step of determining total position data for each word segmentation result out of all word segmentation results concatenated in the feature series, and the step of performing a model mapping process on the feature series to obtain multi-modal input data that conforms to the autoregressive model includes a step of performing vector mapping that conforms to the autoregressive model on the feature series to obtain a data feature vector corresponding to the feature series, a step of performing vector mapping that conforms to the autoregressive model on the total position data to obtain a total position vector, and a step of adding the data feature vector and the total position vector to obtain multi-modal input data that conforms to the autoregressive model.

[0008] In some embodiments, the above model data processing method further includes a step of performing a classification process on each word segmentation result among all word segmentation results concatenated in the feature series to obtain total classification data, and the step of performing a model mapping process on the feature series to obtain multi-modal input data that fits an autoregressive model includes a step of performing vector mapping on the feature series that fits an autoregressive model to obtain a data feature vector corresponding to the feature series, a step of performing vector mapping on the total classification data that fits an autoregressive model to obtain a total classification vector, and a step of adding the data feature vector and the total classification vector to obtain multi-modal input data that fits the autoregressive model.

[0009] In some embodiments, the data of at least two different modalities further includes image data, and the step of obtaining a feature series by concatenation and / or superposition after performing feature extraction on the data of the at least two different modalities further includes a step of performing feature extraction on the image data to obtain image features, in which the image features are further superimposed on the obtained feature series, and the step of performing a model mapping process on the feature series to obtain multi-modal input data that fits an autoregressive model includes a step of performing vector mapping that fits an autoregressive model on all concatenated word segmentation results in the feature series to obtain data feature vectors corresponding to all concatenated word segmentation results, a step of performing vector mapping of the same dimension as the input layer of the autoregressive model on the image features in the feature series to obtain an image feature vector, and a step of adding the data feature vector and the image feature vector to obtain multi-modal input data that fits the autoregressive model.

[0010] In some embodiments, the above model data processing method further includes a step of determining total position data for each word segmentation result among all word segmentation results concatenated in the feature series, and a step of performing a classification process on each word segmentation result among all word segmentation results concatenated in the feature series to obtain total classification data, and the step of performing a model mapping process on the above feature series to obtain multi-modal input data that fits an autoregressive model further includes a step of performing vector mapping that fits an autoregressive model on the total position data to obtain a total position vector, and a step of performing vector mapping that fits an autoregressive model on the total classification data to obtain a total classification vector, and the obtained multi-modal input data that fits the autoregressive model further includes the total position vector and the total classification vector.

[0011] In some embodiments, the data of at least two different modalities includes text data to be processed and image data, and the step of obtaining a feature sequence by concatenation and / or superposition after performing feature extraction on the data of at least two different modalities includes a step of performing word segmentation processing on the text data to be processed and concatenating all word segmentation results to obtain a segmented word sequence, a step of performing feature extraction on the image data to obtain image features, and a step of superimposing the segmented word sequence and the image features to obtain a feature sequence, and the step of performing model mapping processing on the feature sequence to obtain multimodal input data that fits an autoregressive model includes a step of performing vector mapping that fits an autoregressive model on a segmented word sequence in the feature sequence to obtain a data feature vector of the segmented word sequence, a step of performing vector mapping of the same dimension as the input layer of the autoregressive model on image features in the feature sequence to obtain an image feature vector, and a step of adding the data feature vector and the image feature vector to obtain multimodal input data that fits the autoregressive model.

[0012] In some embodiments, the above model data processing method further includes the steps of determining word position data for each word in the split word sequence and classifying each word in the split word sequence to obtain word classification data, and the step of performing a model mapping process on the feature sequence to obtain multi-modal input data that fits an autoregressive model further includes the steps of performing vector mapping on the word position data that fits an autoregressive model to obtain a word position vector, and performing vector mapping on the word classification data that fits an autoregressive model to obtain a word classification vector, and the obtained multi-modal input data that fits the autoregressive model further includes the word position vector and the word classification vector.

[0013] In a second aspect, an embodiment of the present disclosure provides a model data processing apparatus comprising: an acquisition unit configured to acquire at least two different modal data in a dataset to be processed; an extraction unit configured to perform feature extraction on the at least two different modal data and then acquire a feature sequence by concat and / or superposition; an acquisition unit configured to perform a model mapping operation on the feature sequence to acquire multi-modal input data that fits an autoregressive model; and a processing unit configured to input the multi-modal input data to the autoregressive model and acquire a single-modal result output from the autoregressive model.

[0014] In some embodiments, the data of at least two different modalities includes target text data and knowledge graph data, and the extraction unit includes a collection module configured to collect ternary text data corresponding to the knowledge graph data, a total word segmentation module configured to perform word segmentation processing on the ternary text data and the target text data, respectively, and a total concatenation module configured to concatenate all the word segmentation results to obtain a feature sequence.

[0015] In some embodiments, the model data processing device further comprises a total position determination unit configured to determine total position data of each word segmentation result among all word segmentation results concatenated in the feature sequence, and the acquisition unit comprises a data mapping module configured to perform vector mapping on the feature sequence that fits an autoregressive model to obtain a data feature vector corresponding to the feature sequence, a total position mapping module configured to perform vector mapping on the total position data that fits an autoregressive model to obtain a total position vector, and a data position addition module configured to add the data feature vector and the total position vector to obtain multi-modal input data that fits the autoregressive model.

[0016] In some embodiments, the model data processing device further comprises an aggregate classification unit configured to perform a classification process on each word segmentation result among all the word segmentation results concatenated in the feature sequence to obtain aggregate classification data, and the acquisition unit comprises a data mapping module configured to perform vector mapping on the feature sequence to fit an autoregressive model to obtain a data feature vector corresponding to the feature sequence, an aggregate classification mapping module configured to perform vector mapping on the aggregate classification data to fit an autoregressive model to obtain an aggregate classification vector, and a data classification addition module configured to add the data feature vector and the aggregate classification vector to obtain multi-modal input data to fit the autoregressive model.

[0017] In some embodiments, the at least two different modal data further include image data, and the extraction unit is further configured to perform feature extraction on the image data to obtain image features. The image features are further superimposed on the obtained feature sequence. The acquisition unit includes a data mapping module configured to perform vector mapping that fits an autoregressive model on all concatenated word segmentation results in the feature sequence to obtain a data feature vector corresponding to all concatenated word segmentation results, an image mapping module configured to perform vector mapping of the same dimension as an input layer of the autoregressive model on image features in the feature sequence to obtain an image feature vector, and a data image addition module configured to add the data feature vector and the image feature vector to obtain multimodal input data that fits the autoregressive model.

[0018] In some embodiments, the model data processing device further comprises an aggregate position determination unit configured to determine aggregate position data for each word segmentation result among all word segmentation results concatenated in the feature sequence, and an aggregate classification unit configured to perform a classification process on each word segmentation result among all word segmentation results concatenated in the feature sequence to obtain aggregate classification data, and the acquisition unit further comprises an aggregate position mapping module configured to perform vector mapping fitting an autoregressive model on the aggregate position data to obtain an aggregate position vector, and an aggregate classification mapping module configured to perform vector mapping fitting an autoregressive model on the aggregate classification data to obtain an aggregate classification vector, and the obtained multi-modal input data fitting the autoregressive model further includes the aggregate position vector and the aggregate classification vector.

[0019] In some embodiments, the at least two different types of data include text data to be processed and image data, and the extraction unit includes a word segmentation waiting module configured to perform word segmentation processing on the text data to be processed, a concatenation waiting module configured to concatenate all word segmentation results to obtain a segmented word sequence, an extraction module configured to perform feature extraction on the image data to obtain image features, and a superposition module configured to superimpose the segmented word sequence and the image features to obtain a feature sequence, and the acquisition unit includes a word segmentation result mapping module configured to perform vector mapping that fits an autoregressive model on the segmented word sequence in the feature sequence to obtain a data feature vector of the segmented word sequence, an image mapping module configured to perform vector mapping of the same dimension as the input layer of the autoregressive model on the image features in the feature sequence to obtain an image feature vector, and a data feature vector and an image feature vector are added to obtain multimodal input data that fits the autoregressive model. data An image addition module.

[0020] In some embodiments, the model data processing device further comprises a word position identification unit configured to determine word position data for each word in the segmented word sequence, and a word classification unit configured to classify each word of the segmented word sequence to obtain word classification data, and the acquisition unit further comprises a word position mapping module configured to perform vector mapping fitting an autoregressive model on the word position data to obtain word position vectors, and a word classification mapping module configured to perform vector mapping fitting an autoregressive model on the word classification data to obtain word classification vectors, and the obtained multi-modal input data fitting the autoregressive model further includes the word position vectors and the word classification vectors.

[0021] In a third aspect, an embodiment of the present disclosure provides an electronic device comprising one or more processors and a storage device having one or more programs stored thereon, the one or more programs, when executed by the one or more processors, causing the one or more processors to implement a method according to any of the embodiments of the first aspect.

[0022] In a fourth aspect, an embodiment of the present disclosure provides a computer readable medium having a computer program stored thereon, the computer program implementing a method according to any of the embodiments of the first aspect when executed by a processor.

[0023] A fifth aspect, a computer program product which, when executed by a processor, performs the method according to any of the embodiments of the first aspect. of provide.

[0024] The model data processing method and device provided by the embodiment of the present disclosure first obtain data of at least two different modalities of a data set to be processed. Next, the data of at least two different modalities is subjected to feature extraction, and then a feature sequence is obtained by concatenation and / or superposition. Next, a model mapping process is performed on the feature sequence to obtain multimodal input data that fits the autoregressive model. Finally, the multimodal input data is input to the autoregressive model to obtain a single-modal result output from the autoregressive model. In this way, by inputting the multimodal input data obtained by processing at least two modal data of a data set to be processed into the autoregressive model, the autoregressive model can process not only single-modal data but also multimodal data, and the single-modal result output by the entire model becomes more accurate, thereby improving the processing ability of the model data.

[0025] Other features, objects and advantages of the present disclosure will become more apparent from a reading of the detailed description of the non-limiting embodiments which is given with reference to the following drawings. [Brief description of the drawings]

[0026] [Figure 1] FIG. 1 illustrates an exemplary system architecture to which an embodiment of the present disclosure can be applied. [Diagram 2] 1 is a flow chart illustrating an embodiment of a model data processing method according to the present disclosure. [Diagram 3] 4 is a flow chart illustrating another embodiment of a model data processing method according to the present disclosure. [Figure 4] FIG. 2 is a data structure diagram in an actual situation of the present disclosure. [Diagram 5] FIG. 1 is a structural schematic diagram illustrating an embodiment of a model data processing apparatus according to the present disclosure. [Figure 6] FIG. 1 is a structural schematic diagram of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] The present disclosure will be described in more detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of illustrating the related invention, and are not intended to limit the invention. For the sake of convenience, only parts related to the invention are shown in the drawings.

[0028] In addition, the embodiments and features of the embodiments of the present disclosure can be combined with each other as long as no contradiction occurs. Hereinafter, the present disclosure will be described in detail with reference to the drawings and embodiments.

[0029] FIG. 1 illustrates an exemplary system architecture 100 in which the model data processing method according to the present disclosure can be applied.

[0030] 1, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, 103 and server 105. Network 104 may include various types of connections, typically including wireless communication links.

[0031] To send and receive messages, the terminals 101, 102, 103 communicate with a server 105 via a network 104. Various communication client applications can be installed on the terminals 101, 102, 103, such as, for example, instant communication tools, mailbox clients, etc.

[0032] The terminal devices 101, 102, 103 may be hardware or software. If the terminal devices 101, 102, 103 are hardware, they may be user devices having communication and control functions that can communicate with the server 105. If the terminal devices 101, 102, 103 are software, they may be installed in the user devices. The terminal devices 101, 102, 103 may be implemented as multiple software or software modules (e.g., software or software modules for providing distributed services), or may be implemented as a single software or software module. There is no particular limitation here.

[0033] The server 105 may be a server that provides various services, for example, a back-end server for model data processing that supports a data set to be processed in the client Q&A system on the terminal devices 101, 102, and 103. The back-end server can analyze and process related information to be input to an autoregressive model on the network, and feed back the processing results (for example, single-modal results output from the autoregressive model) to the terminal devices.

[0034] The server may be hardware or software. If the server is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. If the server is software, it may be realized as multiple software programs or software modules (e.g., software programs or software modules for providing distributed services), or as a single software program or software module. There are no particular limitations here.

[0035] It should be noted that the model data processing method provided by the embodiment of the present disclosure is typically executed by the server 105 .

[0036] It should be understood that the number of terminal devices, networks and servers in Figure 1 are merely exemplary, and the number of terminal devices, networks and servers may be arbitrarily increased or decreased according to the needs of implementation.

[0037] As shown in FIG. 2, a flow 200 of one embodiment of a model data processing method according to the present disclosure is shown, which includes the following steps.

[0038] In step 201, data of at least two different modalities in a data set to be processed is obtained.

[0039] In this embodiment, the data set to be processed includes multiple types of different modal data, and the multiple types of different modal data may be single-modal data originating from multiple different sources, or single-modal data with different representation formats. In practice, the single-modal data originating from multiple different sources refers to descriptive data of the same object collected by different sensors (e.g., audio data and image data of the same animal at the same time collected during video search), where the audio data and image data are two different types of single-modal data. The single-modal data with different representation formats refers to data that allows the display content of the data to be known through the data (e.g., task-based dialogue communication process, image data sent from a user to a dialogue robot, text data, etc.).

[0040] In this embodiment, the dataset to be processed may include a dataset that is pre-defined for building or training an autoregressive model, or the dataset to be processed may include all single-mortal datasets that need to be processed by the autoregressive model after the training of the autoregressive model is completed.

[0041] An autoregressive model is a method for statistically treating time series, which uses the expression states of the same variable in each previous period to predict the current expression state of that variable, assuming that they have a linear relationship. For example, an autoregressive language model predicts the next possible word based on the previous content, and conversely, predicts the previous word based on the following content.

[0042] In practical application scenarios, for example, during task-based dialogue communication, some knowledge base assistance is required to effectively answer questions. For example, in an online customer service system of a shopping mall, customers often need to ask about some attributes of products. Dialogue generation based on knowledge graph information enables the content to be more accurate and the amount of information contained to be more sufficient, thereby meeting the needs of clients in practical application scenarios. In the above scenarios, the at least two different modal data may include two or more of the text data to be processed, image data, and knowledge graph data.

[0043] In this embodiment, an entity executing the model data processing method (for example, the server 105 shown in FIG. 1) can acquire the data set to be processed by various means, such as collecting the data set to be processed from a user terminal (the terminal devices 101, 102, 103 shown in FIG. 1) in real time and extracting data of at least two different modalities from the data set to be processed. Alternatively, the data set to be processed is acquired from a local memory, and at least two different modal data is extracted from the data set to be processed.

[0044] In step 202, feature extraction is performed on at least two different modal data, and then a feature sequence is obtained by concatenation and / or superposition.

[0045] In this embodiment, since the obtained data of at least two different modalities each have their own criteria, for each single-modal data having a unified format after data conversion, the extracted features can be concatenated after feature extraction to obtain a concatenated series. For example, the knowledge graph data in the knowledge base exists as a triplet of a header entity, a relationship entity, and a tail entity. In the data processing process, the character contents corresponding to the header entity, the relationship entity, and the tail entity among the triplets related to the dialogue can be extracted to obtain ternary text data. Since the ternary text data and the text data have a unified format, the extracted text data to be processed and the ternary text data can be concatenated.

[0046] Therefore, for each single-modal data that does not have a unified data format after data conversion, the extracted features can be superimposed after feature extraction, but the superimposition in this embodiment is only an information superimposition method, and the two exist independently in different feature vector spaces. For example, although image data and text data to be processed cannot be converted into a unified data format, both represent the same thing, and the image data format has a higher imaginativeness and can express the same thing more richly than the text data format.

[0047] In some optional implementations of this embodiment, the data of at least two different modalities includes target text data and knowledge graph data. The step of obtaining a feature sequence by concatenation and / or superposition after performing feature extraction on the data of at least two different modalities includes the steps of collecting ternary text data corresponding to the knowledge graph data, performing word segmentation processing on the ternary text data and the target text data, respectively, and concatenating all the word segmentation results to obtain a feature sequence.

[0048] Specifically, in the embodiment shown in FIG. 4, the process of word segmentation processing of ternary text data and processing target text data will be described in detail.

[0049] In this optional embodiment, when data of at least two different modalities includes text data to be processed and knowledge graph data, the data information can be efficiently enriched by concatenating features extracted from both.

[0050] Alternatively, in the optional embodiment above, the at least two different modal data may comprise image data, and the step of obtaining a feature sequence by concatenation and / or registration after feature extraction on the at least two different modal data comprises feature extracting the image data to obtain image features, and the image features are further registered on the obtained feature sequence.

[0051] In this optional embodiment, all word segmentation results can be concatenated and then image features can be overlaid to obtain a feature sequence, allowing for enrichment of data information in forms other than knowledge graphs and text.

[0052] In some optional embodiments of this embodiment, the data of at least two different modalities includes text data to be processed and image data, and the step of obtaining a feature sequence by concatenation and / or superposition after performing feature extraction on the data of at least two different modalities includes: The method includes the steps of performing word segmentation processing on the text data to be processed and concatenating all the obtained word segmentation results to obtain a segmented word sequence, performing feature extraction on the image data to obtain image features, and superimposing the segmented word sequence and the image features to obtain a feature sequence.

[0053] In this embodiment, since the split word sequence and the image features are different data in two different dimensional spaces, there is only one method for overlaying the two. After obtaining the feature sequence through this overlay method, the split word sequence and the image features can be quickly separated.

[0054] In this optional embodiment, when the data to be processed includes at least two different modalities, including textual data and image data, the features extracted from both can be overlaid to effectively enrich the data information.

[0055] In step 203, a model mapping process is performed on the feature sequence to obtain multimodal input data that fits the autoregressive model.

[0056] In this embodiment, the feature sequence is subjected to model mapping processing in order to convert the feature sequence into data that can be received by the autoregressive model, i.e., the dimension of the converted data is the same as the dimension of the input layer of the autoregressive model, so that the converted data can be input to the autoregressive model, and the converted data can also be adjusted according to parameter changes during training of the autoregressive model.

[0057] In this embodiment, the contents of the data of at least two different modalities are different, and the contents of the obtained feature sequences are different, so the contents of the multi-modal input data are also different.

[0058] In some optional embodiments of this embodiment, when the data of at least two different modalities includes text data to be processed and image data, the step of performing a model mapping process on the feature sequence to obtain multi-modal input data that fits an autoregressive model includes: The method includes the steps of performing vector mapping on the divided word sequence in the feature sequence that conforms to the autoregressive model to obtain a data feature vector of the divided word sequence, performing vector mapping of the same dimension as the input layer of the autoregressive model on the image features in the feature sequence to obtain an image feature vector, and adding the data feature vector and the image feature vector to obtain multi-modal input data that conforms to the autoregressive model.

[0059] In this embodiment, the step of performing vector mapping for different data to fit the autoregressive model refers to converting the different data into a vector with the same dimension as the input layer of the autoregressive model so that the input layer of the autoregressive model can receive the converted different vector. Of course, the means for converting data of different formats into a vector with the same dimension as the input layer of the autoregressive model may be different.

[0060] In this optional embodiment, when the data of at least two different modalities includes text data to be processed and image data, vector mapping is performed on each of the text data to be processed and the image features to enrich the diversity of the content of the multi-modal input data.

[0061] Alternatively, when the data of at least two different modalities includes text data to be processed and knowledge graph data, the step of performing a model mapping process on the feature sequence to obtain multi-modal input data conforming to the autoregressive model includes: performing vector mapping of the feature sequence to an autoregressive model to obtain a data feature vector corresponding to the feature sequence, the data feature vector being multi-modal input data to be fitted to the autoregressive model, where in this optional embodiment the feature sequence is a concatenation of all word segmentation results;

[0062] In some optional embodiments of this embodiment, when the data of at least two different modalities includes text data to be processed, knowledge graph data, and image data, the step of performing a model mapping process on the feature sequence to obtain multi-modal input data that fits the autoregressive model includes: The method includes the steps of performing vector mapping conforming to an autoregressive model on all concatenated word segmentation results in the feature sequence to obtain data feature vectors corresponding to all concatenated word segmentation results, performing vector mapping of the same dimension as the input layer of the autoregressive model on image features in the feature sequence to obtain image feature vectors, and adding the data feature vector and the image feature vector to obtain multi-modal input data conforming to the autoregressive model.

[0063] In this optional embodiment, when the data of at least two different modalities includes text data to be processed, knowledge graph data and image data, a word segmentation process is performed on the text data to be processed and the knowledge graph data to obtain a plurality of word segmentation results, all the word segmentation results are concatenated to obtain concatenated word segmentation results, and vector mapping is performed on each of the concatenated word segmentation results and image features, thereby enriching the content of the multi-modal input data based on the data of the multiple different modalities.

[0064] Furthermore, since each feature in the feature sequence has a location and a type characteristic, location annotations and type annotations may be added to the obtained feature sequence.

[0065] In some optional embodiments of this embodiment, when the data of at least two different modalities includes text data to be processed and knowledge graph data, the above method further includes a step of determining total position data of each word segmentation result among all word segmentation results concatenated in the feature sequence.

[0066] The step of performing model mapping processing on the feature series to obtain multi-modal input data that conforms to the autoregressive model includes the steps of performing vector mapping on the feature series that conforms to the autoregressive model to obtain a data feature vector corresponding to the feature series, performing vector mapping on the total position data that conforms to the autoregressive model to obtain a total position vector, and adding the data feature vector and the total position vector to obtain multi-modal input data that conforms to the autoregressive model.

[0067] In this optional embodiment, position data (e.g., a code) can be assigned to each word segmentation result based on the order of the position of each word segmentation result among all the concatenated word segmentation results. The total position data is a set including the position data of each word segmentation result among all the word segmentation results, and based on the position data, the order value or expression information of each word segmentation result among all the concatenated word segmentation results can be easily and quickly identified.

[0068] In this optional embodiment, by determining the total position data of the word segmentation results, each word segmentation result can be efficiently located in the feature sequence, enriching the position features of the multi-modal input data.

[0069] In some optional embodiments of this embodiment, when the data of at least two different modalities includes the text data to be processed and the knowledge graph data, the above method further includes a step of performing a classification process on each word segmentation result among all the word segmentation results concatenated in the feature series to obtain total classification data. The step of performing a model mapping process on the feature series to obtain multi-modal input data conforming to an autoregressive model includes a step of performing vector mapping conforming to an autoregressive model on the feature series to obtain a data feature vector corresponding to the feature series, a step of performing vector mapping conforming to an autoregressive model on the total classification data to obtain a total classification vector, and a step of adding the data feature vector and the total classification vector to obtain multi-modal input data conforming to an autoregressive model.

[0070] In this optional embodiment, classification data (e.g., a code) can be assigned to each word segmentation result based on the type of each word segmentation result among all the concatenated word segmentation results. The total classification data is a set that includes classification data of each word segmentation result among all the word segmentation results, and based on the classification data, the type of each word segmentation result among all the concatenated word segmentation results can be easily and quickly identified.

[0071] In this optional embodiment, by determining the total classification data of the word segmentation results, the type to which each word segmentation result belongs in the feature sequence can be efficiently identified, thereby enriching the type features of the multi-modal input data.

[0072] Alternatively, when the data of at least two different modalities includes the text data to be processed and the knowledge graph data, the method further includes a step of determining total position data of each word segmentation result among all the word segmentation results concatenated in the feature series, and a step of performing a classification process on each word segmentation result among all the word segmentation results concatenated in the feature series to obtain total classification data. The step of performing a model mapping process on the feature series to obtain multimodal input data conforming to an autoregressive model includes a step of performing vector mapping conforming to an autoregressive model on the feature series to obtain a data feature vector corresponding to the feature series, a step of performing vector mapping conforming to an autoregressive model on the total position data to obtain a total position vector, a step of performing vector mapping conforming to an autoregressive model on the total classification data to obtain a total classification vector, and a step of adding the data feature vector, the total position vector, and the total classification vector to obtain multimodal input data conforming to an autoregressive model.

[0073] In some optional embodiments of this embodiment, when the data of at least two different modalities includes the text data to be processed and the image data, the above method further includes a step of determining word position data of each word of the divided word sequence, and a step of classifying each word of the divided word sequence to obtain word classification data, and the step of performing a model mapping process on the feature sequence to obtain multi-modal input data conforming to the autoregressive model includes: The method includes the steps of performing vector mapping conforming to an autoregressive model on the divided word sequence in the feature sequence to obtain a data feature vector of the divided word sequence, performing vector mapping of the same dimension as the input layer of the autoregressive model on the image features in the feature sequence to obtain an image feature vector, performing vector mapping conforming to an autoregressive model on the word position data to obtain a word position vector, performing vector mapping conforming to an autoregressive model on the word classification data to obtain a word classification vector, and adding the data feature vector, word position vector, word classification vector and image feature vector to obtain multi-modal input data conforming to the autoregressive model.

[0074] In this optional embodiment, by determining word position data and word classification data for each word in the split word sequence, the type to which each word belongs and the position of each word in the split word sequence can be efficiently determined, enriching the position and type features of the multi-modal input data.

[0075] In step 204, the multi-modal input data is input to the autoregressive model to obtain a uni-modal result output from the autoregressive model.

[0076] In this embodiment, the autoregressive model may be an autoregressive model whose output data is single modal data, such as UniLM (Unified Language Model Pre-training for Natural Language Understanding and Generation), GPT (Generative Pre-Training), etc. The autoregressive model may be a newly constructed model (for example, the input side is multimodal data and the output side is single modal data).

[0077] Alternatively, the autoregressive model may be a pre-trained autoregressive model, and both the input and output of the autoregressive model are single-module data; for example, the autoregressive model may be a language model, both the input and output of which are text data, and the data set to be processed by the language model may be data such as a dialogue transcript or a chapter or paragraph of a sentence.

[0078] After the multimodal input data is input to an autoregressive model, the autoregressive model is trained to obtain a trained autoregressive model.

[0079] For a Q&A type autoregressive model, a user's question (having multimodal data such as images, knowledge base, etc.) can be input to a trained autoregressive model to obtain an answer to the user's question.

[0080] In practical application scenarios, a Transformer autoregressive model structure such as a UniLM or DialoGPT model may be adopted as the autoregressive model. With the method disclosed herein, the autoregressive model can not only process traditional text data, but also process multimodal data containing information such as images, knowledge graphs, and text. As a whole, the autoregressive model can achieve a relatively good processing effect on the data set to be processed that has multimodality.

[0081] The model data processing method provided by the embodiment of the present disclosure first obtains at least two different modal data in a data set to be processed. Next, the at least two different modal data are subjected to feature extraction, and then a feature sequence is obtained by concatenation and / or superposition. Next, a model mapping process is performed on the feature sequence to obtain multimodal input data that fits the autoregressive model. Finally, the multimodal input data is input to the autoregressive model to obtain a single-modal result output from the autoregressive model. In this way, by inputting the multimodal input data obtained by processing at least two types of modal data in a data set to be processed into the autoregressive model, the autoregressive model can process not only single-modal data but also multimodal data, and the single-modal result output by the entire model becomes more accurate, and the processing ability of the model data is improved.

[0082] In another embodiment of the present disclosure, the data of at least two different modalities includes text data to be processed, knowledge graph data, and image data, and as shown in FIG. 3, a flow 300 of another embodiment of the model data processing method according to the present disclosure is shown, which includes the following steps:

[0083] In step 301, text data, knowledge graph data, and image data to be processed in a data set to be processed are obtained.

[0084] As shown in Fig. 4, the data set to be processed may be dialogue data extracted and processed from a dialogue system such as a customer service system of a department store, and this dialogue data includes text data and image data, and knowledge graph data of the dialogue content may be obtained from a knowledge graph created in advance. Note that the text corresponding to the text data to be processed may consist of at least one sentence, and each sentence may be divided by a symbol.

[0085] In FIG. 4, the text to be processed corresponding to the text data to be processed includes one question and one answer, and the question is: Do you have any skateboard sneakers? The skateboard sneakers in the question are displayed in an image sent by the user. The answer is: Do you have skateboard sneakers of this style? The skateboard sneakers in the answer are displayed in another image sent by the executing subject.

[0086] In step 302, ternary text data corresponding to the knowledge graph data is collected.

[0087] In this embodiment, the knowledge graph data is stored in the form of a triplet of a header entity, a relationship entity, and a tail entity, and after the knowledge graph data is extracted, the header entity, the relationship entity, and the tail entity of the knowledge graph data can be easily obtained.

[0088] In Fig. 4, the knowledge graph data obtained by searching the knowledge graph are Skateboard Sneakers-Material-Canvas, Skateboard Sneakers-Slip-on / Band / Shoelace Type-Shoelace, Skateboard Sneakers-Upper-Low Cut. The ternary texts corresponding to the ternary text data are Skateboard Sneakers-Material-Canvas, Skateboard Sneakers-Slip-on / Band / Shoelace Type-Shoelace, Skateboard Sneakers-Upper-Low Cut, respectively.

[0089] In step 303, word division processing is performed on the ternary text data and the processing target text data.

[0090] In this embodiment, a conventional word segmentation tool can be used to perform word segmentation on the processing target text corresponding to the processing target text data and the ternary text corresponding to the ternary text data.

[0091] Specifically, for ternary text word segmentation, we can obtain the word segmentation processed knowledge representation as follows: S11, S12, ..., S1n, P11, P12, ..., P1n, O11, O12, ..., O1n, S21, S22, ..., S2n, P21, P22, ..., P2n, O21, O22, ..., O2n, ..., Sn1, Sn2, ..., Snm, Pn1, Pn2, ..., Pnm, On1, On2, ..., Onm, where Snm denotes the m-th tag obtained by word segmentation in the header entity in the n-th triplet (n>0, m>0), Pnm denotes the m-th tag obtained by word segmentation in the relation entity in the n-th triplet, and Onm denotes the m-th tag obtained by word segmentation in the tail entity in the n-th triplet.

[0092] Specifically, in this embodiment, each sentence in the context corresponding to the processing target text data is word-segmented to obtain Wn1, Wn2, Wn3, ... Wnm, where Wnm indicates the m-th tag following the word segmentation result in the n-th (n>0, m>0) sentence. In actual word segmentation, text with special meanings, such as URL (Uniform Resource Location) links, emoticon character expressions, etc., can be replaced with special tags, thereby maximally preserving the meaning of the sentence.

[0093] In step 304, all the word segmentation results are concatenated to obtain all concatenated word segmentation results.

[0094] As shown in Figure 4, the tags of the knowledge graph data (abbreviated as knowledge tags) and the word segmentation result tags of each sentence in the processing target text are sequentially concatenated with the special symbol [CLS] at the beginning of the sentence. Among these, it is necessary to add the special symbol [SEP] between the knowledge graph data and the processing target text, and between sentences, to indicate the interval between tags with different contents.

[0095] In step 305, feature extraction is performed on the image data to obtain image features.

[0096] In this embodiment, image features can be extracted from the images shown in the dataset to be processed using a pre-trained feature collection model (e.g., ResNet50).

[0097] In step 306, all the concatenated word segmentation results are combined with the image features to obtain a feature sequence.

[0098] In this embodiment, all concatenated word segmentation results are superimposed with image features in order to obtain a unified overall feature sequence so that the entire feature sequence can be conveniently applied to subsequent processing.

[0099] In step 307, the total position data of each word segmentation result among all the word segmentation results concatenated in the feature sequence is determined.

[0100] In this embodiment, the total position data is determined by assigning a position code to each word segmentation result in the feature sequence. As shown in FIG. 4, the position data of [CLS] located at the beginning of the sentence is set to 0, the position data of all knowledge tags is set to 1, and the other non-knowledge tags are sequentially increased from 2. As shown in FIG. 4, the obtained total position data is 011111111123456789101112.

[0101] In step 308, a classification process is performed on each word segmentation result among all the word segmentation results concatenated in the feature sequence to obtain total classification data.

[0102] In this embodiment, in order to distinguish different groups of concatenated tags, several categories indicating different tags are introduced as shown in Fig. 4, in which the [CLS] category indicates that the tag is a header tag of a sentence, the [SEP] category indicates that the tag is a boundary between knowledge and a sentence, or between sentences, the [SUB] category indicates that the tag is a header entity belonging to a knowledge triplet, the [PRE] category indicates that the tag is a relation entity belonging to a knowledge triplet, and the [OBJ] category indicates that the tag is a tail entity belonging to a knowledge triplet. The [SP1] category indicates that the tag belongs to the content stated by role 1 in the dialogue, and the [SP2] category indicates that the tag belongs to the content stated by role 2 in the dialogue.

[0103] In step 309, vector mapping is performed on all the concatenated word segmentation results in the feature sequence, fitting the autoregressive model, to obtain data feature vectors corresponding to all the concatenated word segmentation results.

[0104] In this embodiment, vector mapping refers to mapping one piece of feature data to a feature vector that fits an autoregressive model and fusing the data.

[0105] In this embodiment, the knowledge tags (Snm, Pnm, Onm) generated in step 303, the text tag Wnm, and the [CLS] and [SEP] generated in step 304 are semantically consistent, so a data feature vector is obtained using the same vocabulary list and corresponding vector mapping, and the dimension of the obtained data feature vector is the same as the dimension of the input layer of the autoregressive model.

[0106] In practice, we can obtain the data feature vector through the tf.nn.embedding_lookup(params,ids) function, which mainly selects the elements corresponding to the index in the tensor, where params can be a tensor or an array, and ids is the corresponding index value, which is also a vector corresponding to the autoregressive model.

[0107] In step 310, vector mapping of the same dimension as the input layer of the autoregressive model is performed on the image features in the feature sequence to obtain an image feature vector.

[0108] In this embodiment, the image features in the feature sequence are processed using a linear regression model (Linear), so that the image features can be mapped to a specified dimension that is the same dimension as the input layer of the autoregressive model.

[0109] In addition, the autoregressive model may be a language model such as a question-and-answer model, and the dimension of the input layer of the autoregressive model is determined by the dimension of the embedding layer therein, and the dimension of the embedding belongs to the superparameter and may be, for example, 300 or 512 dimensions.

[0110] In step 311, vector mapping is performed on the aggregate position data by fitting an autoregressive model to obtain an aggregate position vector.

[0111] In this embodiment, the total position vector can be determined by a conventional Transformer position encoding method using trigonometric functions, thereby mapping the corresponding total position data into a continuous space of a specified dimension.

[0112] In step 312, vector mapping is performed on the aggregate classification data by fitting an autoregressive model to obtain an aggregate classification vector.

[0113] In this embodiment, the seven different category tags obtained in step 308 are mapped into a continuous space of a specified dimension using a vocabulary list and corresponding vector mapping.

[0114] In step 313, the data feature vector, the image feature vector, the aggregate position vector, and the aggregate classification vector are summed to obtain multi-modal input data that is fitted to the autoregressive model.

[0115] In this embodiment, after the above processing, the data feature vector, image feature vector, total position vector, and total classification vector are all mapped to the specified dimensions, so by adding these four types of vectors, input data for the autoregressive model can be obtained.

[0116] In step 314, the multi-modal input data is input to the autoregressive model to obtain a uni-modal result output from the autoregressive model.

[0117] In this embodiment, the autoregressive model may be an autoregressive language model based on Transformer (a model structure consisting of an encoder and a decoder). As shown in Fig. 4, the dialogue context, image, and knowledge information are data processed to obtain multimodal input data, which are then input into the autoregressive language model and the model is trained, so that the multimodal information and knowledge information can be used when generating user question and answer, and the answers generated in this way are smoother and more accurate.

[0118] In this embodiment, by efficiently fusing three pieces of information, namely, knowledge graph data, image data, and text data, the context information of the dialogue can be fully expressed, and end-to-end training can be performed using a model based on an autoregressive structure to generate smooth and informative dialogue responses.

[0119] In the model data processing method of this embodiment, when data of at least two different modalities includes text data to be processed, knowledge graph data, and image data, image features extracted from the features of the image data are superimposed on all word segmentation results obtained by concatenating the text data to be processed and the knowledge graph data to obtain a feature sequence. Total position data of each word segmentation result out of all word segmentation results concatenated in the feature sequence of all concatenated word segmentation results is determined, and each word segmentation result out of all word segmentation results concatenated in the feature sequence of all concatenated word segmentation results is classified to obtain total classification data. Vector mapping that fits the autoregressive model is performed on the total position data, total classification data, and all concatenated word segmentation results, and a data feature vector, image feature vector, total position vector, and total classification vector are obtained, respectively. The image data is input to the same layer as the input layer of the autoregressive model, and the total position data of each word segmentation result out of all word segmentation results concatenated in the feature sequence of all concatenated word segmentation results is determined, and total classification data is obtained. dimension Vector mapping is performed to obtain an image feature vector, and the data feature vector, image feature vector, total location vector, and total classification vector are added to obtain multi-modal input data that fits the autoregressive model. This can cover data of different modalities such as knowledge base, text, and image, and can effectively locate and classify data of different modalities of knowledge, improving the accuracy of the obtained multi-modal input data, and making the single-modal results output by the autoregressive model smoother and more accurate.

[0120] Further referring to FIG. 5, as an implementation of the method shown in each of the above figures, the present disclosure provides an embodiment of a model data processing apparatus, which corresponds to the embodiment of the method shown in FIG. 2, and which can be specifically applied to various electronic devices.

[0121] As shown in FIG. 5, a model data processing apparatus 500 according to an embodiment of the present disclosure is provided, and the apparatus 500 includes an acquisition unit 501, an extraction unit 502, an acquisition unit 503, and a processing unit 504. The acquisition unit 501 may be configured to acquire at least two different modal data in a dataset to be processed. The extraction unit 502 may be configured to perform feature extraction on the at least two different modal data and then obtain a feature sequence by concat and / or superposition. The acquisition unit 503 may be configured to perform a model mapping process on the feature sequence to obtain multi-modal input data that fits an autoregressive model. The processing unit 504 may be configured to input the multi-modal input data into the autoregressive model to obtain a single-modal result output from the autoregressive model.

[0122] In this embodiment, the specific processing of the acquisition unit 501, the extraction unit 502, the acquisition unit 503, and the processing unit 504 in the model data processing device 500 and the technical effects achieved thereby may refer to step 201, step 202, step 203, and step 204 in the corresponding embodiment of FIG. 2, respectively.

[0123] In some embodiments, the at least two different modal data include target text data and knowledge graph data, and the extraction unit 502 includes a collection module (not shown), a total word segmentation module (not shown), and a total concatenation module (not shown). The collection module may be configured to collect ternary text data corresponding to the knowledge graph data. The total word segmentation module may be configured to perform word segmentation processing on each of the ternary text data and the target text data. The total concatenation module may be configured to concatenate all word segmentation results to obtain a feature sequence.

[0124] In some embodiments, the apparatus 500 further includes a total position determination unit (not shown) that may be configured to determine total position data of each word segmentation result among all the word segmentation results concatenated in the feature sequence. The acquisition unit 503 includes a data mapping module (not shown), a total position mapping module (not shown), and a data-position summing module (not shown). Among them, the data mapping module may be configured to perform vector mapping that fits an autoregressive model on the feature sequence to obtain a data feature vector corresponding to the feature sequence. The total position mapping module may be configured to perform vector mapping that fits an autoregressive model on the total position data to obtain a total position vector. The data-position summing module may be configured to sum the data feature vector and the total position vector to obtain multi-modal input data that fits an autoregressive model.

[0125] In some embodiments, the apparatus 500 further comprises an aggregate classification unit (not shown) that may be configured to perform a classification process on each word segmentation result among all the concatenated word segmentation results in the feature sequence to obtain aggregate classification data. The acquisition unit 503 comprises a data mapping module (not shown), an aggregate classification mapping module (not shown), and a data and classification summing module (not shown). Among them, the data mapping module may be configured to perform vector mapping on the feature sequence that fits an autoregressive model to obtain a data feature vector corresponding to the feature sequence. The aggregate classification mapping module may be configured to perform vector mapping on the aggregate classification data that fits an autoregressive model to obtain an aggregate classification vector. The data and classification summing module may be configured to sum the data feature vector and the aggregate classification vector to obtain multimodal input data that fits an autoregressive model.

[0126] In some embodiments, the at least two different modal data further include image data. The extraction unit 502 may be configured to perform feature extraction on the image data to obtain image features. In this embodiment, image features are further superimposed on the obtained feature sequence. The acquisition unit 503 includes a data mapping module (not shown), an image mapping module (not shown), and a data-image addition module (not shown). Among them, the data mapping module may be configured to perform vector mapping that fits an autoregressive model on all concatenated word segmentation results in the feature sequence to obtain a data feature vector corresponding to all concatenated word segmentation results. The image mapping module may be configured to perform vector mapping of the same dimension as the input layer of the autoregressive model on the image features in the feature sequence to obtain an image feature vector. The data-image addition module may be configured to add the data feature vector and the image feature vector to obtain multimodal input data that fits the autoregressive model.

[0127] In some embodiments, the apparatus 500 further comprises a total position determination unit (not shown) and a total classification unit (not shown). The total position determination unit may be configured to determine total position data of each word segmentation result among all the word segmentation results concatenated in the feature sequence. The total classification unit may be configured to perform a classification process on each word segmentation result among all the word segmentation results concatenated in the feature sequence to obtain total classification data. The acquisition unit 503 further comprises a total position mapping module (not shown) and a total classification mapping module (not shown). However, the total position mapping module is configured to perform vector mapping fitting an autoregressive model on the total position data to obtain a total position vector. The total classification mapping module is configured to perform vector mapping fitting an autoregressive model on the total classification data to obtain a total classification vector. In this embodiment, the obtained multimodal input data fitting an autoregressive model further includes a total position vector and a total classification vector.

[0128] In some embodiments, the at least two different modal data include text data to be processed and image data. The extraction unit 502 includes a word segmentation waiting module (not shown), a concatenation waiting module (not shown), an extraction module (not shown), and a superimposition module (not shown). Among these, the word segmentation waiting module may be configured to perform a word segmentation process on the text data to be processed. The concatenation waiting module may be configured to concatenate all the word segmentation results to obtain a segmented word sequence. The extraction module may be configured to perform feature extraction on the image data to obtain image features. The superimposition module may be configured to superimpose the segmented word sequence and the image features to obtain a feature sequence. The acquisition unit 503 includes a word segmentation result mapping module (not shown), an image mapping module (not shown), and a word-image addition module (not shown). Among these, the word segmentation result mapping module may be configured to perform vector mapping conforming to an autoregressive model on the segmented word sequence in the feature sequence to obtain a data feature vector of the segmented word sequence. The image mapping module may be configured to perform vector mapping of the same dimension as the input layer of the autoregressive model on the image features in the feature sequence to obtain an image feature vector. The word-image addition module may be configured to add the data feature vector and the image feature vector to obtain multimodal input data to fit the autoregressive model.

[0129] In some embodiments, the apparatus 500 further includes a word position determining unit (not shown) and a word classification unit (not shown). The word position determining unit is configured to determine word position data of each word in the segmented word sequence. The word classification unit is configured to classify each word in the segmented word sequence to obtain word classification data. The acquisition unit 503 further includes a word position mapping module (not shown) and a word classification mapping module (not shown). Among them, the word position mapping module may be configured to perform vector mapping conforming to an autoregressive model on the word position data to obtain a word position vector. The word classification mapping module may be configured to perform vector mapping conforming to an autoregressive model on the word classification data to obtain a word classification vector. In this embodiment, the obtained multimodal input data conforming to the autoregressive model further includes a word position vector and a word classification vector.

[0130] Please refer to FIG. 6, which is a structural schematic diagram showing an electronic device 600 applied to realize an embodiment of the present disclosure.

[0131] 6, the electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 601 capable of performing various suitable operations and processes according to programs stored in a read-only memory (ROM) 602 or programs loaded from a storage device 608 into a random access memory (RAM) 603. The RAM 603 further stores various programs and data necessary for the operation of the electronic device 600. The processing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0132] Input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, etc., output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc., storage devices 608 including, for example, a magnetic tape, a hard disk, etc., and a communication device 609 may be connected to the I / O interface 605. The communication device 609 allows the electronic device 600 to communicate wirelessly or wired with other devices to exchange data. Although FIG. 6 illustrates the electronic device 600 having various devices, it should be understood that it is not required to implement or include all of the devices illustrated. Optionally, more or less devices may be implemented. Each block illustrated in FIG. 6 may represent one device, or may represent multiple devices as needed.

[0133] In particular, according to embodiments of the present disclosure, the processes described with reference to the flowcharts above may be implemented as a computer software program. For example, embodiments of the present disclosure may be implemented as a computer program embodied in a computer readable medium. of The computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication device 609, or can be installed from the storage device 608 or the ROM 602. When the computer program is executed by the processing device 601, it performs the above-mentioned functions defined in the method of the embodiment of the present disclosure.

[0134] The computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of the computer-readable storage medium include a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), electrically connected by one or more conductors. )、 It may include, but is not limited to, a portable compact disk read only memory (CD-ROM), an optical memory, a magnetic memory, or any suitable combination thereof. In an embodiment of the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program usable by or embedded in an instruction execution system, apparatus, or device. In an embodiment of the present disclosure, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code carried therein. Such a propagated data signal may take various forms, including, but is not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium. The computer-readable signal medium may transmit, propagate, or transmit a program usable by or embedded in an instruction execution system, apparatus, or device. The program code contained in the computer readable medium can be transmitted over any suitable medium, including but not limited to electrical wire, optical cable, RF (Radio Frequency), or the like, or any suitable combination thereof.

[0135] The computer-readable medium may be included in the server or may exist separately from the server. The computer-readable medium carries one or more programs that, when executed by the server, cause the server to execute the steps of: acquiring data of at least two different modalities in a dataset to be processed, performing feature extraction on the data of the at least two different modalities and then acquiring a feature sequence by concatenation and / or superposition, performing a model mapping process on the feature sequence to acquire multimodal input data that fits an autoregressive model, and inputting the multimodal input data into the autoregressive model to acquire a single-modal result output from the autoregressive model.

[0136] Computer program code for carrying out the operations of the embodiments of the present disclosure can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and traditional procedural programming languages ​​such as "C" or similar programming languages. The program code can run entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via an Internet service provided by an Internet service provider).

[0137] The flowcharts and block diagrams in the drawings illustrate systems, methods and computer programs according to various embodiments of the present disclosure. To The diagrams illustrate the architecture, functions, and operations that can be realized by the above-mentioned embodiment. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a portion of code. The module, program segment, or portion of code includes one or more executable instructions for implementing a certain logic function. It should be noted that in some optional embodiments, the functions shown in the blocks may be executed in a different order than that shown in the drawings. For example, two blocks shown in succession may actually be executed substantially in parallel in response to such functions, or may sometimes be executed in the reverse order. It should be further noted that all blocks in the block diagrams and / or flowcharts, as well as combinations of blocks in the block diagrams and / or flowcharts, may be implemented in a dedicated hardware-based system that executes a certain function or operation, or may be implemented in a combination of dedicated hardware and computer instructions.

[0138] The units described in the embodiments of the present disclosure may be implemented in software or hardware. The described units may be located in a processor, and may be described as "a processor including an acquisition unit, an extraction unit, an acquisition unit, and a processing unit", where the names of the units are not intended to limit the units themselves in some cases. For example, the acquisition unit may be described as a unit "configured to acquire data of at least two different modalities in a dataset to be processed".

[0139] The above description is merely a description of the preferred embodiments of the present disclosure and the technical principles applied thereto. Those skilled in the art should understand that the scope of the present disclosure is not limited to the technical solution consisting of a specific combination of the above-mentioned technical features, but should also include other technical solutions consisting of any combination of the above-mentioned technical features or their equivalent features within the scope of the above-mentioned disclosure. For example, the above features may be substituted with technical features having similar functions (but are not limited to these) disclosed in the embodiments of the present disclosure.

Claims

1. A computer-implemented method for processing model data, comprising the steps of: acquiring data of at least two different modalities in a data set to be processed; performing feature extraction on the at least two different modal data and then performing concatenation and / or superposition to obtain a feature sequence; performing a model mapping process on the feature sequence to obtain multi-modal input data that fits an autoregressive model; inputting the multi-modal input data into the autoregressive model to obtain a uni-modal result output from the autoregressive model; The at least two different types of modal data include text data to be processed and knowledge graph data; The step of obtaining a feature sequence by concatenation and / or superposition after performing feature extraction on the data of at least two different modalities includes: collecting ternary text data corresponding to the knowledge graph data; performing a word segmentation process on the ternary text data and the processing target text data, respectively, and concatenating all the word segmentation results to obtain a feature sequence; A method for processing model data comprising:

2. The model data processing method includes: determining total position data of each word segmentation result among all the word segmentation results concatenated in the feature sequence; The step of performing a model mapping process on the feature sequence to obtain multi-modal input data that fits an autoregressive model includes: performing vector mapping on the feature sequence that fits an autoregressive model to obtain a data feature vector corresponding to the feature sequence; performing vector mapping of the aggregate position data that fits the autoregressive model to obtain an aggregate position vector; adding the data feature vector and the aggregate position vector to obtain multi-modal input data that is fitted to the autoregressive model; 2. The method of claim 1, further comprising:

3. The model data processing method includes: The method further includes a step of performing a classification process on each word segmentation result among all the word segmentation results concatenated in the feature sequence to obtain total classification data; The step of performing a model mapping process on the feature sequence to obtain multi-modal input data that fits an autoregressive model includes: performing vector mapping on the feature sequence that fits an autoregressive model to obtain a data feature vector corresponding to the feature sequence; performing vector mapping of the aggregate classification data that fits the autoregressive model to obtain an aggregate classification vector; adding the data feature vector and the aggregate classification vector to obtain multi-modal input data that is fit to the autoregressive model; 2. The method of claim 1, further comprising:

4. the at least two different modal data further comprises image data; The step of obtaining a feature sequence by concatenation and / or superposition after performing feature extraction on the data of at least two different modalities includes: performing feature extraction on the image data to obtain image features; the image features are further superimposed on the obtained feature sequence; The step of performing a model mapping process on the feature sequence to obtain multi-modal input data that fits an autoregressive model includes: performing vector mapping that fits an autoregressive model on all the concatenated word segmentation results in the feature sequence to obtain data feature vectors corresponding to all the concatenated word segmentation results; performing vector mapping of the same dimension as the input layer of the autoregressive model on the image features in the feature sequence to obtain an image feature vector; adding the data feature vector and the image feature vector to obtain multi-modal input data that is fitted to the autoregressive model; 2. The method of claim 1, further comprising:

5. The model data processing method includes: determining total position data of each word segmentation result among all the word segmentation results concatenated in the feature sequence; and performing a classification process on each word segmentation result among all the word segmentation results concatenated in the feature sequence to obtain total classification data. The step of performing a model mapping process on the feature sequence to obtain multi-modal input data that fits an autoregressive model includes: performing vector mapping of the aggregate position data that fits the autoregressive model to obtain an aggregate position vector; performing vector mapping of the aggregate classification data to fit the autoregressive model to obtain an aggregate classification vector; The resulting multi-modal input data for fitting the autoregressive model further includes the aggregate position vector and the aggregate classification vector.

5. The model data processing method according to claim 4.

6. The at least two different types of data include text data to be processed and image data; The step of obtaining a feature sequence by concatenation and / or superposition after performing feature extraction on the data of at least two different modalities includes: a step of performing a word segmentation process on the text data to be processed and concatenating all the word segmentation results to obtain a segmented word sequence; performing feature extraction on the image data to obtain image features; and overlapping the segmented word sequence with the image features to obtain a feature sequence; The step of performing a model mapping process on the feature sequence to obtain multi-modal input data that fits an autoregressive model includes: performing vector mapping conforming to an autoregressive model on the divided word sequence in the feature sequence to obtain a data feature vector of the divided word sequence; performing vector mapping of the same dimension as the input layer of the autoregressive model on the image features in the feature sequence to obtain an image feature vector; adding the data feature vector and the image feature vector to obtain multi-modal input data that is fitted to the autoregressive model; 2. The method of claim 1, further comprising:

7. The model data processing method includes: determining word position data of each word in the divided word sequence; and classifying each word in the divided word sequence to obtain word classification data. The step of performing a model mapping process on the feature sequence to obtain multi-modal input data that fits an autoregressive model includes: performing vector mapping on the word position data that fits the autoregressive model to obtain word position vectors; performing vector mapping on the word classification data that fits the autoregressive model to obtain a word classification vector; The resulting multi-modal input data for fitting the autoregressive model further includes the word position vector and the word classification vector.

7. The model data processing method according to claim 6.

8. A model data processing device, an acquisition unit configured to acquire data of at least two different modalities in a data set to be processed; an extraction unit configured to obtain a feature sequence by concatenation and / or superposition after performing feature extraction on the at least two different modal data; an acquisition unit configured to perform a model mapping process on the feature sequence to acquire multi-modal input data that fits an autoregressive model; a processing unit configured to input the multi-modal input data into the autoregressive model to obtain a uni-modal result output from the autoregressive model; The at least two different types of modal data include text data to be processed and knowledge graph data; The extraction unit comprises: Collecting ternary text data corresponding to the knowledge graph data; The model data processing device is further configured to perform a word segmentation process on the ternary text data and the processing target text data, respectively, and to obtain a feature sequence by concatenating all word segmentation results.

9. An electronic device, An electronic device comprising one or more processors and a storage device storing one or more programs, An electronic device that, when the one or more programs are executed by the one or more processors, causes the one or more processors to implement the model data processing method according to any one of claims 1 to 7.

10. A computer-readable medium having a computer program stored thereon, A computer readable medium which, when executed by a processor, implements the model data processing method according to any one of claims 1 to 7.

11. A computer program which, when executed by a processor, implements the model data processing method according to any one of claims 1 to 7.

Images