Method, apparatus and computer program product for generating data
By encoding multimodal data including text and image data to generate multimodal table data, the method addresses the limitations of existing tabular data generation methods, improving data representation and reducing costs, suitable for applications like data privacy protection and model training.
Patent Information
- Application Number
- CN202310652143.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-02
- Publication Date
- 2024-12-03
AI Technical Summary
Existing tabular data generation methods focus on numerical and categorical data, which are simple in structure and limited in information, leading to insufficient representation and increased model configuration costs.
A method and device for generating multimodal table data by encoding multimodal data comprising text and image data, obtaining multimodal and table embeddings, and generating conditional embeddings to improve representation and reduce model configuration costs.
Enhances the representation ability of generated data, making it more descriptive of real scenarios while reducing computing and storage resources, and facilitating applications such as data privacy protection and model training.
Smart Images

Figure CN119067083A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure relate generally to the field of computers, and more particularly to methods, devices, and computer program products for generating data. Background Art
[0002] In today's information age, data and artificial intelligence technologies are widely used in all aspects. With the development of artificial intelligence technology and the improvement of data processing capabilities, it is becoming more and more common to use data to build smart products. Data may be obtained through various forms of sharing.
[0003] People are beginning to realize the importance of sharing data. By sharing data, we can make full use of information resources. For example, we can better understand needs, optimize products and services, etc. However, in the process of sharing data, we will face many challenges. Summary of the invention
[0004] The embodiments of the present disclosure provide a solution for generating data. According to the solution for generating data according to the embodiments of the present disclosure, the data characterization capability of the output can be improved and the model configuration cost can be reduced.
[0005] In a first aspect of the present disclosure, a method for generating data is provided, the method comprising obtaining a multimodal embedding by encoding multimodal data, the multimodal data comprising text data and image data. The method further comprises obtaining a table embedding by encoding table data associated with the multimodal data. The method further comprises generating a conditional embedding based on the multimodal embedding and the table embedding. The method further comprises generating multimodal table data based on the conditional embedding.
[0006] In another aspect of the present disclosure, a device for generating data is provided, the device comprising a processor, and a memory, the memory being coupled to the processor and having instructions stored thereon, the instructions, when executed by the processor, causing the device to perform actions, the actions comprising obtaining a multimodal embedding by encoding multimodal data, the multimodal data comprising text data and image data. The actions also comprise obtaining a table embedding by encoding table data associated with the multimodal data. The actions also comprise generating a conditional embedding based on the multimodal embedding and the table embedding. The actions also comprise generating multimodal table data based on the conditional embedding.
[0007] In another aspect of the present disclosure, a computer program product is provided, which is tangibly stored on a non-transitory computer-readable storage medium and includes computer-executable instructions, which, when executed, cause a computer to perform a method or process according to an embodiment of the present disclosure.
[0008] Please note that the invention summary is provided to introduce a series of concepts in a simplified form, which will be further described in the detailed description below. The invention summary is not intended to identify key features or essential features of the present disclosure, nor is it intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] The above and other purposes, features and advantages of the present disclosure will become more clearly understood by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings, in which:
[0010] Figure 1 A schematic diagram illustrating an example environment in which methods and / or processes according to embodiments of the present disclosure may be implemented;
[0011] Figure 2 A flowchart of a method for generating data according to an embodiment of the present disclosure is illustrated;
[0012] Figure 3 is a diagram illustrating the architecture of a multimodal tabular data generation model according to an embodiment of the present disclosure;
[0013] Figure 4 is a diagram illustrating a multimodal conversion process according to an embodiment of the present disclosure;
[0014] Figure 5 is a diagram illustrating a process of generating multimodal table data according to an embodiment of the present disclosure;
[0015] Figure 6 illustrates an example of generated multimodal table data according to an embodiment of the present disclosure; and
[0016] Figure 7 A schematic block diagram of an example device that can be used to implement an embodiment according to the present disclosure is illustrated.
[0017] Throughout the drawings, same or similar reference numbers generally refer to same or similar elements. Specific embodiments
[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0019] In the description of the embodiments of the present disclosure, the term "including" and its variations should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects, unless explicitly indicated to be different.
[0020] As mentioned above, when using data, data sharing, synchronization, or presentation may be involved, so that information resources can be fully utilized. However, in the process of using data, there will be many challenges, and data privacy is one of them. The shared data may involve sensitive information such as identity information and financial information. Once this information is leaked, it will cause serious losses to individuals and organizations. Under the current wave of technological advances such as big data and artificial intelligence, the amount of data that needs to be processed has increased dramatically, and these data may include the above-mentioned sensitive information. In order to avoid privacy leakage, the mechanism of protecting data privacy becomes crucial. Therefore, it is desirable and meaningful to strike a balance between data sharing and data privacy.
[0021] Tabular data generation is an important task in various fields such as data privacy, data augmentation, and machine learning. Existing tabular data generation methods only focus on tables with numerical data and categorical data. Although numerical data and categorical data are simple in structure and easy to process for generation, they only contain less information or even meaningless information. In other words, such tables only have limited information to describe or characterize real scenarios, resulting in limitations in the tabular data generated by these methods in practical application scenarios.
[0022] In order to solve at least some of the above and other potential problems, an embodiment of the present disclosure proposes a scheme for generating data. The scheme includes obtaining a multimodal embedding by encoding multimodal data, and the multimodal data includes text data and image data. The scheme also includes obtaining a table embedding by encoding tabular data associated with the multimodal data. The scheme also includes generating conditional embeddings based on the multimodal embedding and the table embedding. The scheme also includes generating multimodal table data based on the conditional embedding. In this way, tabular data generation can be combined with multimodal learning, which improves the representation ability of the generated data, so that the generated data has sufficient information to describe the real scene. At the same time, the model configuration cost can be reduced, saving computing and storage resources.
[0023] Reference below Figures 1 to 7It should be understood that these exemplary embodiments are provided only to enable those skilled in the art to better understand and implement the embodiments of the present disclosure, and are not intended to limit the scope of the present disclosure in any way.
[0024] Figure 1 A schematic diagram of an example environment 100 in which methods and / or processes according to embodiments of the present disclosure may be implemented is shown. In the example environment 100, table data 111 and multimodal data 112 may be input as input 110 into a multimodal table data generation model according to embodiments of the present disclosure disposed at a computing device 120. Corresponding processing and calculations (e.g., the encoding and decoding process for multimodal table data generation described below, etc.) are performed by the multimodal table data generation model at the computing device 120 to implement multimodal table data generation according to embodiments of the present disclosure. After the multimodal table data is generated, the generated multimodal table data may be used for various purposes, such as data privacy protection 131, data enhancement 132, model training 133, etc.
[0025] like Figure 1 As shown in , input 110 includes tabular data 111 and multimodal data 112 . Figure 1 The table data 111 in exemplarily shows numerical data and category data. Figure 1 In , a row of a table (such as the first row) may include categories of data, such as category 1, category 2, and category 3, etc. One or more columns of a table (such as the first column) may include values, which may indicate the number of a row of data or may have a predetermined meaning. Figure 1 , categories are shown in rows, and values are shown in columns. However, this is by no means restrictive, and according to an embodiment of the present disclosure, categories may be shown in columns, and values may be shown in rows.
[0026] It should be understood that any row of the table may include a category of data, and two or more rows are also feasible, and the present disclosure does not limit this. It should also be understood that for ease of illustration, Figure 1 Only the first column is shown to include values. However, other columns (such as the second column or the third column) or multiple columns (such as any combination of the first column, the second column and the third column) of the table may also include values, and the present disclosure does not limit this. It should also be noted that for ease of illustration, Figure 1 Only three rows and three columns are shown in FIG. However, the table may include more or fewer rows and columns, and the present disclosure is not limited thereto.
[0027] like Figure 1As shown in , multimodal data 112 includes image data 113 and text data 114. As mentioned above, the existing table data generation method can only process simple numerical data and category data, but cannot process more complex high-level data, resulting in limitations in the generated table data. To this end, the present disclosure introduces the concept of multimodal learning to help improve the representation and description capabilities of the output. It should be understood that for the purpose of ease of understanding, Figure 1 exemplarily shows that the multimodal data 112 includes image data 113 and text data 114. However, the multimodal data 112 may include more or fewer types of data, such as audio data and video data, and the present disclosure does not limit this. It should also be noted that the present disclosure does not specify or limit the format, size, etc. of the data (such as, image data 113 and text data 114, etc.) included in the multimodal data 112.
[0028] Figure 1 The computing device 120 shown in can be any device with computing capabilities. According to an embodiment of the present disclosure. After receiving the input 110 as described above, the computing device 120 performs corresponding processing and calculations through the multimodal table data generation model thereon, so as to realize the multimodal table data generation according to an embodiment of the present disclosure. Examples of the computing device 120 may include smart phones, tablet computers, personal computers, laptop computers, server computers, multiprocessor systems, wearable electronic devices, multimedia players, personal digital assistants (PDAs), smart home devices, consumer electronics, etc. Examples of the computing device 120 may also include a distributed computing environment including any of the above-mentioned devices or systems, etc. In the following, the multimodal table data generation according to an embodiment of the present disclosure will be further described in detail.
[0029] After completing the generation of multimodal table data according to the embodiment of the present disclosure, the generated multimodal table data can be used for a variety of purposes. Data privacy protection 131 is one of them. Multimodal table data can be published under the Progressive Privacy-Preserving Data Publishing (PPDP) framework. In the field of privacy-preserving data publishing, PPDP technology is proposed as a method for generating synthetic data to protect the privacy of original data. The PPDP framework allows the generation of synthetic data to protect the privacy of original data by gradually publishing more detailed information about the data. Thereby, while fully protecting data privacy, the availability and exploitability of data are maximized.
[0030] In addition to data privacy protection 131, the generated multimodal tabular data can also be used for data enhancement 132. Data enhancement 132 refers to a technique for generating more data by performing a series of transformations or expansions on the original data. By generating multimodal tabular data according to an embodiment of the present disclosure, the scale and diversity of the data set can be effectively expanded.
[0031] The generated multimodal tabular data can also be used for model training 133. By generating multimodal tabular data according to an embodiment of the present disclosure, the scale and diversity of the generated multimodal tabular data are improved compared to the original data. Using the generated multimodal tabular data as a training set to train, for example, a machine learning model can improve the generalization ability and robustness of the model, reduce the risk of overfitting, and thus improve the performance of the model. It should be understood that the uses described above are not intended to be restrictive, but are merely exemplified for the purpose of aiding understanding. The generated multimodal tabular data can also be used for more purposes, which are not limited by the present disclosure.
[0032] Combined with the above Figure 1 A schematic diagram of an example environment 100 is depicted in which methods and / or processes according to embodiments of the present disclosure may be implemented. Figure 2 1. A flowchart of a method 200 for generating data according to an embodiment of the present disclosure is described. As described above with respect to the computing device 120, the method 200 for generating data according to an embodiment of the present disclosure can be executed at an edge device with computing capabilities, or at a cloud server, and the present disclosure does not limit this.
[0033] Figure 2 A flow chart of a method 200 for generating data according to an embodiment of the present disclosure is shown. In order to improve the representation capability of the generated data and reduce the model configuration cost, a method 200 for generating data according to an embodiment of the present disclosure is proposed.
[0034] At box 210, a multimodal embedding is obtained by encoding the multimodal data, where the multimodal data includes text data and image data. The multimodal table data generation model according to an embodiment of the present disclosure can process multimodal data 112 with higher complexity, and can convert the multimodal data 112 into corresponding vector representations. In the following, the multimodal conversion process will be further described in detail.
[0035] At box 220, a table embedding is obtained by encoding the tabular data associated with the multimodal data. As described above, the tabular data 111 may include numerical data and category data. For example, since the data in the multimodal data 112 can be identified by the numerical data in the tabular data 111, or can belong to a category indicated by the category data in the tabular data 111, the multimodal data 112 is associated with the tabular data 111. The multimodal tabular data generation model according to an embodiment of the present disclosure can process relatively simple tabular data 111, and can convert the tabular data 111 into a corresponding vector representation. In the following, the conversion process for the tabular data 111 will be described in further detail.
[0036] At block 230, conditional embedding is generated based on the multimodal embedding and the table embedding. According to an embodiment of the present disclosure, conditional embedding can be generated based on the multimodal embedding generated at block 210 and the table embedding generated at block 220, and the conditional embedding can be used as a constraint in the data generation process to control the mode of the output. In the following, the generation process of the conditional embedding will be described in further detail.
[0037] At box 240, multimodal tabular data is generated based on conditional embedding. According to an embodiment of the present disclosure, the introduction of multimodal learning enables the model to process not only conventional simple data, but also more complex data. In this way, the generated multimodal tabular data can be closer to the real scene, and the representation ability of the output is improved. At the same time, the tabular data 111 and the multimodal data 112 are encoded separately, so that models of different capacities are used for data of different levels, saving computing and storage resources. In the following, an example implementation of multimodal tabular data generation according to an embodiment of the present disclosure will be given.
[0038] Figure 3 is a diagram showing the architecture of a multimodal table data generation model 300 according to an embodiment of the present disclosure. Figure 3 As shown in , the multimodal table data generation model 300 includes a multimodal converter 310, a generator 320 and a discriminator 330. In addition, Figure 3 Schematically, an input 110 including tabular data 111 and multimodal data 112 is also shown. It should be noted that the components included in the multimodal tabular data generation model 300 are exemplary and non-limiting. The multimodal tabular data generation model 300 may include more or fewer components, and the disclosure does not limit this.
[0039] According to an embodiment of the present disclosure, the input 110 is input to the multimodal converter 310 in the multimodal table data generation model 300 for conversion. Figure 3As exemplarily shown in , input 110 includes tabular data 111, which includes category data and numerical data. By way of example, the category data includes "ID", "magnetic resonance imaging (MRI)", and "evaluation", and the numerical data "1172931" belongs to the "ID" category for identifying the data item (this row of the table). Input 110 also includes multimodal data 112, which includes image data 113 and text data 114. By way of example, the image data 113 includes an MRI image for the brain, and the text data includes an evaluation. Data in different columns in the table may be associated. For example, ID "1172931" identifies the corresponding MRI image and evaluation, or the evaluation corresponding to an MRI image showing a normal brain is "appears normal". It should be understood that for ease of illustration, only one data item (a row of the table) is shown here, but the table may include more data items.
[0040] According to an embodiment of the present disclosure, the multimodal converter 310 included in the multimodal table data generation model 300 processes the table data 111 and the multimodal data 112 in the input 110 respectively to generate a conditional embedding 311. The generated conditional embedding 311 constrains the generation process performed by the generator 320 to control the mode of the output. In the following, the conversion process performed by the multimodal converter 310 will be described in further detail.
[0041] According to an embodiment of the present disclosure, the generator 320 included in the multimodal table data generation model 300 can generate multimodal table data based on the conditional embedding 311 generated by the multimodal converter 310. The generated multimodal table data can be presented in the form of a table, for example, in the form of Figure 3 The multimodal table data may be generated based on the noise data 312. The noise data 312 may include a series of noise data that obeys a Gaussian distribution, i.e., Z~N(0,1), where Z refers to the series of noise data. In the following, the generation process performed by the generator 320 will be described in further detail.
[0042] According to an embodiment of the present disclosure, the discriminator 330 included in the multimodal table data generation model 300 can discriminate the generated multimodal table data to distinguish positive samples from negative samples. The generated multimodal table data may include multiple columns and multiple rows. In the case where the columns of the table correspond to the categories and the rows correspond to the data items, the discriminator 330 can discriminate for each of the multiple rows. In other words, each row is regarded as a data item, and the discriminator 330 can discriminate based on the correlation of the data in each of the generated data items. Conversely, it is also feasible when the rows of the table correspond to the categories and the columns correspond to the data items. In this case, the discriminator 330 can discriminate for each of the multiple columns.
[0043] In one example, if a generated data item includes an MRI image showing a normal brain and includes an evaluation that "looks normal", then the data item can be identified as a positive sample. Conversely, if a generated data item includes an MRI image showing a normal brain but includes an evaluation that "looks abnormal", then the data item can be identified as a negative sample. The sample quality of the generated multimodal table data can be determined based on the discrimination result of the discriminator 330. For example, the sample quality of the multimodal table data can be determined based on the ratio between positive samples and negative samples in the generated multimodal table data. Figure 4 To describe the multimodal conversion process according to an embodiment of the present disclosure.
[0044] Figure 4 is a diagram showing a multi-modal conversion process according to an embodiment of the present disclosure. Figure 4 As shown in FIG. 4 , the multimodal converter 310 includes a converter 410, a multilayer perceptron 420, and an aggregator 430. The multimodal data 112 and the tabular data 111 may be input to the converter 410 and the multilayer perceptron 420 included in the multimodal converter 310, respectively.
[0045] According to an embodiment of the present disclosure, the transformer-based model 410 is a model with a large model capacity and strong processing capability to process complex data. The transformer-based model 410 may include a bidirectional encoder representation from transformers (BERT) model to process text data 114. BERT is a transformer-based language model trained on a large corpus of text data that can encode input text. The input text can be represented as a sequence of text tokens, which are then encoded into a text embedding h textThe BERT model is tuned for task-specific data to learn task-specific representations of text. The text encoder can be expressed as follows (1):
[0046] h text =BERT(x text θ BERRT ) (1)
[0047] where x text represents the input text, BERT represents the above transformer-based text encoder, and θ BERT Represents the tuned parameters of the BERT model.
[0048] According to an embodiment of the present disclosure, the converter-based model 410 may include a convolutional neural network CNN (such as a residual neural network ResNet) to process the image data 113. Image features of the input image may be extracted, and the extracted image features may be encoded as an image embedding h image CNN is tuned for task-specific data to learn task-specific representations of images. The image encoder can be expressed as follows:
[0049] h image =ResNet(x image θ ResNet ) (2)
[0050] where x image represents the input image, ResNet represents the CNN-based image encoder, and θ ResNet Represents the adjusted parameters of the ResNet model. The above text embedding and image embedding h image May have a predetermined length.
[0051] According to an embodiment of the present disclosure, after obtaining the text embedding h text and image embedding h image Afterwards, they can be concatenated to form the multimodal embedding h multi It should be noted that the present disclosure does not limit the connection mechanism. multi It can be expressed by the following formula (3):
[0052] h multi =Concatenate(h text ,h image ) (3)
[0053] Conatenate means any text embedding text With image embedding h image The mechanism of connection, such as splicing.
[0054] In some embodiments, the multimodal embedding h can be made to multi Focus on the interesting parts of the text data 114 and the image data 113, such as the most important or effective parts. The focusing process can be expressed by the following formula (4):
[0055] h multi =MultiHeadSelfAttention(h multi θ attention ) (4)
[0056] Where MultiHeadSelfAttention represents the multi-head self-attention mechanism, and θ attention Represents the parameters of the mechanism.
[0057] According to an embodiment of the present disclosure, the model 420 based on a multilayer perceptron (MLP) is a relatively small model. The MLP-based model can use one-hot encoding or normalization to convert numerical values into embeddings. These embeddings are then passed through a fully connected layer to generate the final embedding. The conversion process of numerical embedding can be represented by the following formula (5):
[0058] h num =MLP(Embedding(x num );θ MLP ) (5)
[0059] where x num represents the input value, Embedding represents the embedding function, MLP represents the fully connected layer, and θ MLP represents the parameters of the MLP.
[0060] According to an embodiment of the present disclosure, the MLP-based model 420 can use an embedding layer to convert the input category into an embedding. These embeddings are then passed through a fully connected layer to generate the final embedding. The conversion process of the category embedding can be represented by the following formula (6):
[0061] h cat =MLP(Embedding(x cat );θ MLP ) (6)
[0062] where x cat represents the input category, Embedding represents the embedding function, MLP represents the fully connected layer, and θ MLP represents the parameters of the MLP.
[0063] In order to effectively combine the embeddings from the transformer-based model 410 and the MLP-based model 420, the aggregator 430 according to an embodiment of the present disclosure may aggregate the multimodal embedding and the table embedding including the numerical embedding and the category embedding to form the conditional embedding 311. The aggregator may include a simple MLP, a complex neural network (such as a transformer, etc.), and a traditional machine learning algorithm (such as a random forest, a support vector machine, etc.). The aggregation process may be represented by the following equation (7):
[0064] h final =Aggregator(h multi ,h num ,h cat θ agg ) (7)
[0065] where h final Indicates conditional embedding, Aggregator indicates aggregation function, h num represents numerical embedding, h cat represents the category embedding, and θ agg Represents the parameters of the aggregator.
[0066] According to an embodiment of the present disclosure, the aggregator 430 can perform the aggregation process based on any one of 1) an attention-based aggregation mechanism, 2) a gated aggregation mechanism, or 3) a graph-based aggregation mechanism or a combination thereof. The attention-based aggregation mechanism can weigh the importance of different parts of multimodal embedding, numerical embedding, and category embedding when generating conditional embedding 311. The gated aggregation mechanism can control the flow of information between different parts of multimodal embedding, numerical embedding, and category embedding. The graph-based aggregation mechanism can model the relationship between different parts of multimodal embedding, numerical embedding, and category embedding. The aggregation process can also be performed using any combination of these aggregation mechanisms, which is also referred to as a hierarchical aggregation mechanism. The hierarchical aggregation mechanism can combine multimodal embedding, numerical embedding, and category embedding, which includes multiple levels of aggregation, each level focusing on different aspects of the data.
[0067] Figure 5 is a diagram showing a process of generating multimodal table data according to an embodiment of the present disclosure. Figure 5 As shown in , the generator 320 includes an image decoder 510, a text decoder 520, and a table decoder 530. The image decoder 510, the text decoder 520, and the table decoder 530 can each generate images, texts, and numerical values and categories based on the conditional embedding 311 to ultimately generate multimodal table data without splitting the conditional embedding 311.
[0068] Before generating the desired multimodal table data, the generator 320 needs to be trained. According to an embodiment of the present disclosure, the parameter set of the generator 320 can be adjusted by minimizing the loss function based on the difference between the multimodal table data and the multimodal data and table data used to generate the multimodal table data, that is, the parameter sets of the image decoder 510, the text decoder 520, and the table decoder 530 included in the generator 320 are adjusted. When the loss satisfies certain conditions, such as being lower than a predetermined threshold, it can be determined that the generator 320 has been fully trained, that is, the image decoder 510, the text decoder 520, and the table decoder 530 included in the generator 320 have been fully trained.
[0069] According to an embodiment of the present disclosure, multimodal table data includes multiple columns. Columns can correspond to categories. Components included in generator 320 can correspond to categories. When the categories include image categories, text categories, and numerical categories, the image decoder 510, text decoder 520, and table decoder 530 included in generator 320 can correspond to image categories, text categories, and numerical categories, respectively. It should be understood that this is only exemplary, and there are other corresponding ways.
[0070] According to an embodiment of the present disclosure, the image decoder 510 may include a generative adversarial network (GAN) or a variational autoencoder (VAE) to embed the conditional h final Generate an image. The process can be expressed by the following formula (8):
[0071] I gen =ImageDecoder(h final θ img ) (8)
[0072] Among them I gen represents the generated image, ImageDecoder represents the image decoding function, and θ img Represents the parameters of the image decoder.
[0073] According to an embodiment of the present disclosure, the text decoder 520 may include GPT-2 or BERT to embed n final Generate text, the process can be expressed by the following formula (9):
[0074] T gen =TextDecoder(h final θ txt ) (9)
[0075] Where T gen represents the generated text, TextDecoder represents the text decoding function, and θ txtRepresents the parameters of the text decoder.
[0076] According to an embodiment of the present disclosure, the table decoder 530 may include an MLP to embed h final Generate numerical values and categories. The process can be expressed by the following formula (10):
[0077] X gen = TableDecoder(h final θ tab ) (10)
[0078] Where X gen represents the generated values and categories, TableDecoder represents the table decoding function, and θ tab Represents the parameters of the table decoder.
[0079] Figure 6 An example 600 of generated multimodal table data according to an embodiment of the present disclosure is shown. The generated multimodal table data is Figure 6 The table shown in . The table includes three categories, namely "number", "traffic sign" and "evaluation", where "traffic sign" and "evaluation" correspond to image data and text data respectively. For the convenience of illustration, Figure 6 Two data items are shown, row numbered 1 and row numbered 2.
[0080] exist Figure 6 In Example 600, the traffic sign in the row numbered 1 indicates that no passage is allowed, and is evaluated as "no passage". Therefore, the data item 1 can be regarded as a positive sample. The traffic sign in the row numbered 2 indicates that no passage is allowed, and is evaluated as "passable". Therefore, the data item 2 can be regarded as a negative sample. It should be understood that the number of generated multimodal table data may be greater than the number of input data used to generate the multimodal table data, or may be equal to or less than the number of input data used to generate the multimodal table data, and the present disclosure does not limit this.
[0081] Through the embodiments of the present disclosure, although the generated multimodal tabular data is different from the input data, it can reflect the characteristics of the input data, such as pattern, distribution, correlation, etc. By sharing the generated multimodal tabular data, the data privacy of the original data can be effectively protected. In addition, the generated multimodal tabular data can also be used for other purposes such as data enhancement, model training, etc. Through the embodiments of the present disclosure, tabular data generation can be combined with multimodal learning, which improves the representation ability of the generated data, so that the generated data has sufficient information to describe the real scene. At the same time, it can reduce the model configuration cost and save computing and storage resources.
[0082] Figure 7 8 shows a schematic block diagram of an example device 700 that can be used to implement some embodiments of the present disclosure. Figure 7 As shown in , the device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 or loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0083] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0084] The various processes and processing described above, such as method 200, may be performed by processing unit 701. For example, in some embodiments, method 200 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by CPU 701, one or more actions of method 200 described above may be performed.
[0085] The present disclosure may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.
[0086] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive lists) of computer readable storage medium include: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disk read-only memories (CD-ROM), digital versatile disks (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or convex structures in grooves on which instructions are stored, and any suitable combination thereof. Computer readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (e.g., light pulses by optical fiber cables), or electrical signals transmitted by wires.
[0087] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0088] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.
[0089] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions.
[0090] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0091] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0092] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.
[0093] The embodiments of the present disclosure are described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A method for generating data, comprising: Obtaining a multimodal embedding by encoding multimodal data, wherein the multimodal data includes text data and image data; Obtaining a table embedding by encoding table data associated with the multimodal data; generating a conditional embedding based on the multimodal embedding and the table embedding; as well as Based on the conditional embedding, multimodal tabular data is generated.
2. The method of claim 1, wherein obtaining the multimodal embedding comprises: representing the text data as a sequence of text tokens for the text data; Converting the text token sequence into a text embedding by encoding the text token sequence; extracting image features for the image data from the image data; as well as The image features are converted into image embeddings by encoding the image features.
3. The method according to claim 2, wherein: The conversion from the text token sequence to the text embedding and the conversion from the image features to the image embedding are performed by a transformer-based model; and The text embedding and the image embedding have a predetermined length.
4. The method of claim 2, wherein obtaining the multimodal embedding further comprises: Obtaining the multimodal embedding by concatenating the text embedding and the image embedding; as well as The focused multimodal embedding is obtained by focusing the multimodal embedding on the interesting parts in the text data and the image data.
5. The method of claim 1 , wherein the tabular data comprises numerical data and categorical data, and wherein obtaining the tabular embedding comprises: converting the numerical data into a numerical embedding; as well as Convert the category data into category embeddings, The conversion from the numerical data to the numerical embedding and the conversion from the category data to the category embedding are performed by a model based on a multi-layer perceptron.
6. The method of claim 1, wherein generating the conditional embedding comprises aggregating the multimodal embedding and the table embedding into the conditional embedding, the aggregating process being by any one or any combination of the following: Attention-based aggregation mechanism; Gated aggregation mechanisms; or Graph-based aggregation mechanism.
7. The method according to claim 1, wherein the multimodal table data comprises a plurality of columns and a plurality of rows, so the method further comprises: For each of the plurality of rows, identifying positive samples and negative samples in the multimodal tabular data based on correlations between the plurality of columns; as well as Based on a ratio between the positive samples and the negative samples, a sample quality of the multimodal tabular data is indicated.
8. The method according to claim 7, wherein: Generating the multimodal table data is based on noise data, the noise data comprising a series of noise data obeying a Gaussian distribution, and The multimodal table data is generated by a generator, and the method further includes: A set of parameters of the generator is adjusted by minimizing a loss function based on the multimodal tabular data and a difference between the multimodal data and the tabular data.
9. The method according to claim 8, wherein: The generator includes an image decoder, a text decoder and a table decoder, and The image decoder, the text decoder, and the table decoder each correspond to one column among the plurality of columns.
10. The method according to claim 9, further comprising: The image decoder generates an image based on the conditional embedding; The text decoder generates text based on the conditional embedding; as well as and the table decoder embeds the conditions to generate values and categories.
11. A device for generating data, comprising: processor; as well as a memory coupled to the processor and storing instructions that, when executed by the processor, cause the device to perform actions comprising: Obtaining a multimodal embedding by encoding multimodal data, wherein the multimodal data includes text data and image data; Obtaining a table embedding by encoding table data associated with the multimodal data; generating a conditional embedding based on the multimodal embedding and the table embedding; and Based on the conditional embedding, multimodal tabular data is generated.
12. The apparatus of claim 11, wherein obtaining the multimodal embedding comprises: representing the text data as a sequence of text tokens for the text data; Converting the text token sequence into a text embedding by encoding the text token sequence; extracting image features for the image data from the image data; as well as The image features are converted into image embeddings by encoding the image features.
13. The apparatus of claim 12, wherein: The conversion from the text token sequence to the text embedding and the conversion from the image features to the image embedding are performed by a transformer-based model; and The text embedding and the image embedding have a predetermined length.
14. The apparatus of claim 12, wherein obtaining the multimodal embedding further comprises: Obtaining the multimodal embedding by concatenating the text embedding and the image embedding; as well as The focused multimodal embedding is obtained by focusing the multimodal embedding on the interesting parts in the text data and the image data.
15. The apparatus of claim 11, wherein the tabular data comprises numerical data and categorical data, and wherein obtaining the tabular embedding comprises: converting the numerical data into a numerical embedding; as well as Convert the category data into category embeddings, The conversion from the numerical data to the numerical embedding and from the category data to the category embedding is performed by a multi-layer perceptron based model.
16. The apparatus of claim 11, wherein generating the conditional embedding comprises aggregating the multimodal embedding and the table embedding into the conditional embedding, the aggregating process being by any one or any combination of the following: Attention-based aggregation mechanism; Gated aggregation mechanisms; or Graph-based aggregation mechanism.
17. The apparatus according to claim 11, wherein the multimodal table data comprises a plurality of columns and a plurality of rows, so the method further comprises: For each of the plurality of rows, identifying positive samples and negative samples in the multimodal tabular data based on correlations between the plurality of columns; as well as Based on a ratio between the positive samples and the negative samples, a sample quality of the multimodal tabular data is indicated.
18. The apparatus of claim 17, wherein: Generating the multimodal table data is based on noise data, the noise data comprising a series of noise data obeying a Gaussian distribution, and The multimodal table data is generated by a generator, and the method further includes: A set of parameters of the generator is adjusted by minimizing a loss function based on the multimodal tabular data and a difference between the multimodal data and the tabular data.
19. The apparatus according to claim 18, wherein the generator comprises an image decoder, a text decoder, and a table decoder, and each of the image decoder, the text decoder, and the table decoder corresponds to one of the plurality of columns, and the actions further comprise: The image decoder generates an image based on the conditional embedding; The text decoder generates text based on the conditional embedding; as well as and the table decoder embeds the conditions to generate values and categories.
20. A computer program product tangibly stored on a non-transitory computer-readable storage medium and comprising computer-executable instructions which, when executed, cause a computer to perform the method according to any one of claims 1-10.
