Data generation method and apparatus, device, and storage medium

Abstract

This invention belongs to the field of computer technology and discloses a data generation method, apparatus, device, and storage medium. The method includes: acquiring first-dimensional data to be identified; segmenting the first-dimensional data to be identified to obtain multiple sub-data to be identified; extracting feature information from each sub-data to obtain a feature sequence; sequentially inputting the multiple feature sequences into a target data generation model to obtain a target sub-sequence output by the target data generation model; and generating second-dimensional data based on the target sub-sequence. By processing the data in the input data generation model in the above manner, multiple sets of sub-data to be identified are obtained, enabling the model to analyze detailed parts and improving the accuracy of image content description.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data generation method, apparatus, device, and storage medium. Background Technology

[0002] Image description technology refers to using images as input and employing mathematical models and calculations to enable computers to output natural language descriptions of the corresponding images, giving computers the ability to "tell stories from pictures".

[0003] However, current methods for automatically generating text descriptions by machines often require a large number of labeled samples and suffer from problems such as monotonous text styles and inaccurate descriptions of image content.

[0004] Correspondingly, current methods for generating images from text suffer from problems such as inaccurate image content, unnatural details, and errors.

[0005] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this invention is to provide a data generation method, apparatus, device, and storage medium, aiming to solve the technical problems of current machine-generated text description methods, such as the monotonous text style and inaccurate image content descriptions.

[0007] To achieve the above objectives, the present invention provides a data generation method, the method comprising the following steps:

[0008] Obtain the first dimension of the data to be identified;

[0009] The data to be identified in the first dimension is segmented to obtain multiple sub-data to be identified;

[0010] Feature information is extracted from each of the sub-data to be identified to obtain a feature sequence;

[0011] Multiple feature sequences are sequentially input into the target data generation model to obtain the target subsequence output by the target data generation model;

[0012] The second dimension data is generated based on the target subsequence.

[0013] Optionally, the target data generation model includes a decoder and an encoder, and the step of sequentially inputting multiple feature sequences into the target data generation model to obtain the target subsequence output by the target data generation model includes:

[0014] Multiple feature sequences are sequentially input into the encoder to obtain several target feature sequences of fixed length;

[0015] Determine the first target feature sequence at the current position and the second target feature sequence at the previous position;

[0016] The first target feature sequence and the second target feature sequence are input into the decoder to obtain the target subsequence output by the decoder.

[0017] Optionally, the step of inputting the first target feature sequence and the second target feature sequence into the decoder to obtain the target subsequence output by the decoder includes:

[0018] The first target feature sequence and the second target feature sequence are input into the decoder to obtain multiple candidate feature sequences output by the decoder;

[0019] Each of the candidate feature sequences is concatenated with the second target feature sequence to obtain multiple candidate sub-sequences;

[0020] Calculate the confidence level corresponding to each of the candidate sub-sequences;

[0021] Select the candidate subsequence with the highest confidence as the target subsequence.

[0022] Optionally, before obtaining the first dimension of data to be identified, the method further includes:

[0023] Acquire joint pair data and train an encoder based on the joint pair data to obtain an initial encoder and a fusion encoder;

[0024] The encoder of the data generation model is initialized using the initial encoder, and the decoder of the data generation model is initialized using the fusion encoder to obtain the initial data generation model.

[0025] The initial data generation model is trained based on the first dimension feature information and the second dimension annotation information to obtain the trained target data generation model.

[0026] Optionally, the joint pair data includes image sample data and text sample data with consistent content;

[0027] The process of training the encoder based on the joint pair data to obtain the initial encoder and the fused encoder includes:

[0028] The image sample data is segmented into image blocks according to a preset pixel size to obtain multiple image feature sequences;

[0029] The image feature sequence is input into the first encoder to obtain the first feature output by the first encoder;

[0030] The text sample data is segmented to obtain multiple text feature sequences;

[0031] The text feature sequence is input into the second encoder to obtain the second feature output by the second encoder;

[0032] The first feature and the second feature are input into the image-text fusion encoder;

[0033] The first encoder, the second encoder, and the image-text fusion encoder are jointly trained to obtain the initial encoder and the fusion encoder.

[0034] Optionally, the first dimension of data to be identified includes image data, and the second dimension of data includes text data.

[0035] Optionally, the step of segmenting the data to be identified in the first dimension to obtain multiple sub-data to be identified includes:

[0036] The image data is segmented into image blocks according to a preset pixel size to obtain multiple sub-data to be identified.

[0037] Optionally, generating the second-dimensional data based on the target subsequence includes:

[0038] Translate the target subsequence into sentence data;

[0039] Text data is generated based on several of the aforementioned sentence data.

[0040] Optionally, the data to be identified in the first dimension includes text data, and the data in the second dimension includes image data.

[0041] Optionally, the step of segmenting the data to be identified in the first dimension to obtain multiple sub-data to be identified includes:

[0042] The text data is segmented into word data to obtain multiple sub-data to be identified.

[0043] Optionally, generating the second-dimensional data based on the target subsequence includes:

[0044] The target subsequence is input into the image decoding model to obtain image data.

[0045] Furthermore, to achieve the above objectives, the present invention also proposes a data generation apparatus, the data generation apparatus comprising:

[0046] The acquisition module is used to acquire the data to be identified in the first dimension;

[0047] The segmentation module is used to segment the data to be identified in the first dimension to obtain multiple sub-data to be identified;

[0048] The extraction module is used to extract feature information from each of the sub-data to be identified to obtain a feature sequence;

[0049] A generation module is used to sequentially input multiple feature sequences into a target data generation model to obtain a target subsequence output by the target data generation model.

[0050] The generation module is also used to generate second-dimensional data based on the target subsequence.

[0051] Optionally, the target data generation model includes a decoder and an encoder. The generation module is further configured to input multiple feature sequences sequentially into the encoder to obtain several target feature sequences of fixed length, determine the first target feature sequence at the current position and the second target feature sequence at the previous position, and input the first target feature sequence and the second target feature sequence into the decoder to obtain the target subsequence output by the decoder.

[0052] Optionally, the generation module is further configured to input the first target feature sequence and the second target feature sequence into the decoder, obtain multiple candidate feature sequences output by the decoder, concatenate each candidate feature sequence with the second target feature sequence to obtain multiple candidate sub-sequences, calculate the confidence level corresponding to each candidate sub-sequence, and select the candidate sub-sequence with the highest confidence level as the target sub-sequence.

[0053] Optionally, the device further includes a training module;

[0054] The training module is used to acquire joint pair data, train an encoder based on the joint pair data to obtain an initial encoder and a fusion encoder, initialize the encoder of the data generation model using the initial encoder, and initialize the decoder of the data generation model using the fusion encoder to obtain an initial data generation model. The initial data generation model is then trained based on the first dimension feature information and the second dimension annotation information to obtain a trained target data generation model.

[0055] Optionally, the joint pair data includes image sample data and text sample data with consistent content;

[0056] The training module is further configured to segment the image sample data into image blocks according to a preset pixel size to obtain multiple image feature sequences, input the image feature sequences into a first encoder to obtain a first feature output by the first encoder, perform word segmentation on the text sample data to obtain multiple text feature sequences, input the text feature sequences into a second encoder to obtain a second feature output by the second encoder, input the first feature and the second feature into an image-text fusion encoder, and perform joint training on the first encoder, the second encoder and the image-text fusion encoder to obtain an initial encoder and a fusion encoder.

[0057] Optionally, the first dimension of data to be identified includes image data, and the second dimension of data includes text data.

[0058] Optionally, the segmentation module is further configured to segment the image data into image blocks according to a preset pixel size to obtain multiple sub-data to be identified.

[0059] Furthermore, to achieve the above objectives, the present invention also proposes a data generation device, which includes: a memory, a processor, and a data generation program stored in the memory and executable on the processor, the data generation program being configured to implement the data generation method as described above.

[0060] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing a data generation program, which, when executed by a processor, implements the data generation method as described above.

[0061] This invention obtains first-dimensional data to be identified; segments this first-dimensional data to obtain multiple sub-data to be identified; extracts feature information from each sub-data to obtain feature sequences; inputs these feature sequences sequentially into a target data generation model to obtain a target sub-sequence output by the model; and generates second-dimensional data based on the target sub-sequence. By segmenting the data in the input data generation model, the model can analyze detailed parts, improving its recognition accuracy, further enhancing the accuracy of image content description, and enriching text styles. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of the structure of the data generation device of the hardware operating environment involved in the embodiments of the present invention;

[0063] Figure 2 This is a flowchart illustrating the first embodiment of the data generation method of the present invention;

[0064] Figure 3This is a flowchart illustrating the second embodiment of the data generation method of the present invention;

[0065] Figure 4 This is a schematic diagram of the encoder training process of the data generation method of the present invention;

[0066] Figure 5 This is a flowchart illustrating the third embodiment of the data generation method of the present invention;

[0067] Figure 6 This is a schematic diagram illustrating the text data generation method of the present invention.

[0068] Figure 7 This is a flowchart illustrating the fourth embodiment of the data generation method of the present invention;

[0069] Figure 8 This is a schematic diagram illustrating the image data generation method of the present invention.

[0070] Figure 9 This is a structural block diagram of the first embodiment of the data generation device of the present invention.

[0071] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0072] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0073] Reference Figure 1 , Figure 1 This is a schematic diagram of the data generation device structure of the hardware operating environment involved in the embodiments of the present invention.

[0074] like Figure 1As shown, the data generation device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0075] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the data generation device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0076] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a data generation program.

[0077] exist Figure 1 In the data generation device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the data generation device of the present invention can be set in the data generation device, and the data generation device calls the data generation program stored in the memory 1005 through the processor 1001 and executes the data generation method provided in the embodiment of the present invention.

[0078] This invention provides a data generation method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the data generation method of the present invention.

[0079] In this embodiment, the data generation method includes the following steps:

[0080] Step S10: Obtain the first dimension of data to be identified.

[0081] It is understood that the execution entity in this embodiment is a data generation device, which can be a computer, server, processor, or other devices with the same or similar functions. This embodiment does not limit this. The first dimension of the data to be identified in this embodiment is image data or text data to be identified.

[0082] Step S20: The data to be identified in the first dimension is segmented to obtain multiple sub-data to be identified.

[0083] It should be noted that for image data, the image data is divided into 16*16 image blocks, and the two-dimensional image data is unfolded into a one-dimensional image block sequence. For text data, the text data is segmented into word data.

[0084] Step S30: Extract feature information from each of the sub-data to be identified to obtain a feature sequence.

[0085] It should be understood that, for image data, the image feature sequence is obtained by calculating the image patch sequence through a convolutional network. For text data, the word vector generation model is used to determine the word vectors corresponding to the word data, thereby obtaining the feature sequence. The word vector generation model can be the word2vec model, or other models, which are not limited in this embodiment.

[0086] Step S40: Input the multiple feature sequences sequentially into the target data generation model to obtain the target subsequence output by the target data generation model.

[0087] It should be noted that the target data generation model is a model that has undergone preliminary training on joint data and a second-stage training using rigorously labeled images and image description data. In the specific implementation, an image encoder, a text encoder, a first image-text fusion encoder, and a second image-text fusion encoder are trained using a large amount of image-text pair data. The image encoder is used to initialize the encoder of the text data generation model, and the first image-text fusion encoder is used to initialize the decoder of the text data generation model. Rigorously labeled images are used as model input, and image description data is used as the model prediction result. The second-stage training of the text data generation model yields the text data generation model. Similarly, the text encoder is used to initialize the encoder of the image data generation model, and the second image-text fusion encoder is used to initialize the decoder of the image data generation model. Rigorously labeled image description data is used as model input, and images are used as the model prediction result. The second-stage training of the image data generation model yields the image data generation model.

[0088] Specifically, the target data generation model includes a decoder and an encoder. Step S40 includes: inputting multiple feature sequences sequentially into the encoder to obtain several target feature sequences of fixed length; determining the first target feature sequence at the current position and the second target feature sequence at the previous position; inputting the first target feature sequence and the second target feature sequence into the decoder to obtain the target subsequence output by the decoder.

[0089] It should be understood that by using an encoder to encode the feature sequence into several standard target feature sequences of fixed length, when the decoder decodes the target feature sequence at the current position, this embodiment also inputs the preceding target feature sequence into the decoder, so that the decoder generates the target subsequence based on the correlation between the preceding and following feature sequences, thereby improving the accuracy of the target data generation model.

[0090] Specifically, the step of inputting the first target feature sequence and the second target feature sequence into the decoder to obtain the target subsequence output by the decoder includes: inputting the first target feature sequence and the second target feature sequence into the decoder to obtain multiple candidate feature sequences output by the decoder; concatenating each of the candidate feature sequences with the second target feature sequence to obtain multiple candidate subsequences; calculating the confidence level corresponding to each candidate subsequence; and selecting the candidate subsequence with the highest confidence level as the target subsequence.

[0091] It should be noted that the decoder outputs n candidate feature sequences with the highest probability, specifically n=5, which together with the preceding target feature sequence form candidate subsequences. The confidence of the candidate subsequences is calculated, and the candidate subsequence with the highest confidence is output as the target subsequence. The target data generation model is then used to perform step-by-step parsing until the sequence length reaches the maximum value, at which point the decoding ends.

[0092] Step S50: Generate second-dimensional data based on the target subsequence.

[0093] It should be understood that the target subsequence output by the text data generation model is translated into text data using a dictionary; and the target subsequence output by the image data generation model is generated into image data using a preset image decoding model.

[0094] This embodiment acquires the first dimension of data to be identified; segments the first dimension of data to be identified to obtain multiple sub-data to be identified; extracts feature information from each sub-data to obtain a feature sequence; inputs the multiple feature sequences sequentially into the target data generation model to obtain the target sub-sequence output by the target data generation model; and generates the second dimension of data based on the target sub-sequence. By segmenting the data in the input data generation model in the above manner, the model can parse detailed parts, improving the model's recognition accuracy, further enhancing the accuracy of image content description, and enriching text styles.

[0095] refer to Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the data generation method of the present invention.

[0096] Based on the first embodiment described above, the data generation method of this embodiment further includes, before step S10:

[0097] Step S101: Obtain joint pair data and train an encoder based on the joint pair data to obtain an initial encoder and a fusion encoder.

[0098] It should be understood that, in specific implementations, the training of text data generation models and image data generation models can be carried out simultaneously, training text encoders, image encoders, and one or two image-text fusion encoders.

[0099] Further, the joint pair data includes image sample data and text sample data with consistent content; the step of training the encoder based on the joint pair data to obtain the initial encoder and the fusion encoder includes: segmenting the image sample data into image blocks according to a preset pixel size to obtain multiple image feature sequences; inputting the image feature sequences into a first encoder to obtain a first feature output by the first encoder; performing word segmentation on the text sample data to obtain multiple text feature sequences; inputting the text feature sequences into a second encoder to obtain a second feature output by the second encoder; inputting the first feature and the second feature into the image-text fusion encoder; and jointly training the first encoder, the second encoder, and the image-text fusion encoder to obtain the initial encoder and the fusion encoder.

[0100] It should be noted that, referring to Figure 4 , Figure 4This is a schematic diagram of the encoder training process of the data generation method of the present invention; taking image-text pairs as input, the image is divided into 16*16 image blocks, the two-dimensional image blocks are unfolded into a one-dimensional image block sequence, and the image block sequence is convolved to obtain an image feature sequence, which is then input into the encoder; the text is segmented into text tokens to obtain a text token sequence, which is then input into the text encoder; the features output by the text encoder and the image encoder are respectively input into the image-text fusion encoder A and the image-text fusion encoder B, and the four encoders are trained based on the output to obtain stable encoder parameters for the second stage of model training.

[0101] Step S102: Initialize the encoder of the data generation model using the initial encoder, and initialize the decoder of the data generation model using the fusion encoder to obtain the initial data generation model.

[0102] Step S103: Train the initial data generation model based on the first dimension feature information and the second dimension annotation information to obtain the trained target data generation model.

[0103] It should be noted that the strictly labeled image-text pairs are used as the input and target of the encoder and decoder. The parameters of the image encoder and fusion encoder obtained in the first stage are used to initialize the parameters of the encoder and decoder of the text data generation model in the second stage, and a new text data generation model is trained. Similarly, the parameters of the text encoder and fusion encoder obtained in the first stage are used to initialize the parameters of the encoder and decoder of the image data generation model in the second stage, and a new image data generation model is trained.

[0104] This embodiment obtains joint pair data and trains an encoder based on the joint pair data to obtain an initial encoder and a fusion encoder. The initial encoder is used to initialize the encoder of the data generation model, and the fusion encoder is used to initialize the decoder of the data generation model, resulting in an initial data generation model. The initial data generation model is then trained using the first-dimensional feature information and the second-dimensional annotation information to obtain a trained target data generation model. Through this method, a reliable data generation model can be trained using a small amount of joint pair data, improving the model's recognition accuracy.

[0105] refer to Figure 5 , Figure 5 This is a flowchart illustrating the third embodiment of the data generation method of the present invention.

[0106] Based on the first embodiment described above, in the data generation method of this embodiment, the first dimension of the data to be identified includes image data, and the second dimension of the data includes text data.

[0107] Step S20 includes:

[0108] Step S201: Divide the image data into image blocks according to the preset pixel size to obtain multiple sub-data to be identified.

[0109] Step S50 includes:

[0110] Step S501: Translate the target subsequence into sentence data;

[0111] Step S502: Generate text data based on the aforementioned sentence data.

[0112] It should be understood that the default pixel size is 16*16. (Refer to...) Figure 6 , Figure 6 This is a schematic diagram illustrating the text data generation method of the present invention. The text data generation process is as follows: The input image is divided into 16*16 image blocks. The two-dimensional image blocks are unfolded into a one-dimensional image block sequence. The image block sequence is convolved to obtain an image feature sequence. The image feature sequence is then input into the encoder and decoder sequentially. The decoder decodes character by character. When decoding the image token sequence at the current position, the preceding image token sequence should also be input. The decoder outputs the five tokens with the highest probability as candidate tokens, which, together with the preceding image token sequence, form a subsequence. The confidence level of each subsequence is calculated, and the subsequence with the highest confidence level is output as the target subsequence. Decoding ends when the sequence length reaches its maximum value or the end token is encountered. Based on a dictionary, the tokens are translated into sentences to form text data.

[0113] This embodiment acquires image data; segments the image data into image blocks according to a preset pixel size to obtain multiple sub-data to be identified; extracts feature information from each sub-data to obtain a feature sequence; inputs multiple feature sequences sequentially into a target data generation model to obtain a target sub-sequence output by the target data generation model; translates the target sub-sequence into sentence data; and generates text data based on several sentence data. Through the above method, the image data of the input data generation model is segmented, enabling the model to identify detailed parts, improving the model's recognition accuracy, further enhancing the accuracy of image content description, and enriching the text style.

[0114] refer to Figure 7 , Figure 7 This is a flowchart illustrating the fourth embodiment of the data generation method of the present invention.

[0115] Based on the first embodiment described above, in the data generation method of this embodiment, the first dimension of the data to be identified includes text data, and the second dimension of the data includes image data.

[0116] Step S20 includes:

[0117] Step S202: The text data is segmented into word data to obtain multiple sub-data to be identified.

[0118] Step S50 includes:

[0119] Step S503: Input the target subsequence into the image decoding model to obtain image data.

[0120] It should be noted that, referring to Figure 8 , Figure 8 This is a schematic diagram illustrating image data generation using the data generation method of the present invention. The image data generation process is as follows: The input text is segmented to obtain a text token sequence, which is then sequentially input into the encoder and decoder. The decoder decodes block by block. When decoding the text token sequence at the current position, the preceding text token sequence is also input. The decoder outputs the five tokens with the highest probability as candidate tokens, which, together with the preceding text token sequence, form a subsequence. The confidence level of the subsequence is calculated, and the subsequence with the highest confidence level is output as the target subsequence. Decoding ends when the sequence length reaches its maximum value. The obtained image block token sequence is then input into the image decoding model for image decoding to obtain the final image.

[0121] This embodiment acquires text data; segments the text data into word data to obtain multiple sub-data to be identified; extracts feature information from each sub-data to obtain a feature sequence; inputs multiple feature sequences sequentially into a target data generation model to obtain the target sub-sequence output by the target data generation model; and inputs the target sub-sequence into an image decoding model to obtain image data. Through the above method, word segmentation is performed on the text data input to the data generation model, enabling the model to parse detailed parts and improving the accuracy, naturalness, and detail richness of the image generated from the text.

[0122] Furthermore, this embodiment of the invention also proposes a storage medium storing a data generation program, which, when executed by a processor, implements the data generation method described above.

[0123] Since this storage medium adopts all the technical solutions of all the above embodiments, it has at least all the beneficial effects brought about by the technical solutions of the above embodiments, which will not be repeated here.

[0124] Reference Figure 9 , Figure 9 This is a structural block diagram of the first embodiment of the data generation device of the present invention.

[0125] like Figure 9 As shown, the data generation apparatus proposed in this embodiment of the invention includes:

[0126] Module 10 is used to acquire the data to be identified in the first dimension.

[0127] The segmentation module 20 is used to segment the data to be identified in the first dimension to obtain multiple sub-data to be identified.

[0128] The extraction module 30 is used to extract feature information from each of the sub-data to be identified to obtain a feature sequence.

[0129] The generation module 40 is used to input multiple feature sequences sequentially into the target data generation model to obtain the target subsequence output by the target data generation model.

[0130] The generation module 40 is also used to generate second-dimensional data based on the target subsequence.

[0131] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0132] This embodiment acquires the first dimension of data to be identified; segments the first dimension of data to be identified to obtain multiple sub-data to be identified; extracts feature information from each sub-data to obtain a feature sequence; inputs the multiple feature sequences sequentially into the target data generation model to obtain the target sub-sequence output by the target data generation model; and generates the second dimension of data based on the target sub-sequence. By segmenting the data in the input data generation model in the above manner, the model can parse detailed parts, improving the model's recognition accuracy, further enhancing the accuracy of image content description, and enriching text styles.

[0133] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0134] In addition, for technical details not described in detail in this embodiment, please refer to the data generation method provided in any embodiment of the present invention, which will not be repeated here.

[0135] In one embodiment, the target data generation model includes a decoder and an encoder. The generation module 40 is further configured to input multiple feature sequences sequentially into the encoder to obtain several target feature sequences of fixed length, determine the first target feature sequence at the current position and the second target feature sequence at the previous position, and input the first target feature sequence and the second target feature sequence into the decoder to obtain the target subsequence output by the decoder.

[0136] In one embodiment, the generation module 40 is further configured to input the first target feature sequence and the second target feature sequence into the decoder, obtain multiple candidate feature sequences output by the decoder, connect each candidate feature sequence to the second target feature sequence to obtain multiple candidate sub-sequences, calculate the confidence level corresponding to each candidate sub-sequence, and select the candidate sub-sequence with the highest confidence level as the target sub-sequence.

[0137] In one embodiment, the device further includes a training module;

[0138] The training module is used to acquire joint pair data, train an encoder based on the joint pair data to obtain an initial encoder and a fusion encoder, initialize the encoder of the data generation model using the initial encoder, and initialize the decoder of the data generation model using the fusion encoder to obtain an initial data generation model. The initial data generation model is then trained based on the first dimension feature information and the second dimension annotation information to obtain a trained target data generation model.

[0139] In one embodiment, the joint pair data includes image sample data and text sample data with consistent content;

[0140] The training module is further configured to segment the image sample data into image blocks according to a preset pixel size to obtain multiple image feature sequences, input the image feature sequences into a first encoder to obtain a first feature output by the first encoder, perform word segmentation on the text sample data to obtain multiple text feature sequences, input the text feature sequences into a second encoder to obtain a second feature output by the second encoder, input the first feature and the second feature into an image-text fusion encoder, and perform joint training on the first encoder, the second encoder and the image-text fusion encoder to obtain an initial encoder and a fusion encoder.

[0141] In one embodiment, the first dimension of data to be identified includes image data, and the second dimension of data includes text data.

[0142] In one embodiment, the segmentation module 20 is further configured to segment the image data into image blocks according to a preset pixel size to obtain multiple sub-data to be identified.

[0143] In one embodiment, the generation module 40 is further configured to translate the target subsequence into sentence data;

[0144] Text data is generated based on several of the aforementioned sentence data.

[0145] In one embodiment, the first dimension of data to be identified includes text data, and the second dimension of data includes image data.

[0146] In one embodiment, the segmentation module 20 is further configured to segment the text data into word data to obtain multiple sub-data to be identified.

[0147] In one embodiment, the generation module 40 is further configured to input the target subsequence into an image decoding model to obtain image data.

[0148] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0149] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0150] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0151] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A data generation method, characterized in that, The data generation method includes: Obtain the first dimension of the data to be identified; The data to be identified in the first dimension is segmented to obtain multiple sub-data to be identified; Feature information is extracted from each of the sub-data to be identified to obtain a feature sequence; Multiple feature sequences are sequentially input into the target data generation model to obtain the target subsequence output by the target data generation model; The second dimension data is generated based on the target subsequence, wherein the first dimension data to be identified and the second dimension data include text data or image data, and when the first dimension data to be identified includes text data, the second dimension data includes image data. Before obtaining the first dimension of data to be identified, the method further includes: Acquire joint pair data and train an encoder based on the joint pair data to obtain an initial encoder and a fusion encoder; The encoder of the data generation model is initialized using the initial encoder, and the decoder of the data generation model is initialized using the fusion encoder to obtain the initial data generation model. The initial data generation model is trained based on the first dimension feature information and the second dimension annotation information to obtain the trained target data generation model.

2. The data generation method as described in claim 1, characterized in that, The target data generation model includes a decoder and an encoder. The step of sequentially inputting multiple feature sequences into the target data generation model to obtain the target sub-sequence output by the target data generation model includes: Multiple feature sequences are sequentially input into the encoder to obtain several target feature sequences of fixed length; Determine the first target feature sequence at the current position and the second target feature sequence at the previous position; The first target feature sequence and the second target feature sequence are input into the decoder to obtain the target subsequence output by the decoder.

3. The data generation method as described in claim 2, characterized in that, The step of inputting the first target feature sequence and the second target feature sequence into the decoder to obtain the target sub-sequence output by the decoder includes: The first target feature sequence and the second target feature sequence are input into the decoder to obtain multiple candidate feature sequences output by the decoder; Each of the candidate feature sequences is concatenated with the second target feature sequence to obtain multiple candidate sub-sequences; Calculate the confidence level corresponding to each of the candidate sub-sequences; Select the candidate subsequence with the highest confidence as the target subsequence.

4. The data generation method as described in claim 1, characterized in that, The combined data includes image sample data and text sample data with consistent content; The process of training the encoder based on the joint pair data to obtain the initial encoder and the fused encoder includes: The image sample data is segmented into image blocks according to a preset pixel size to obtain multiple image feature sequences; The image feature sequence is input into the first encoder to obtain the first feature output by the first encoder; The text sample data is segmented to obtain multiple text feature sequences; The text feature sequence is input into the second encoder to obtain the second feature output by the second encoder; The first feature and the second feature are input into the image-text fusion encoder; The first encoder, the second encoder, and the image-text fusion encoder are jointly trained to obtain the initial encoder and the fusion encoder.

5. The data generation method according to any one of claims 1-3, characterized in that, The first dimension of the data to be identified includes image data, and the second dimension of the data includes text data.

6. The data generation method as described in claim 5, characterized in that, The process of segmenting the data to be identified in the first dimension yields multiple sub-data to be identified, including: The image data is segmented into image blocks according to a preset pixel size to obtain multiple sub-data to be identified.

7. The data generation method as described in claim 6, characterized in that, The step of generating second-dimensional data based on the target subsequence includes: Translate the target subsequence into sentence data; Text data is generated based on several of the aforementioned sentence data.

8. The data generation method according to any one of claims 1-3, characterized in that, The first dimension of the data to be identified includes text data, and the second dimension of the data includes image data.

9. The data generation method as described in claim 8, characterized in that, The process of segmenting the data to be identified in the first dimension yields multiple sub-data to be identified, including: The text data is segmented into word data to obtain multiple sub-data to be identified.

10. The data generation method as described in claim 8, characterized in that, The step of generating second-dimensional data based on the target subsequence includes: The target subsequence is input into the image decoding model to obtain image data.

11. A data generation apparatus, characterized in that, The data generation device includes: The acquisition module is used to acquire the data to be identified in the first dimension; The segmentation module is used to segment the data to be identified in the first dimension to obtain multiple sub-data to be identified; The extraction module is used to extract feature information from each of the sub-data to be identified to obtain a feature sequence; A generation module is used to sequentially input multiple feature sequences into a target data generation model to obtain a target subsequence output by the target data generation model. The generation module is further configured to generate second-dimensional data based on the target sub-sequence, wherein the first-dimensional data to be identified and the second-dimensional data include text data or image data, and when the first-dimensional data to be identified includes text data, the second-dimensional data includes image data. The device also includes a training module; The training module is used to acquire joint pair data, train an encoder based on the joint pair data to obtain an initial encoder and a fusion encoder, initialize the encoder of the data generation model using the initial encoder, and initialize the decoder of the data generation model using the fusion encoder to obtain an initial data generation model. The initial data generation model is then trained based on the first dimension feature information and the second dimension annotation information to obtain a trained target data generation model.

12. The data generation apparatus as claimed in claim 11, characterized in that, The target data generation model includes a decoder and an encoder. The generation module is further configured to input multiple feature sequences sequentially into the encoder to obtain several target feature sequences of fixed length, determine the first target feature sequence at the current position and the second target feature sequence at the previous position, and input the first target feature sequence and the second target feature sequence into the decoder to obtain the target subsequence output by the decoder.

13. The data generation apparatus as described in claim 12, characterized in that, The generation module is further configured to input the first target feature sequence and the second target feature sequence into the decoder, obtain multiple candidate feature sequences output by the decoder, concatenate each candidate feature sequence with the second target feature sequence to obtain multiple candidate sub-sequences, calculate the confidence level corresponding to each candidate sub-sequence, and select the candidate sub-sequence with the highest confidence level as the target sub-sequence.

14. The data generation apparatus as claimed in claim 11, characterized in that, The combined data includes image sample data and text sample data with consistent content; The training module is further configured to segment the image sample data into image blocks according to a preset pixel size to obtain multiple image feature sequences, input the image feature sequences into a first encoder to obtain a first feature output by the first encoder, perform word segmentation on the text sample data to obtain multiple text feature sequences, input the text feature sequences into a second encoder to obtain a second feature output by the second encoder, input the first feature and the second feature into an image-text fusion encoder, and perform joint training on the first encoder, the second encoder and the image-text fusion encoder to obtain an initial encoder and a fusion encoder.

15. The data generation apparatus according to any one of claims 11-13, characterized in that, The first dimension of the data to be identified includes image data, and the second dimension of the data includes text data.

16. The data generation apparatus as described in claim 15, characterized in that, The segmentation module is also used to segment the image data into image blocks according to a preset pixel size to obtain multiple sub-data to be identified.

17. A data generation device, characterized in that, The device includes: a memory, a processor, and a data generation program stored in the memory and executable on the processor, the data generation program being configured to implement the data generation method as described in any one of claims 1 to 10.

18. A storage medium, characterized in that, The storage medium stores a data generation program, which, when executed by a processor, implements the data generation method as described in any one of claims 1 to 10.

Images