A deep learning-based data set augmentation method and system

Abstract

The present application relates to the technical field of data set enhancement, and particularly relates to a data set enhancement method and system based on deep learning, which comprises the following steps: inputting an original text into a pre-trained cross-modal text encoder to obtain an original semantic vector; presetting multiple target modes, constructing projection functions of each mode, mapping the original semantic vector into style guide vectors of each target mode, and generating a comprehensive style vector through weighted fusion; performing syntax analysis on the original text, extracting a word-level mask, and differentiating and fusing context word vectors and the comprehensive style vector based on the mask to generate a global fusion semantic vector; mapping the global fusion semantic vector into an input space of a large language model through a trainable projection matrix to form a soft prompt vector and inject the soft prompt vector into a model input layer to guide the generation of multiple enhanced texts which are semantically faithful and stylistically diverse, and complete data set enhancement. The present application realizes the effect of enhancing the multi-stylization of generated texts while retaining original semantic information.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more particularly to a dataset augmentation method and system based on deep learning. Background Technology

[0002] With the widespread application of deep learning in natural language processing, high-quality, large-scale labeled data has become crucial for improving model performance. However, in many practical applications, such as healthcare, law, and finance, acquiring large amounts of labeled data is costly and time-consuming, leading to prominent issues of sparse training data or class imbalance. Traditional data augmentation methods, such as synonym replacement, back-translation, and random insertion / deletion, can increase the amount of data to some extent, but the generated text often lacks diversity, is prone to semantic drift, and fails to meet the dual requirements of semantic fidelity and expressive richness for complex tasks.

[0003] In recent years, text enhancement techniques based on generative models have gradually emerged, especially with the advent of Large Language Models (LLM), which have provided powerful tools for generating high-quality text. However, directly using LLM for enhancement often lacks control, resulting in highly randomized outputs that easily deviate from the original semantics, producing "illusions" or irrelevant content.

[0004] Furthermore, most existing methods are limited to internal textual transformations and fail to effectively introduce non-linguistic information (such as visual, emotional, and auditory information) to guide the stylistic expression of the text. This results in insufficient enhancement of the text's vividness, concreteness, and emotional appeal, making it difficult to meet the specific requirements of tasks such as creative writing, advertising copywriting, and sentiment analysis that demand a particular style of expression. Summary of the Invention

[0005] To address the technical problems of existing text data augmentation techniques, such as the difficulty in balancing semantic fidelity and expressive diversity, weak style guidance capabilities, and uncontrollable generation processes, this invention provides solutions in the following aspects.

[0006] In the first aspect, a deep learning-based dataset augmentation method includes: acquiring original text; inputting the original text into a pre-trained cross-modal text encoder to obtain original semantic vectors; pre-setting a set of multi-target modalities; constructing a projection function for each target modality; inputting the original semantic vector into each modality projection function to generate a style guidance vector for each target modality; combining all target modality style guidance vectors to obtain a comprehensive style vector; performing syntactic analysis on the original text to extract a mask; fusing the context word vectors of the original text with the comprehensive style vector based on the mask to generate a global fused semantic vector; mapping the global fused semantic vector to the input embedding space of a pre-trained large language model via a trainable linear projection matrix to form a soft cue vector; injecting the soft cue vector into the input layer of the large language model to generate multiple augmented texts, thus completing the dataset augmentation.

[0007] Preferably, the pre-trained cross-modal text encoder is a pre-trained CLIP model, which consists of an image encoder and a text encoder.

[0008] Preferably, the set of multi-target modalities includes the expressive features of non-verbal information such as image target modalities and emotional target modalities.

[0009] Preferably, the comprehensive style vector includes: a preset multi-target modality set, obtaining the modality projection function corresponding to each target modality; a function that maps the original semantic vector to the semantic subspace of the target modality according to the modality projection function corresponding to each target modality, and outputting it as a style guidance vector; and weighted fusion of the style guidance vectors of different modalities to generate a comprehensive style vector.

[0010] Preferably, the modal projection function includes: taking image modality as an example, the projection function can be modeled based on the directional characteristics of a cross-modal image encoder; according to the semantics of the original text, image-text data pairs that are similar to the semantics of the original text are obtained from the image-text annotation data through manual screening; from the screened image-text data pairs, the cross-modal text encoder obtains the corresponding semantic vector and the cross-modal image encoder obtains the corresponding image vector; the image-text offset of each image-text data is obtained; the average of the image-text offsets of all image-text data is obtained as the image style guidance direction; the image style guidance direction is weighted and added to the original semantic vector of the original text to obtain the modal projection function.

[0011] Preferably, the globally fused semantic vector includes: performing syntactic analysis on the original text to extract the semantic core words and non-semantic core words of the original text; generating word-level masks based on the semantic core words and non-semantic core words of the original text; using a pre-trained language model to encode and obtain the context-aware word vector of each word in the original text; combining the context-aware word vector of each word with the comprehensive style vector of the original text, and fusing them according to the corresponding mask information to obtain the fused word-level fused vector of each word; and averaging and aggregating the word-level fused vectors of each word to obtain the globally fused semantic vector.

[0012] Preferably, the generation of multiple enhanced texts includes the following steps: setting a projection matrix, multiplying the global fusion semantic vector with the projection matrix to obtain a soft cue vector with the required dimension of the input space of the large language model, wherein the size of the projection matrix is ​​consistent with the size of the dimension of the global fusion semantic vector and the hidden layer dimension of the large language model; segmenting the original text using the word segmenter of the large language model to obtain a word sequence and obtaining the corresponding word embedding vector; concatenating the soft cue vector and the word embedding vector in the sequence dimension to form an extended input embedding sequence, wherein the soft cue vector is at the beginning of the extended input embedding sequence; the extended input embedding sequence is used as the input of the large language model and fed into the Transformer structure, and the large language model generates multiple enhanced texts in an autoregressive manner.

[0013] Secondly, a deep learning-based dataset augmentation system includes a processor and a memory, wherein the memory stores computer program instructions that, when executed by the processor, implement the aforementioned deep learning-based dataset augmentation method.

[0014] The present invention has the following effects:

[0015] 1. By introducing a pre-trained cross-modal text encoder to generate original semantic vectors and constructing a learnable projection function for multi-objective modalities, the original semantic vectors are mapped to style guidance vectors for each modality. This achieves effective transfer of cross-modal knowledge to text enhancement tasks. It can utilize the rich non-linguistic information contained in the cross-modal model to provide clear and quantifiable style guidance for text generation, thereby enhancing the expressive effect of the generated text in different modal spaces.

[0016] 2. By performing dependency parsing on the original text, a mask vector is generated that distinguishes between semantic core words and non-core modifiers. Based on this mask, the context word vectors and the comprehensive style vector are differentially fused, achieving fine-grained semantic control of "core freezing and edge perturbation". This ensures the stability of the core semantics of the sentence and avoids semantic drift caused by global style transfer. It also enhances the expressive diversity and vividness of the generated text without destroying the original meaning. Attached Figure Description

[0017] Figure 1 This is a flowchart of steps S1-S4 in a deep learning-based dataset augmentation method according to an embodiment of the present invention.

[0018] Figure 2 This is a structural block diagram of a dataset augmentation system based on deep learning according to an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0020] Reference Figure 1 A deep learning-based dataset augmentation method includes steps S1-S4, as detailed below:

[0021] S1: Obtain the original text and input it into the pre-trained cross-modal text encoder to obtain the original semantic vector.

[0022] The input text samples to be augmented are read in batches from the original training set as the original text. The original text is preprocessed, including removing extra spaces and standardizing punctuation, to obtain the standardized text.

[0023] The preprocessed original text is input into the pre-trained cross-modal text encoder to generate the corresponding original semantic vector. The original semantic vector represents the vector of the original text in the semantic space. The cross-modal text encoder adopts the text encoder in the CLIP (Contrastive Language-Image Pre-Training) model. The CLIP model is a well-known technique and will not be described in detail here.

[0024] S2: Preset a set of multi-target modalities, construct a projection function for each target modality, input the original semantic vector into each modality projection function, generate a style guidance vector for each target modality, and combine all target modality style guidance vectors to obtain a comprehensive style vector.

[0025] Original text Inputting the data into a pre-trained cross-modal text encoder yields the corresponding original semantic vector. .

[0026] After obtaining the original semantic vector corresponding to the original text, the original semantic vector represents the vector of the original text in the semantic space. Since different texts have different expression styles in multimodal scenarios, a preset target modality set is introduced to guide the original semantic vector to migrate towards a specific expression style, thereby achieving the generalization of the content represented by the original text.

[0027] The target modality set includes, but is not limited to, image modality, emotion modality, etc., and each modality corresponds to a non-verbal information expression feature.

[0028] For example, the image modality is used to enhance the text's ability to describe spatial scenes and its visual representation; the emotion modality is used to strengthen the text's subjective emotional tone and intensity.

[0029] By constructing modal projection functions that correspond one-to-one with each target modality, a controllable stylistic shift of the original semantic vector is achieved, thereby providing guiding signals for the subsequent generation of diverse and stylistically consistent enhanced text.

[0030] Let the modal projection function be... Defined as the original semantic vector The function maps to the target modality semantic subspace, and its output is a style guidance vector. ,Right now .

[0031] The purpose is not to completely change the original semantics, but to moderately perturb along the semantic direction defined by the target modality while maintaining the core semantic structure, so that the generated text presents the typical characteristics of that modality in terms of expression.

[0032] For example, when the target modality is "emotion", The output should be closer to semantic regions with high emotional intensity, such as "excitement," "sadness," or "anger," so as to guide the subsequent generation of more emotionally impactful text.

[0033] For different target modes, the construction method of the modal projection function can be designed differently according to the modal characteristics.

[0034] For example, taking image modality as an example, the projection function Modeling can be performed based on the directional characteristics of a cross-modal image encoder. Based on the semantics of the original text, image-text data pairs that are similar in semantics to the original text are obtained from the image-text annotation data through manual selection. From the selected image-text data pairs, the corresponding semantic vectors are obtained using the cross-modal text encoder in step S1. And using a cross-modal image encoder to obtain the corresponding image vectors. Get the image and text offset of each image and text data. The average text and image offset of all text and image data is obtained. As a guide for image style and define The cross-modal image encoder uses the image encoder from the CLIP model. This CLIP model is the same as the CLIP model in step S1. Therefore, the cross-modal image encoder and the cross-modal text encoder in step S1 are trained using the same image-text dataset. The adjustable gain coefficient is taken as an empirical value. To control the intensity of style transfer, the number of image and text data points that are semantically similar to the original text is set to [number]. It can be adjusted by the implementer according to the specific implementation scenario.

[0035] For example, taking the emotion modality as an example, the projection function A trainable multilayer perceptron (MLP) architecture is used. The MLP consists of two fully connected layers, with the activation function being... The overall structure is as follows:

[0036] ;

[0037] During training, the original text was obtained through manual screening. Related sentiment-annotated text pairs ,in, yes An emotionally enhanced version, It can be generated using artificial standards, with parameters optimized by minimizing the loss function of the mean square error. , , , These are learnable optimization parameters in an MLP, which enable the input raw semantic text vector to be optimized. The corresponding output vector tends to the semantic region with high emotional intensity.

[0038] When multimodal collaborative enhancement is enabled, style guidance vectors corresponding to the original text in different modalities are generated through multiple modal projection functions, and a comprehensive style vector is generated through a weighted fusion strategy:

[0039]

[0040] in, For the first The weight values ​​corresponding to each preset target mode. The preset total number of target modes, For the original text in the first A style guidance vector for a predefined target modality. The weights between different target modalities are summed to 1. The weight values ​​can be adjusted by the implementer according to the implementation scenario. In this invention, the weight values ​​between different target modalities are taken to be equal. By adjusting the weight values ​​between different target modalities, the weights of different target modalities in the integrated style vector can be adjusted.

[0041] The comprehensive style vector serves as a key input for subsequent semantic fusion, maintaining the same dimensions as the original semantic vector to ensure compatibility of vector operations. The comprehensive style vector does not directly correspond to a specific text but rather represents a "semantic trend" or "expressive tendency," used to introduce style perturbations at the non-core vocabulary level.

[0042] S3: Perform syntactic analysis on the original text to extract the mask, and fuse the context word vectors of the original text with the comprehensive style vector based on the mask to generate a fused semantic vector.

[0043] Step S2 can obtain the comprehensive style vector corresponding to the original text to achieve modal fusion. However, if comprehensive style transfer is performed on the entire original text, it will cause semantic deviation in the original text. Therefore, word-level semantic analysis is performed on the original text to retain the main semantic information of the original text.

[0044] To achieve stable preservation of semantic core and stylistic enhancement of non-core components, dependency parsing is performed on the original text to identify its core grammatical structure and generate word-level mask vectors accordingly. The mask values ​​of semantic core words in the word-level mask vectors are set to 1, and the mask values ​​of non-semantic core words are set to 0, which is used to distinguish between semantic core words and non-core words. Dependency parsing is a well-known technique and will not be elaborated further.

[0045] Based on this, combined with comprehensive style vectors The context-aware word vectors of each word are differentially fused to ultimately generate a word-level fused semantic vector that balances semantic fidelity and expressive diversity. This provides high-quality semantic input for subsequent controllable text generation.

[0046] After obtaining the word-level mask vector, the context-aware word vector of each word in the original text is obtained. This vector is encoded by a pre-trained language model, which can reflect the semantic information of the word in a specific context. The pre-trained language model encoder is a pre-trained language model such as BERT, RoBERTa, or sense2vec.

[0047] The first part of the original text Each word is input into a pre-trained language model encoding to obtain the corresponding context-aware word vector. ;

[0048] Will Compared with the style guidance vectors generated above During the fusion process, a mask-based differential weighting strategy is employed to achieve the fusion principle of "main backbone freezing and edge perturbation." The specific fusion formula is as follows:

[0049] For the Each word, and its fused word vector Defined as:

[0050]

[0051] in, The fusion coefficient is the backbone fusion coefficient. To ensure that the backbone word vectors mainly retain the original semantics and only introduce very slight style perturbations, it needs to be small. An empirical value is taken. It can be adjusted by the implementer according to the specific implementation scenario;

[0052] This is the non-core fusion coefficient, which allows non-core words to fully absorb information from style-guided vectors, enabling diversified expansion of expression methods. A relatively large value is required; an empirical value is chosen. It can be adjusted by the implementer according to the specific implementation scenario.

[0053] For the first The mask value corresponding to each word in the word-level mask vector. Represented as the core semantic words in the original text. This refers to non-core semantic words in the original text.

[0054] For the original text The corresponding integrated style vector.

[0055] For the first The context-aware word vectors corresponding to each word.

[0056] This design ensures that the core semantic structure of the sentence is not destroyed, while the modifiers that are not part of the core semantic information can be enhanced to make the expression more vivid or emotional.

[0057] All word-level fusion vectors are aggregated into a global fusion semantic vector through an averaging operation:

[0058]

[0059] The vector It preserves the original text The vector, which incorporates the core semantic intent and style guidance information of the target modality, constitutes a semantically rich and style-controllable enhanced semantic representation. This vector will serve as a soft cue input for the subsequent large language model generation process, guiding the model to generate enhanced text that is both faithful to the original meaning and possesses diverse expressive styles.

[0060] S4: Map the global fusion semantic vector to the input embedding space of the pre-trained large language model via a trainable linear projection matrix to form soft cue vectors. Inject the soft cue vectors into the input layer of the large language model to generate multiple enhanced texts and complete the dataset augmentation.

[0061] After obtaining the global fusion semantic vector generated in step S3 Subsequently, since the semantic space of the globally fused semantic vector and the semantic space of the large language model input embedding space are not consistent, a learnable soft cue vector is formed for mapping alignment, and then enhanced text generation is performed.

[0062] because The semantic space of the cross-modal encoder derived from the CLIP model, while the input embedding space of the large language model has a different dimensional structure, and there is a mismatch between the spatial dimension and semantic distribution between the two.

[0063] To address this, a trainable linear projection matrix is ​​introduced. ,in, To globally fuse semantic vector dimensions, For the hidden layer dimension of a large language model, for The matrix. Through matrix multiplication, the globally fused semantic vector (vf) is mapped to the input space of the large language model:

[0064]

[0065] The resulting vector This refers to soft cue vectors, which do not correspond to any actual words, but carry fused semantic and stylistic information, serving as initial guiding signals for the generation process.

[0066] Use a large language model's word segmenter to process the original text. Perform word segmentation to obtain a word sequence. , This represents the number of words segmented by the word segmenter.

[0067] Obtain the corresponding word embedding vector .

[0068] The soft cue vector h0 is concatenated with the word embedding vector along the sequence dimension to form an extended input embedding sequence:

[0069]

[0070] This extended input embedding sequence is used as input to the large language model and fed into the Transformer structure. Because... Located at the very beginning of the sequence, the semantic information it carries is used as a context bias in the attention mechanism to participate in the computation, thereby globally guiding the generation direction.

[0071] The large language model generates multiple enhanced texts in an autoregressive manner. ,in The preset number of generators is 1000, which can be adjusted by the implementer according to the specific implementation scenario.

[0072] The generation process can employ various decoding strategies, including: Top-k sampling: randomly sampling from the k words with the highest probabilities to increase diversity; Nucleus sampling (Top-p): sampling from the smallest set of words with a cumulative probability of p to balance fluency and creativity; BeamSearch: retaining multiple candidate paths and selecting the output with the highest overall probability, suitable for scenarios that emphasize semantic coherence.

[0073] Preferably, the system is set with a temperature coefficient (temperature ≈ 0.7–0.9) to encourage diversity while maintaining fluency. The generated text semantics are consistent with the original text, but reflect the style of the target modality.

[0074] For example, when the globally fused semantic vector is guided by the sentiment modality, the generated text may emphasize subjective emotions more; when guided by the image modality, the text may contain more spatial descriptions and visual details.

[0075] Then As the input layer of the large language model, it injects contextual guidance information to drive the model to generate multiple semantically stable but stylistically diverse augmented texts, thus completing the dataset augmentation.

[0076] This invention also provides a dataset augmentation system based on deep learning. For example... Figure 2 As shown, the system includes a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement a deep learning-based dataset augmentation method according to the first aspect of the present invention. The system also includes other components well-known to those skilled in the art, such as a communication bus and a communication interface, the setup and functions of which are known in the art and will not be described further here.

[0077] It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of this invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A deep learning-based data set augmentation method, characterized in that, The method comprises the following steps: obtaining an original text, inputting the original text into a pre-trained cross-modal text encoder to obtain an original semantic vector; a set of pre-set multi-target modalities is constructed, and a projection function of each target modality is constructed, the original semantic vector is input into each modality projection function, and each target modality style guide vector is generated, and the comprehensive style vector is obtained by synthesizing all target modality style guide vectors; performing syntax analysis on the original text to extract a mask, and fusing the context word vector of the original text and the comprehensive style vector based on the mask to generate a global fusion semantic vector; the global fusion semantic vector comprises: performing syntax analysis on the original text to extract the semantic main vocabulary and non-semantic main vocabulary of the original text, generating a word-level mask according to the semantic main vocabulary and non-semantic main vocabulary of the original text; using a pre-trained language model to encode to obtain a context-aware word vector of each word in the original text, combining the context-aware word vector of each word and the comprehensive style vector of the original text, and fusing according to the corresponding mask information to obtain a word-level fusion vector of each word after fusion; the word-level fusion vector of each word is averaged and aggregated to obtain a global fusion semantic vector; mapping the global fusion semantic vector to the input embedding space of the pre-trained large language model through a trainable linear projection matrix to form a soft prompt vector, and injecting the soft prompt vector into the input layer of the large language model to generate a plurality of enhanced texts, and completing the data set enhancement. 2.The deep learning based data set augmentation method of claim 1, wherein, The pre-trained cross-modal text encoder is a pre-trained CLIP model, and the pre-trained CLIP model comprises an image encoder and a text encoder. 3.The deep learning based data set augmentation method of claim 1, wherein, The set of multi-target modalities comprises image target modalities, emotional target modalities and the like non-language information expression features.

4. The data set augmentation method based on deep learning according to claim 1, characterized in that, The comprehensive style vector comprises: a set of pre-set multi-target modalities is constructed, and a projection function of each target modality is constructed, the original semantic vector is input into each modality projection function, and each target modality style guide vector is generated, and the comprehensive style vector is obtained by synthesizing all target modality style guide vectors. The modality projection function comprises: Taking the image modality as an example, the projection function can be modeled based on the direction characteristics of the cross-modal image encoder, the image-text data pairs similar to the original text semantics are obtained by manually screening the image-text data in the image-text annotation data according to the original text semantics, the cross-modal text encoder obtains the corresponding semantic vector and the cross-modal image encoder obtains the corresponding image vector, and the image-text offset of each image-text data is obtained; the average of the image-text offsets of all image-text data is taken as the image style guide direction, the image style guide direction and the original semantic vector of the original text are weighted and added to obtain the modality projection function.

5. The data set augmentation method based on deep learning according to claim 4, characterized in that, The method for generating a plurality of enhanced texts comprises the following steps: setting a projection matrix, multiplying the global fusion semantic vector by the projection matrix to obtain a soft prompt vector with the required dimension of the input space of the large language model, wherein the size of the projection matrix is consistent with the size of the global fusion semantic vector dimension and the hidden layer dimension of the large language model; 6. The data set augmentation method based on deep learning according to claim 1, characterized in that, ​ ​ The original text is segmented by using a tokenizer of a large language model to obtain a token sequence and obtain a corresponding word embedding vector; The soft prompt vector and the word embedding vector are spliced in the sequence dimension to form an extended input embedding sequence, wherein the soft prompt vector is at the forefront of the extended input embedding sequence; The extended input embedding sequence is input into a Transformer structure as an input of the large language model, and the large language model generates a plurality of enhanced texts in a self-recurrent manner. 7.A deep learning based data set augmentation system, characterized in that, The application further provides a computer readable storage medium storing the computer program instructions. The application further provides a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, a deep learning-based data set enhancement method according to any one of claims 1-6 is realized.

Images