Data processing method and device, computer equipment and storage medium

Abstract

The embodiment of the invention discloses a data processing method and device, computer equipment and a storage medium. The method comprises the steps of obtaining original image-text data, and performing text expansion on an original text extracted from the original image-text data to obtain an expanded text; determining the original text and the extended text as target texts; obtaining an expanded image set, and matching a target expanded image matched with the original image extracted from the original image-text data in the expanded image set; and determining the original image and the target extended image as a target image, and combining the target image and the target text to obtain extended image-text data which is used for image-text recognition model training. According to the technical scheme, multi-modal expansion can be performed on the text and the image of the original image-text data, and the expanded image-text data is used as the new image-text data, so that the data volume and the richness of the image-text data are increased, and the robustness of a downstream training task is improved.

Description

Technical Field

[0001] This application relates to the field of text processing technology, specifically to a data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the development of artificial intelligence technology, some large models now require a large amount of data for training. When the amount of data is limited, it is necessary to expand the original data to obtain new data with a larger amount of data.

[0003] In related technologies, the expansion of the original data often involves expanding data of a single modality, such as expanding text data or image data separately. The expansion method only involves making minor modifications to the original data and does not take into account the interaction between data of different modalities.

[0004] Therefore, the data augmentation methods in related technologies can lead to technical problems such as low richness of the augmented data. Summary of the Invention

[0005] This application provides a data processing method, apparatus, computer device, and storage medium. These can improve the richness of the expanded data, thereby enhancing the robustness of downstream training tasks.

[0006] According to one aspect of this application, a data processing method is provided, comprising:

[0007] Obtain the original image and text data, and expand the original text extracted from the original image and text data to obtain the expanded text;

[0008] The original text and the expanded text are identified as the target text;

[0009] Obtain the expanded image set, and match the target expanded image that matches the original image extracted from the original image and text data in the expanded image set;

[0010] The original image and the target augmented image are identified as the target image. The target image and the target text are then combined to obtain augmented image-text data, which is used to train the image-text recognition model.

[0011] According to one aspect of this application, a data processing apparatus is provided, comprising:

[0012] The text augmentation module is used to acquire the original image and text data and augment the original text extracted from the original image and text data to obtain augmented text.

[0013] The determination module is used to identify the original text and the expanded text as the target text.

[0014] The image augmentation module is used to acquire an augmented image set and match the target augmented image that matches the original image extracted from the original image and text data in the augmented image set.

[0015] The combination module is used to identify the target image from the original image and the target augmented image, and to combine the target image and the target text to obtain augmented image-text data, which is used for training the image-text recognition model.

[0016] In some implementations, the determining module includes a splitting submodule, a combining submodule, and a determining submodule;

[0017] The splitting submodule is used to split the original text to obtain the split text.

[0018] The combination submodule is used to combine split text to obtain combined text;

[0019] The Determine submodule is used to determine the target text from the combined text, the original text, and the expanded text.

[0020] In some implementations, the combined submodule is used for:

[0021] Determine the limit on the number of split texts corresponding to the combined text;

[0022] Within the limit on the number of split texts, select the corresponding number of target split texts from the split texts and combine them to obtain the combined text.

[0023] In some implementations, the text expansion module is used for:

[0024] Extract the identity identifier text and the corresponding signature text from the original text extracted from the original image and text data;

[0025] Generate corresponding text expansion instructions based on the identity identification text and signature text;

[0026] The text augmentation instructions are input into the trained text augmentation model, which outputs augmented text, including augmented identity text and augmented signature text.

[0027] In some implementations, the image augmentation module is used for

[0028] Obtain the first image features of the expanded image from the expanded image set;

[0029] Obtain the second image features of the original image extracted from the original image and text data;

[0030] The first image features and the second image features are matched to obtain a similarity score between the augmented image and the original image.

[0031] Expanded images with similarity scores higher than a preset score threshold are identified as target expanded images that match the original image.

[0032] In some implementations, the combining module includes an image generation submodule and a combining submodule;

[0033] The image generation submodule is used to determine the descriptive text corresponding to each image in the original image and the target augmented image, and generate the corresponding output image based on the descriptive text;

[0034] The combination submodule is used to determine the target image from the output image, the original image, and the target augmented image.

[0035] In some implementations, the image generation submodule is used for:

[0036] Each image from the original image and the target augmented image is input into the trained text generation model, which outputs descriptive text corresponding to each image.

[0037] The descriptive text is input into the trained image generation model, which outputs the corresponding image for each descriptive text.

[0038] In some implementations, the combination module is used for:

[0039] Identify at least one data type corresponding to the original image and text data;

[0040] The target image and target text are filtered according to their data types to obtain the target image and target text corresponding to each data type;

[0041] The target image and target text corresponding to each data type are combined to obtain the augmented image and text data corresponding to each data type.

[0042] In some implementations, the combination module includes a category determination submodule, a selection submodule, and a combination submodule;

[0043] The determination submodule is used to determine the image category of the target image and the text category of the target text;

[0044] The selection submodule is used to select any one target image from the target images of each image category as the selected target image, and to select any one target text from the target text of each text category as the selected target text;

[0045] The combination submodule is used to generate expanded graphic data by combining selected target text and selected target image.

[0046] In some implementations, the combined submodule is used for:

[0047] Perform at least one of the following operations on the selected target text: text addition, text deletion, text replacement, and text back translation, to obtain enhanced target text;

[0048] The selected target image is subjected to at least one of the following operations: image size adjustment, image color parameter adjustment, and image brightness parameter adjustment, to obtain an enhanced target image;

[0049] By combining the enhanced target image and the enhanced target text, augmented image and text data is obtained.

[0050] According to one aspect of this application, a computer-readable storage medium is provided, which stores a plurality of instructions adapted for loading by a processor to execute the data processing method provided in the embodiments of this application.

[0051] According to one aspect of this application, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the data processing method provided in the embodiments of this application.

[0052] According to one aspect of this application, a computer program product includes a computer program or instructions that, when executed by a processor, implement the data processing method provided in the embodiments of this application.

[0053] In this embodiment, the process involves acquiring original image and text data, augmenting the original text extracted from the original image and text data to obtain augmented text, determining the original text and augmented text as target text, acquiring an augmented image set, matching the target augmented image in the augmented image set with the original image extracted from the original image and text data, determining the original image and target augmented image as target image, and combining the target image and target text to obtain augmented image and text data, which is used for training the image and text recognition model. In this embodiment, the original text in the original image and text data is augmented to obtain augmented text. The augmented text and the original text are determined as the target text. The original image in the original image and text data is augmented to obtain the target augmented image. The target augmented image and the original image are determined as the target image. Finally, the target text and the target image are randomly combined to obtain augmented image and text data. The augmented image and text data is used as new image and text data. Since the augmented image and text data is augmented in a multimodal manner through text and images, it has a large amount of data and more types of image and text data, thereby increasing the amount and richness of image and text data, and thus improving the robustness of downstream training tasks.

[0054] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a system architecture diagram of the data processing method provided in the embodiments of this application;

[0057] Figure 2 This is a schematic diagram of a data processing scenario provided in an embodiment of this application;

[0058] Figure 3 This is a flowchart illustrating the data processing method provided in an embodiment of this application;

[0059] Figure 4 This is a flowchart illustrating the steps included in step 220 of the embodiments of this application;

[0060] Figure 5 This is a flowchart illustrating the steps included in step 240 provided in an embodiment of this application;

[0061] Figure 6 This is another schematic flowchart of the data processing method provided in the embodiments of this application;

[0062] Figure 7 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;

[0063] Figure 8 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application;

[0064] Figure 9 This is another structural schematic diagram of the computer device provided in the embodiments of this application. Detailed Implementation

[0065] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0066] It is understood that in the specific embodiments of this application, data such as text and images are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0067] It should be noted that while some processes described in the specification, claims, and accompanying drawings contain multiple steps that appear in a specific order, it should be clearly understood that these steps may not be performed in the order they appear herein, or may be performed in parallel. The step numbers are merely used to distinguish different steps and do not represent any particular order of execution. Furthermore, descriptions such as "first," "second," or "objective" in this document are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0068] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:

[0069] Text generation models are a type of model that uses deep learning techniques to transform image content into natural language descriptions. Image-to-text models first need to understand the content of the input image, which involves computer vision techniques. The model learns through training to convert image features into internal semantic representations. After understanding the image, the model needs to map these visual features to a natural language space. After obtaining the text features through mapping, the model enters the text generation stage, thereby generating the text corresponding to the image.

[0070] Image generation models are deep learning techniques used to generate corresponding images based on input natural language descriptions. First, an image generation model needs to understand the input text content, which involves natural language processing techniques. The model learns through training to convert text features into internal semantic representations. After understanding the text, the model needs to map these semantic representations to image space. After obtaining the image features through mapping, the model enters the image generation stage, thereby generating the image corresponding to the text.

[0071] Text augmentation models are a type of Natural Language Processing (NLP) technique used to increase the size and diversity of text datasets. The main purpose of these models is to transform and augment the original text to generate new text samples that are associated with it. Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies various theories and methods that enable effective communication between humans and computers using natural language. NLP deals with natural language, the language people use in daily life, and is closely related to linguistics; it also involves computer science and mathematics. Pre-trained models, an important technique for model training in artificial intelligence, evolved from large language models in NLP. After fine-tuning, large language models can be widely applied to downstream tasks. Natural Language Processing techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0072] The text generation, image generation, and text augmentation models mentioned above are all pre-trained models. These can be large models, referring to deep neural networks (DNNs) with a large number of parameters. They are trained on massive amounts of unlabeled data, leveraging the function approximation capabilities of large-parameter DNNs to enable PTMs to extract common features from the data. Through fine-tuning, parameter efficient fine-tuning (PEFT), and prompt-tuning techniques, they are suitable for downstream tasks. Therefore, pre-trained models can achieve ideal results in small-shot or zero-shot scenarios. PTMs can be categorized according to the data modality they process: language models (ELMO, BERT, GPT), visual models (swin-transformer, ViT, V-MOE), speech models (VALL-E), and multimodal models (ViBERT, CLIP, Flamingo, Gato), among others. Multimodal models refer to models that establish feature representations for two or more data modalities. Pre-trained models are important tools for outputting AI-generated content (AIGC) and can also serve as a general interface connecting multiple specific task models.

[0073] First, we will describe some of the technical problems that exist in the relevant technologies.

[0074] With the development of artificial intelligence technology, some large models now require a large amount of data for training. When the amount of data is limited, it is necessary to expand the original data to obtain new data with a larger amount of data.

[0075] In related technologies, the expansion of the original data often involves expanding data of a single modality, such as expanding text data or image data separately. The expansion method only involves making minor modifications to the original data and does not take into account the interaction between data of different modalities.

[0076] Therefore, the data augmentation methods in related technologies can lead to technical problems such as low richness of the augmented data, which in turn affects the effectiveness of downstream training tasks.

[0077] To address the aforementioned technical problems, embodiments of this application provide a data processing method, apparatus, computer device, and storage medium. Specifically, in these embodiments, the original text in the original image-text data is augmented to obtain augmented text. The augmented text and the original text are then used as target text. Similarly, the original image in the original image-text data is augmented to obtain a target augmented image. The target augmented image and the original image are then used as target images. Finally, the target text and target images are randomly combined to obtain augmented image-text data. This augmented image-text data is used as new image-text data. Because the augmented image-text data is augmented using multimodal methods involving text and images, it possesses a larger data volume and more diverse types of image-text data, thereby increasing the data volume and richness of the image-text data and improving the robustness of downstream training tasks.

[0078] The data processing methods, apparatus, computer equipment, and storage media provided in the embodiments of this application will be described in detail below.

[0079] This application provides a data processing method, apparatus, computer equipment, and storage medium. Specifically, this application will describe the data processing apparatus from the perspective of the data processing apparatus, which can be integrated into a computer equipment, such as a server or a terminal. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart home appliance, vehicle terminal, smart voice interaction device, aircraft, etc., but is not limited to these. This application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0080] Please see Figure 1 , Figure 1This is a system architecture diagram of the data processing method provided in the embodiments of this application. It includes a terminal 140, an Internet 130, a gateway 120, a server 110, etc.

[0081] Terminal 140 includes, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, and aircraft. This application embodiment can be applied to various scenarios, including but not limited to cloud technology and artificial intelligence. Furthermore, it can be a single device or a collection of multiple devices. For example, multiple desktop computers connected to each other via a local area network, sharing a single monitor, etc., can work collaboratively to form a terminal 140. Terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data.

[0082] Server 110 refers to a computer system that can provide certain services to terminal 140. Compared to ordinary terminal 140, server 110 has higher requirements in terms of stability, security, and performance. Server 110 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0083] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal 140 to server 110 are forwarded to the corresponding server 110 via gateway 120. Messages sent from server 110 to terminal 140 are also forwarded to the corresponding terminal 140 via gateway 120.

[0084] The data processing method in this application embodiment can be applied to various scenarios, such as model training scenarios, to increase the richness of training samples during model training. It can also be used to increase the richness of data types in a database expansion scenario. No limitations are imposed here.

[0085] Please refer to the following: Figure 2 , Figure 2 This is a schematic diagram of a data processing scenario provided in an embodiment of this application.

[0086] In the embodiments of this application, the original graphic data includes original text and original images. For example, a page contains corresponding original text and original images, and the original text and original images of different pages can be used as original graphic data.

[0087] For example in Figure 2 In this context, a single page contains an avatar, background image, nickname, and signature. The avatar and background image are identified as the original images, while the nickname and signature are identified as the original text. The data corresponding to the original text and original images across multiple pages constitutes the original text-image data.

[0088] In some scenarios, since the amount of original image and text data is limited, if we want to increase the amount and categories of image and text data, one approach is to legally obtain image and text data from the real Internet. However, this method of obtaining image and text data is time-consuming and may result in obtaining unwanted types of image and text data, leading to low efficiency in adding image and text data and the addition of unnecessary and redundant data.

[0089] In this embodiment of the application, the original image and original text in the original image are obtained, and then the original image and original text are expanded respectively to obtain the expanded text and the target expanded image. Finally, the original text and expanded text are determined as the target text, the original image and the target expanded image are determined as the target image, and the target text and target image are combined to obtain the expanded graphic data.

[0090] Specifically, the process involves acquiring original image and text data, expanding the original text extracted from the original image and text data to obtain expanded text, determining the original text and expanded text as target text, acquiring an expanded image set, matching the target expanded image in the expanded image set with the original image extracted from the original image and text data, determining the original image and target expanded image as target image, and combining the target image and target text to obtain expanded image and text data.

[0091] like Figure 2 As shown, the expanded image and text data, taking a specific page as an example, includes an avatar, nickname, background image, and signature. The avatar is the same as the one in the original image and text data, the nickname is the same as the one in the original data, the background image is a newly combined background image, and the signature is a newly combined signature. In other words, the image and text data for this page has changed compared to the original data, becoming the newly obtained expanded image and text data. This expands the original image and text data, thereby increasing its richness.

[0092] For a more detailed understanding of the data processing method provided in the embodiments of this application, please refer to [link / reference]. Figure 3 , Figure 3This is a flowchart illustrating the data processing method provided in this application embodiment. The data processing method can be applied to terminal devices such as mobile phones, computers, and tablets, or computer equipment such as servers. This data processing method can enhance the richness of text and image data. The data processing method may include the following steps:

[0093] Step 210: Obtain the original image and text data, and expand the original text extracted from the original image and text data to obtain the expanded text.

[0094] Step 220: Determine the original text and the expanded text as the target text;

[0095] Step 230: Obtain the expanded image set, and match the target expanded image that matches the original image extracted from the original image and text data in the expanded image set;

[0096] Step 240: Determine the target image from the original image and the target augmented image, and combine the target image and the target text to obtain augmented image-text data. The augmented image-text data is used for training the image-text recognition model.

[0097] Steps 210 to 240 will be described in detail below.

[0098] In step 210, the original image and text data is obtained, and the original text extracted from the original image and text data is expanded to obtain expanded text.

[0099] The original image and text data includes original text and original images. For example, a page contains corresponding original text and original images, and the original text and original images from different pages can be used as original image and text data. After obtaining the original image and text data, the original text can be extracted. Then, the original text is expanded to obtain the expanded text.

[0100] In some implementations, the original text extracted from the original image and text data is augmented to obtain augmented text, including:

[0101] (1.1) Extract the identity identification text and the corresponding signature text from the original text extracted from the original image and text data;

[0102] (1.2) Generate corresponding text expansion instructions based on the identity identification text and signature text;

[0103] (1.3) Input the text augmentation instructions into the trained text augmentation model and output the augmented text, which includes the augmented identity text and the augmented signature text.

[0104] The original text contains identity identification text and corresponding signature text. For example, in some social applications, an object can set its own nickname and signature. The nickname is the identity identification text, and the signature is the corresponding signature text.

[0105] After obtaining the identity identifier text and its corresponding signature text, corresponding text augmentation instructions can be generated based on these texts. These text augmentation instructions can be structured text written through prompt engineering. Prompt engineering, also known as in-context prompting, refers to refining prompt words using structured text and other methods to generate prompt text input into a large language model, thereby guiding the model to generate corresponding prediction results.

[0106] For example, the prompt text corresponding to the text expansion instruction is:

[0107] Some social media accounts have the following nicknames and signatures:

[0108] Nickname 1 + Signature 1

[0109] Nickname 2 + Signature 2

[0110] Nickname 3 + Signature 3

[0111] Nickname 4 + Signature 4

[0112] ...

[0113] Nickname N + Signature N,

[0114] Please write 500 similar nicknames and signatures in this style.

[0115] It should be noted that this prompt text can be considered a text expansion instruction. This instruction is then input into a trained text expansion model, which can be a trained large language model. This model can expand the text according to the instruction, generating new expanded text. The expanded text includes expanded identity text and expanded signature text. This achieves the expansion of the original text, increasing its richness.

[0116] Text augmentation models with different architectures or training methods can be used to augment text according to augmentation instructions, resulting in augmented text. After augmenting the original text, deduplication can be performed. For example, the augmented text can be input into a trained semantic recognition model to output text features corresponding to each augmented text. Then, the text features of different augmented texts are matched. Augmented texts with near-100% text feature similarity can be deduplicated to obtain the final augmented text.

[0117] In step 220, the original text and the expanded text are identified as the target text.

[0118] By combining the original text and the expanded text as the target text, the amount of text data can be increased, and the richness of the text data can be improved. Subsequently, the target text can be used to generate expanded image and text data to further enhance the richness of the image and text data.

[0119] For better text expansion results, please refer to [link / reference]. Figure 4 , Figure 4 This is a flowchart illustrating step 220 provided in an embodiment of this application. In some implementations, determining the original text and expanded text as the target text includes the following steps:

[0120] Step 301: Perform text splitting on the original text to obtain the split text;

[0121] Step 302: Combine the split text to obtain the combined text;

[0122] Step 303: Determine the combined text, original text, and expanded text as the target text.

[0123] Steps 301 to 303 will be described in detail below.

[0124] In step 301, the original text is split to obtain split text.

[0125] In addition to expanding the original text, the original text can also be split into split texts, and then the split texts can be combined to obtain new combined texts.

[0126] For example, for the signature text in the original text, it can be segmented according to punctuation marks such as periods, semicolons, and exclamation marks to obtain multiple split texts. Then, the split texts can be deduplicated, such as removing duplicate strings or words, to obtain deduplicated split texts. The deduplicated split texts can also be grouped, for example, into general word groups and contact information word groups based on consecutive numerical features. The contact information word group can include information such as 1560800xxxx, A1395898xxxx, Q1300663xxxx, etc. Further, the numbers in the contact information word group can be randomly replaced to enrich the split text within that group. General word groups can contain common words, such as words frequently used in everyday life and literature, like "beautiful," "tomorrow," and "scenery."

[0127] In step 302, the split text is combined to obtain the combined text.

[0128] In some implementations, combining the split text to obtain combined text includes:

[0129] (1.1) Determine the limit on the number of split texts corresponding to the combined text;

[0130] (1.2) Within the limit of the number of split texts, select the corresponding number of target split texts from the split texts and combine them to obtain the combined text.

[0131] Among them, there is a limit to the number of split texts in the combined text. For example, in some social applications, the signature of each object is often limited in number of characters. By setting a limit on the number of split texts, the generated combined text can be more consistent with the real signature.

[0132] Within the limit on the number of split texts, a certain number of split texts can be selected as target split texts, and then these target split texts can be combined to obtain combined text. For example, within the limit on the number of split texts, the number of split texts can be randomly determined first, and then corresponding target split texts can be randomly selected from the split texts according to the number of split texts. Finally, the target split texts can be combined to form new combined text.

[0133] For example, in a general word group, a portion of the target split text can be selected, and then at least one target split text can be selected from the contact information word group. Finally, the target split texts are combined to form a combined text. The combined text can be understood as a newly generated signature text, in which the split texts are combined according to semantics.

[0134] In step 303, the combined text, the original text, and the expanded text are identified as the target text.

[0135] In some implementations, the combined text, original text, and expanded text can all be defined as the target text. The combined text can be understood as the newly generated signature text, the original text contains the identity identifier text and the signature text, and the expanded text contains the expanded identity identifier text and the expanded signature text. In other words, the target text mainly contains the text corresponding to the two main categories: identity identifier and signature.

[0136] As can be seen from steps 301 to 303, in this embodiment of the application, the original text is split and then combined to obtain combined text. Since the combined text is obtained by splitting and combining the original text, the text type of the combined text is closer to the original text that was actually obtained. However, the combined text is different from the original text, which can increase the number of texts under this text type, thereby improving the richness of the target text.

[0137] In step 230, an expanded image set is obtained, and a target expanded image that matches the original image extracted from the original image and text data is matched in the expanded image set.

[0138] The expanded image set is an image set obtained from publicly available or authorized data, containing various images. Then, the original image can be obtained from the original image and text data, and target expanded images that match the original images extracted from the original image and text data are matched from the expanded image set. The target expanded image can be an image similar to the original image.

[0139] In some implementations, matching a target augmented image that matches the original image extracted from the original image and text data within the augmented image set may include the following steps:

[0140] (1.1) Obtain the first image features of the expanded image from the expanded image set;

[0141] (1.2) Obtain the second image features of the original image extracted from the original image and text data;

[0142] (1.3) Match the first image features and the second image features to obtain a similarity score between the augmented image and the original image;

[0143] (1.4) The augmented image with a similarity score higher than the preset score threshold is identified as the target augmented image that matches the original image.

[0144] The augmented image set contains multiple augmented images. The first image feature of each augmented image can be obtained; for example, a trained image feature extraction model can be used to extract the first image feature corresponding to the augmented image. The original image can be input into the trained image feature extraction model, which outputs the second image feature of the original image.

[0145] Then, the first image feature and the second image feature are matched pairwise to obtain the similarity score between the first image feature and the second image feature, which can be understood as the similarity score between the corresponding augmented image and the original image.

[0146] For example, the cosine distance between the first image feature and the second image feature can be determined. Different cosine distances correspond to different similarity scores. When the similarity score is greater than a preset score threshold, it indicates that the first image feature and the second image feature match. In this case, the expanded image corresponding to the first image feature is considered to match the original image corresponding to the second image feature. When the similarity score is less than or equal to the preset score threshold, it indicates that the first image feature and the second image feature do not match. In this case, the expanded image corresponding to the first image feature is considered to match the original image corresponding to the second image feature.

[0147] An augmented image with a similarity score higher than a preset score threshold can be identified as a target augmented image that matches the original image.

[0148] In some implementations, a first identifier value for each target augmented image and a second identifier value for each original image can be obtained. The first and second identifier values ​​are then matched to determine the target first identifier value that matches the second identifier value. The target augmented image corresponding to the target first identifier value is then identified as the same image as the original image, and the target augmented image corresponding to the target first identifier value can be deleted. Here, the identifier value can be understood as an identity identifier corresponding to the image; it is a calculated unique value that can represent the identity of the image. The identifier value can be a hash value, an MD5 value, etc.

[0149] In step 240, the original image and the target augmented image are determined as the target image, and the target image and target text are combined to obtain augmented image-text data, which is used for training the image-text recognition model.

[0150] After obtaining the target augmented image, both the target augmented image and the original image can be identified as the target image. The target image can then be used to generate augmented text and image data.

[0151] In some implementations, determining the target image as the original image and the target augmented image includes:

[0152] (1.1) Determine the descriptive text corresponding to each image in the original image and the target augmented image, and generate the corresponding output image based on the descriptive text;

[0153] (1.2) The output image, the original image and the target augmented image are determined as the target image.

[0154] In some implementations, the descriptive text corresponding to each image in the original image and the target augmented image is determined, and the corresponding output image is generated based on the descriptive text, including:

[0155] (1.1.1) Input each image from the original image and the target augmented image into the trained text generation model and output the descriptive text corresponding to each image;

[0156] (1.1.2) Input the descriptive text into the trained image generation model and output the output image corresponding to each descriptive text.

[0157] The original image and the target augmented image can be input into a pre-trained text generation model. This model outputs descriptive text for each original image and descriptive text for each target augmented image. The descriptive text describes the information within the image. For example, if the original image contains a female cartoon head against a blue sky and white clouds background, the descriptive text can describe the information in the image.

[0158] After generating the descriptive text, it can be input into the image generation model. This model is pre-trained and outputs a corresponding image based on the descriptive text. The image information contained in the output image is the same as or similar to that described in the text. For example, the output image might contain a generated female cartoon avatar and a generated blue sky and white clouds background image. The image generation model can generate multiple output images based on the descriptive text, and these output images will differ from each other. Finally, the output image, the original image, and the target augmented image are used to determine the target image.

[0159] As shown above, after obtaining the target text and target image, the target image and target text can be combined to obtain expanded image-text data. Expanded image-text data differs from the original image-text data; it is newly generated image-text data to enhance its richness.

[0160] Please see Figure 5 , Figure 5 This is a flowchart illustrating step 240 provided in an embodiment of this application. In some implementations, combining the target image and target text to obtain augmented image and text data may include the following steps:

[0161] Step 401: Determine at least one data type corresponding to the original image and text data;

[0162] Step 402: Filter the target image and target text according to the data type to obtain the target image and target text corresponding to each data type;

[0163] Step 403: Combine the target image and target text corresponding to each data type to obtain the extended image and text data corresponding to each data type.

[0164] Steps 401 to 403 will be described in detail below.

[0165] In step 401, at least one data type corresponding to the original graphic data is determined.

[0166] The original image and text data can correspond to at least one data type. For example, the data type can include different age groups, and the object corresponding to each age group can be manipulated to generate the original image and text data corresponding to each age group.

[0167] For example, data types can be different regions, specifically the region to which the original image and text data belongs. Data types can also be different generation time periods, specifically the generation time period to which the original image and text data belongs, etc., without any restrictions.

[0168] In step 402, the target image and target text are filtered according to the data type to obtain the target image and target text corresponding to each data type.

[0169] Taking the age range of the object corresponding to the original image and text data as an example, the age range can be multiple age ranges, such as 18-24 years old, 25-34 years old, 35-45 years old, etc. Each age range corresponds to the original image and text data. For example, the original image and text data includes identification text, signature text, avatar and background image. Objects in each age range can set the corresponding identification text, signature text, avatar and background image according to their own preferences, thus forming the original image and text data corresponding to each age range.

[0170] The original text and image data contain the original text and the original image. Target text is generated based on the original text corresponding to each age group, and target image is generated based on the original image corresponding to each age group. Therefore, there are related target text and target image for each age group. The target text and target image corresponding to each age group can be filtered out from all the target text and target image.

[0171] In step 403, the target image and target text corresponding to each data type are combined to obtain the extended graphic data corresponding to each data type.

[0172] For example, after determining the target text and target image for each age group, the text and image to be combined can be selected from the target text and target image for each age group. Then, the text and image to be combined for each age group are combined to generate expanded text-image data for each age group. This expansion process can be repeated multiple times, each time yielding a corresponding amount of expanded text-image data for each age group, until a certain scale of expanded text-image data for each age group is obtained, such as 500 or 1000 times, until the amount of expanded text-image data for each age group meets the requirements of the downstream training task. The expanded text-image data may include expanded identity identification text, signature text, avatar, and background image.

[0173] Since the target text and images for each age group are generated based on the original text and images for that age group, the augmented text-image data generated by combining the target text and images for each age group is closer to the real original text-image data. This makes the augmented text-image data more closely resemble the real text-image data for the corresponding age group, thus ensuring the authenticity of the augmented text-image data for that age group and improving data quality. Furthermore, it is not limited to augmenting a single original text-image dataset; instead, it combines and augments original text-image datasets from the same group simultaneously to further enhance the richness of the augmented data. Subsequent downstream model training scenarios require the use of augmented text-image data. Because the augmented text-image data is closer to the real text-image data and has higher richness, it can improve the accuracy and robustness of downstream model training.

[0174] As can be seen from steps 401 to 403, in this embodiment of the application, by combining the target image and target text corresponding to each data type, the augmented image and text data corresponding to each data type is obtained. This can improve the authenticity of the augmented image and text data, making the augmented image and text data closer to the real image and text data, thereby improving the quality of the augmented image and text data and preventing the generation of redundant data.

[0175] In some implementations, the target image and target text are combined to obtain augmented graphic data, including:

[0176] (2.1) Determine the image category of the target image and the text category of the target text;

[0177] (2.2) Select any one target image from the target images of each image category as the selected target image, and select any one target text from the target text of each text category as the selected target text;

[0178] (2.3) Generate expanded graphic data by combining the selected target text and the selected target image.

[0179] In this embodiment, the original image contains two image categories: an avatar image category and a background image category. The original text contains two text categories: an identity identification text category and a signature text category.

[0180] Therefore, the generated target images corresponding to the original image also have categories of avatar image and background image. Similarly, the generated target text corresponding to the original text also has categories of identity text and signature text.

[0181] Then, you can select any target image from the target images corresponding to the avatar image category, any target image from the target images corresponding to the background image category, any target text from the target text corresponding to the identity identification text category, and any target text from the target text corresponding to the signature text category. Finally, combine these selected target images and selected target texts to generate expanded image and text data. It should be noted that the steps for expanding image and text data described above can be repeated multiple times, such as 500 or 1000 times, depending on the actual amount of image and text data required, to obtain expanded image and text data of a certain number of dimensions.

[0182] In some implementations, augmented text and image data is generated by combining selected target text and selected target image, including:

[0183] (2.3.1) Perform at least one of the following operations on the selected target text: text addition, text deletion, text replacement, and text back translation, to obtain the enhanced target text;

[0184] (2.3.2) Perform at least one of the following operations on the selected target image: image size adjustment, image color parameter adjustment, and image brightness parameter adjustment, to obtain an enhanced target image;

[0185] (2.3.3) Combine the enhanced target image and enhanced target text to obtain expanded image and text data.

[0186] For example, adding or deleting text from selected target text can be achieved in the following way:

[0187] Random insertion: Randomly inserts words into the selected target text to generate enhanced target text. This method can increase the length and complexity of the selected target text.

[0188] Random deletion: Randomly deletes some words from the selected target text to produce enhanced target text. This method can reduce the length and complexity of the selected target text while preserving most of its semantics.

[0189] For example, text replacement of selected target text can be achieved in the following way:

[0190] Random swapping: Randomly swapping two words in a selected target text produces enhanced target text. This method preserves the semantics of the selected target text while increasing the diversity of the text data.

[0191] Synonym replacement: This method replaces selected words in the target text with their synonyms to produce enhanced target text. It preserves the semantics of the selected target text while increasing the diversity of the text data.

[0192] For example, back-translating selected target text can be achieved in the following way:

[0193] The selected target text is translated into another language and then translated back into the original language to produce enhanced target text. This method can generate enhanced target text that is semantically similar to the selected target text but expresses it differently, thereby increasing the diversity of text data.

[0194] For example, image resizing of a selected target image can be achieved in the following way:

[0195] Cropping: Randomly cropping a region from a selected target image to produce an enhanced target image. This method can focus on a specific region in the selected target image while increasing the diversity of the image data.

[0196] For example, adjusting the color parameters of a selected target image can be achieved in the following way:

[0197] The saturation and hue of a selected target image can be randomly adjusted to produce an enhanced target image.

[0198] For example, adjusting the image brightness adoption rate of a selected target image can be achieved in the following way:

[0199] The brightness and contrast of a selected target image can be randomly adjusted to produce an enhanced target image.

[0200] In addition to the image adjustment methods mentioned above, the selected target image can also be rotated: rotating the selected target image by a certain angle produces an enhanced target image. This method can increase the diversity of image data while retaining most of the information of the selected target image.

[0201] Flipping: Flipping an image horizontally or vertically to produce an enhanced target image. This method can increase the diversity of image data while retaining most of the information in the selected target image.

[0202] Translation: The selected target image is translated a certain distance horizontally or vertically to produce an enhanced target image. This method can increase the diversity of image data while retaining most of the information of the selected target image.

[0203] Scaling: Scales the selected target image proportionally to produce an enhanced image. This method can increase the diversity of image data while retaining most of the information in the selected target image.

[0204] Noise addition: Random noise, such as Gaussian noise or salt-and-pepper noise, is added to a selected target image to enhance it. This method can simulate the performance of an image under different quality conditions, thereby increasing the diversity of image data.

[0205] Blur: Blur the selected target image using techniques such as Gaussian blur or median blur to produce an enhanced target image. This method can simulate the image's performance under different sharpness conditions, thereby increasing the diversity of image data.

[0206] Geometric transformation: Applying geometric transformations, such as affine transformations and perspective transformations, to a selected target image to produce an enhanced image. This method can simulate the appearance of an image under different viewpoints and shape changes, thereby increasing the diversity of image data.

[0207] After obtaining the target image and the enhanced target text, the enhanced target image and the enhanced target text can be combined to obtain augmented image and text data.

[0208] In some implementations, after obtaining the augmented image and text data, the original image and text data and the augmented image and text data can be combined into an image and text dataset, which can be used to train an image and text recognition model or to improve the amount and richness of data in the database.

[0209] Compared to related technologies that only augment data for one modality, this application's embodiments can perform multimodal augmentation on both text and images, and then combine the data from different modalities, thereby greatly increasing the amount and richness of the augmented text and image data. Because the augmented text and image data is richer, the accuracy and robustness of downstream text and image recognition model training can be improved. The text and image recognition model can be a large model, such as a large model based on the transformer architecture, or other lightweight models compared to large models. The text and image recognition model can recognize images and text on a page and can combine extracted image and text features to determine the page type.

[0210] As can be seen from steps 210 to 240, in this embodiment of the application, the original image and text data is obtained, and the original text extracted from the original image and text data is augmented to obtain augmented text; the original text and augmented text are determined as target text; an augmented image set is obtained, and a target augmented image that matches the original image extracted from the original image and text data is matched in the augmented image set; the original image and target augmented image are determined as target image, and the target image and target text are combined to obtain augmented image and text data, which is used for training the image and text recognition model. In this embodiment, the original text in the original image and text data is augmented to obtain augmented text. The augmented text and the original text are determined as the target text. The original image in the original image and text data is augmented to obtain the target augmented image. The target augmented image and the original image are determined as the target image. Finally, the target text and the target image are randomly combined to obtain augmented image and text data. The augmented image and text data is used as new image and text data. Since the augmented image and text data is augmented in a multimodal manner through text and images, it has a large amount of data and more types of image and text data, thereby increasing the amount and richness of image and text data, and thus improving the robustness of downstream training tasks.

[0211] Please continue reading. Figure 6 , Figure 6 This is another schematic flowchart of the data processing method provided in this application embodiment. The data processing method may include the following steps:

[0212] In step 501, the identity identification text and the corresponding signature text are obtained from the original text extracted from the original image and text data.

[0213] The original image and text data can be understood as the image and text data corresponding to the page. For example, if a page has an avatar, background image, identity identification text (nickname), and signature text (signature), then the avatar and background image can be used as the original image, while the identity identification text and signature text can be used as the original text. The original image and original text constitute the original image and text data.

[0214] In step 502, corresponding text expansion instructions are generated based on the identity text and signature text.

[0215] For example, prompt text can be written through a prompting process, and then the written prompt text can be used as a text expansion instruction. The text expansion instruction is then input into the text expansion model, which is used to guide the text expansion model to output the corresponding output text.

[0216] For example, the prompt text corresponding to the text expansion instruction is:

[0217] Some social media accounts have the following nicknames and signatures:

[0218] Nickname 1 + Signature 1

[0219] Nickname 2 + Signature 2

[0220] Nickname 3 + Signature 3

[0221] Nickname 4 + Signature 4

[0222] Please write 500 similar nicknames and signatures in this style.

[0223] In step 503, the text augmentation instruction is input into the trained text augmentation model, and the augmented text is output. The augmented text includes the augmented identity text and the augmented signature text.

[0224] The prompt text is then input into the trained text augmentation model, which outputs augmented text. This augmented text includes both augmented identity information and augmented signature text.

[0225] For example, the text augmentation instructions input into a large model are:

[0226] Some social media accounts have the following nicknames and signatures:

[0227] Nickname 1 + Signature 1

[0228] Nickname 2 + Signature 2

[0229] Please write 500 similar nicknames and signatures in this style.

[0230] The expanded text output by the text expansion model is: nickname5 + signature5, nickname6 + signature6... thus realizing the text expansion of nickname1 + signature1 and nickname2 + signature2.

[0231] In step 504, the original text is split to obtain split text.

[0232] For example, the signature text can be obtained from the original text. Then, based on punctuation marks such as periods, semicolons, and exclamation marks, the signature text can be segmented into multiple sub-texts. These sub-texts can then be deduplicated, such as removing duplicate strings or words, resulting in deduplicated sub-texts. The deduplicated sub-texts can also be grouped, for example, into general word groups and contact information word groups based on consecutive numerical features. The contact information word groups can include information such as 1560800xxxx, A1395898xxxx, Q1300663xxxx, etc. Further, the numbers in the contact information word groups can be randomly replaced to enrich the sub-texts within those groups. General word groups can contain common words, such as words frequently used in everyday life and literature, like "beautiful," "tomorrow," and "scenery."

[0233] In step 505, the split text is combined to obtain combined text, and the combined text, the original text, and the expanded text are determined as the target text.

[0234] For example, in a general word group, a portion of the target split text can be selected, and then at least one target split text can be selected from the contact information word group. Finally, the target split texts are combined to form a combined text. The combined text can be understood as a newly generated signature text, in which the split texts are combined according to semantics.

[0235] Finally, the combined text, original text, and expanded text can all be identified as the target text. The combined text can be understood as the newly generated signature text, the original text contains the identity identifier text and the signature text, and the expanded text contains the expanded identity identifier text and the expanded signature text. In other words, the target text mainly contains the text corresponding to the two main categories: identity identifier and signature.

[0236] In step 506, an expanded image set is obtained, and the first image features of the expanded images are obtained from the expanded image set.

[0237] The augmented image set contains multiple augmented images, and the first image features of each augmented image can be obtained. For example, a trained image feature extraction model can be used to extract the first image features corresponding to the augmented image.

[0238] In step 507, the second image features of the original image extracted from the original image and text data are obtained.

[0239] The original image can be input into a trained image feature extraction model, which outputs the second image features of the original image.

[0240] In step 508, the first image features and the second image features are matched to obtain a similarity score between the augmented image and the original image.

[0241] For example, the cosine distance between the first image feature and the second image feature can be determined. Different cosine distances correspond to different similarity scores. When the similarity score is greater than a preset score threshold, it indicates that the first image feature and the second image feature match. In this case, the expanded image corresponding to the first image feature is considered to match the original image corresponding to the second image feature. When the similarity score is less than or equal to the preset score threshold, it indicates that the first image feature and the second image feature do not match. In this case, the expanded image corresponding to the first image feature is considered to match the original image corresponding to the second image feature.

[0242] In step 509, the augmented images with similarity scores higher than a preset score threshold are identified as target augmented images that match the original image.

[0243] After identifying augmented images with similarity scores higher than a preset threshold as target augmented images matching the original image, a first identifier value for each target augmented image and a second identifier value for each original image can be obtained. Then, the first and second identifier values ​​are matched to determine the target first identifier value that matches the second identifier value. The target augmented image corresponding to the target first identifier value is then identified as the same as the original image, and the target augmented image corresponding to the target first identifier value can be deleted. Here, the identifier value can be understood as the image's identity identifier, a calculated unique value that represents the image's identity. The identifier value can be a hash value, MD5 value, etc.

[0244] In step 510, each image from the original image and the target augmented image is input into the trained text generation model, and the descriptive text corresponding to each image is output.

[0245] The original image and the target augmented image can be input separately into a text generation model. This model is pre-trained and can output descriptive text for each original image and each target augmented image. The descriptive text describes the information within the image. For example, if the original image contains a female cartoon head against a blue sky and white clouds background, the descriptive text can describe the information in the image.

[0246] In step 511, the descriptive text is input into the trained image generation model, and the output image corresponding to each descriptive text is output.

[0247] After generating the descriptive text, it can be input into the image generation model. This model is pre-trained and outputs a corresponding image based on the descriptive text. The image information contained in the output image is the same as or similar to that described in the text. For example, the output image might contain a generated female cartoon avatar and a generated blue sky and white clouds background image. The image generation model can generate multiple output images based on the descriptive text, and these output images will differ from each other. Finally, the output image, the original image, and the target augmented image are used to determine the target image.

[0248] In step 512, the output image, the original image, and the target augmented image are determined as the target image.

[0249] In step 513, the image category of the target image and the text category of the target text are determined.

[0250] The original image contains two image categories: avatar images and background images. The original text contains two text categories: identity text and signature text.

[0251] Therefore, the generated target images corresponding to the original image also have categories of avatar image and background image. Similarly, the generated target text corresponding to the original text also has categories of identity text and signature text.

[0252] In step 514, any target image is selected from the target images of each image category as the selected target image, and any target text is selected from the target text of each text category as the selected target text.

[0253] You can select any target image from the target images corresponding to the avatar image category, any target image from the target images corresponding to the background image category, any target text from the target text corresponding to the identity identification text category, or any target text from the target text corresponding to the signature text category.

[0254] In step 515, expanded graphic data is generated based on the combination of the selected target text and the selected target image.

[0255] For example, selected avatars, background images, nicknames, and signatures can be combined to generate expanded text and image data. This expanded text and image data corresponds to a page, which can be understood as a newly generated page after data processing. Expanded text and image data is generated through multimodal processing of text and images, possessing a large data volume and richness.

[0256] It should be noted that the steps described above for expanding the text and image data can be repeated multiple times, such as 500 or 1000 times, depending on the actual amount of text and image data required, in order to obtain a certain number of dimensions of expanded text and image data.

[0257] In this embodiment, after obtaining the expanded image and text data, the expanded image and text data and the original image and text data can be combined into an image and text dataset. This image and text dataset contains a large amount of image and text data with high richness. Subsequently, the image and text recognition model can be trained using this dataset to obtain a trained image and text recognition model. Because the image and text dataset has higher richness, the accuracy and robustness of the image and text recognition model training can be improved.

[0258] Taking a social media app as an example, after registering an account, users edit their nickname, signature, avatar, and background image on their personal profile page, creating a personal page. A trained image-text recognition model can then identify this page by analyzing the image and text data. Combining text and image features extracted from the data, the model determines the page type. Subsequently, the model can determine whether the page contains black market or illegal information. This allows for the screening and investigation of personal pages, identifying potentially malicious social media accounts. This application scenario is just one example; the image-text recognition model trained using expanded image and text data in this application can also be applied to other scenarios, such as the rapid review of articles and videos.

[0259] In this embodiment, the identity identification text and the corresponding signature text are obtained from the original text extracted from the original image and text data; a corresponding text expansion instruction is generated based on the identity identification text and the signature text, and the text expansion instruction is input into the trained text expansion model to output the expanded text, which includes the expanded identity identification text and the expanded signature text; the split text is combined to obtain the combined text, and the combined text, the original text, and the expanded text are determined as the target text.

[0260] Then, an expanded image set is acquired, and the first image features of the expanded images are obtained from the expanded image set. The second image features of the original images extracted from the original image and text data are also acquired. The first and second image features are matched to obtain a similarity score between the expanded image and the original image. Expanded images with similarity scores higher than a preset threshold are identified as target expanded images that match the original image. Each image from the original image and the target expanded image is then input into a trained text generation model, which outputs descriptive text corresponding to each image. The descriptive text is then input into a trained image generation model, which outputs an output image corresponding to each descriptive text. The output image, the original image, and the target expanded image are then identified as the target image.

[0261] Finally, the image category of the target image and the text category of the target text are determined. For each image category, one target image is selected as the chosen target image; for each text category, one target text is selected as the chosen target text. Augmented text-image data is generated by combining the chosen target text and the chosen target image. In this way, by defining the combined text, original text, and augmented text as the target text, and the output image, original image, and target augmented image as the target image, and then combining the target text and target image, the augmented text-image data is obtained. This augmented text-image data is used as the new text-image data. Because the augmented text-image data is multimodal augmented using text and images, it has a larger data volume and more types of text-image data, thus increasing the data volume and richness of the text-image data, and thereby improving the robustness of downstream training tasks.

[0262] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application. This data processing apparatus can be used to execute the data processing methods described in the above embodiments.

[0263] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0264] like Figure 7 As shown, the data processing device 600 includes:

[0265] The text expansion module 610 is used to acquire the original image and text data and expand the original text extracted from the original image and text data to obtain expanded text.

[0266] The determination module 620 is used to determine the original text and the expanded text as the target text;

[0267] The image augmentation module 630 is used to acquire an augmented image set and match the target augmented image that matches the original image extracted from the original image and text data in the augmented image set.

[0268] The combination module 640 is used to determine the target image from the original image and the target augmented image, and to combine the target image and the target text to obtain augmented image-text data, which is used for training the image-text recognition model.

[0269] In some implementations, the determining module 620 includes a splitting submodule, a combining submodule, and a determining submodule;

[0270] The splitting submodule is used to split the original text to obtain the split text.

[0271] The combination submodule is used to combine split text to obtain combined text;

[0272] The Determine submodule is used to determine the target text from the combined text, the original text, and the expanded text.

[0273] In some implementations, the combined submodule is used for:

[0274] Determine the limit on the number of split texts corresponding to the combined text;

[0275] Within the limit on the number of split texts, select the corresponding number of target split texts from the split texts and combine them to obtain the combined text.

[0276] In some implementations, the text expansion module 610 is used for:

[0277] Extract the identity identifier text and the corresponding signature text from the original text extracted from the original image and text data;

[0278] Generate corresponding text expansion instructions based on the identity identification text and signature text;

[0279] The text augmentation instructions are input into the trained text augmentation model, which outputs augmented text, including augmented identity text and augmented signature text.

[0280] In some embodiments, the image augmentation module 630 is used for

[0281] Obtain the first image features of the expanded image from the expanded image set;

[0282] Obtain the second image features of the original image extracted from the original image and text data;

[0283] The first image features and the second image features are matched to obtain a similarity score between the augmented image and the original image.

[0284] Expanded images with similarity scores higher than a preset score threshold are identified as target expanded images that match the original image.

[0285] In some implementations, the combining module 640 includes an image generation submodule and a combining submodule;

[0286] The image generation submodule is used to determine the descriptive text corresponding to each image in the original image and the target augmented image, and generate the corresponding output image based on the descriptive text;

[0287] The combination submodule is used to determine the target image from the output image, the original image, and the target augmented image.

[0288] In some implementations, the image generation submodule is used for:

[0289] Each image from the original image and the target augmented image is input into the trained text generation model, which outputs descriptive text corresponding to each image.

[0290] The descriptive text is input into the trained image generation model, which outputs the corresponding image for each descriptive text.

[0291] In some implementations, the combination module 640 is used for:

[0292] Identify at least one data type corresponding to the original image and text data;

[0293] The target image and target text are filtered according to their data types to obtain the target image and target text corresponding to each data type;

[0294] The target image and target text corresponding to each data type are combined to obtain the augmented image and text data corresponding to each data type.

[0295] In some implementations, the combination module 640 includes a category determination submodule, a selection submodule, and a combination submodule;

[0296] The determination submodule is used to determine the image category of the target image and the text category of the target text;

[0297] The selection submodule is used to select any one target image from the target images of each image category as the selected target image, and to select any one target text from the target text of each text category as the selected target text;

[0298] The combination submodule is used to generate expanded graphic data by combining selected target text and selected target image.

[0299] In some implementations, the combined submodule is used for:

[0300] Perform at least one of the following operations on the selected target text: text addition, text deletion, text replacement, and text back translation, to obtain enhanced target text;

[0301] The selected target image is subjected to at least one of the following operations: image size adjustment, image color parameter adjustment, and image brightness parameter adjustment, to obtain an enhanced target image;

[0302] By combining the enhanced target image and the enhanced target text, augmented image and text data is obtained.

[0303] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed description of the data processing method above, which will not be repeated here.

[0304] In this embodiment, the text augmentation module 610 acquires the original image and text data and augments the original text extracted from the original image and text data to obtain augmented text; the determination module 620 determines the original text and augmented text as target text; the image augmentation module 630 acquires an augmented image set and matches the target augmented image that matches the original image extracted from the original image and text data in the augmented image set; the combination module 640 determines the original image and target augmented image as target image and combines the target image and target text to obtain augmented image and text data, which is used for training the image and text recognition model. In this embodiment, the original text in the original image and text data is augmented to obtain augmented text. The augmented text and the original text are determined as the target text. The original image in the original image and text data is augmented to obtain the target augmented image. The target augmented image and the original image are determined as the target image. Finally, the target text and the target image are randomly combined to obtain augmented image and text data. The augmented image and text data is used as new image and text data. Since the augmented image and text data is augmented in a multimodal manner through text and images, it has a large amount of data and more types of image and text data, thereby increasing the amount and richness of image and text data, and thus improving the robustness of downstream training tasks.

[0305] This application also provides a computer device, which may be a server, such as... Figure 8 As shown, it illustrates a schematic diagram of the server structure involved in an embodiment of this application. Specifically:

[0306] The server may include components such as a processor 701 with one or more processing cores, a memory 702 with one or more computer-readable storage media, a power supply 703, and an input unit 704. Those skilled in the art will understand that... Figure 8 The server architecture shown does not constitute a limitation on the server and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Wherein:

[0307] The processor 701 is the control center of the server, connecting various parts of the server via various interfaces and lines. It performs various server functions and processes data by running or executing software programs and / or modules stored in the memory 702, and by calling data stored in the memory 702, thereby providing overall control of the server. Optionally, the processor 701 may include one or more processing cores; preferably, the processor 701 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 701.

[0308] The memory 702 can be used to store software programs and modules. The processor 701 executes various functional applications and data processing by running the software programs and modules stored in the memory 702. The memory 702 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 702 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 702 may also include a memory controller to provide the processor 701 with access to the memory 702.

[0309] The computer device also includes a power supply 703 that supplies power to the various components. Optionally, the power supply 703 can be logically connected to the processor 701 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 703 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0310] The computer device may also include an input unit 704, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0311] Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 701 in the computer device loads the executable files corresponding to the processes of one or more application programs into the memory 702 according to the following instructions, and the processor 701 runs the application programs stored in the memory 702, thereby implementing the various method steps provided in the foregoing embodiments, as follows:

[0312] Obtain the original image and text data, and expand the original text extracted from the original image and text data to obtain the expanded text;

[0313] The original text and the expanded text are identified as the target text;

[0314] Obtain the expanded image set, and match the target expanded image that matches the original image extracted from the original image and text data in the expanded image set;

[0315] The original image and the target augmented image are identified as the target image. The target image and the target text are then combined to obtain augmented image-text data, which is used to train the image-text recognition model.

[0316] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed description of the data processing method above, which will not be repeated here.

[0317] In this embodiment, the process involves acquiring original image and text data, augmenting the original text extracted from the original image and text data to obtain augmented text, determining the original text and augmented text as target text, acquiring an augmented image set, matching the target augmented image in the augmented image set with the original image extracted from the original image and text data, determining the original image and target augmented image as target image, and combining the target image and target text to obtain augmented image and text data, which is used for training the image and text recognition model. In this embodiment, the original text in the original image and text data is augmented to obtain augmented text. The augmented text and the original text are determined as the target text. The original image in the original image and text data is augmented to obtain the target augmented image. The target augmented image and the original image are determined as the target image. Finally, the target text and the target image are randomly combined to obtain augmented image and text data. The augmented image and text data is used as new image and text data. Since the augmented image and text data is augmented in a multimodal manner through text and images, it has a large amount of data and more types of image and text data, thereby increasing the amount and richness of image and text data, and thus improving the robustness of downstream training tasks.

[0318] This application also provides a computer device, which can be a terminal, such as... Figure 9 As shown, it illustrates the structural diagram of the terminal involved in the embodiments of this application, specifically:

[0319] The computer device may include a radio frequency (RF) circuit 801, a memory 802 including one or more computer-readable storage media, an input unit 803, a display unit 804, a sensor 805, an audio circuit 806, a wireless fidelity (WiFi) module 807, a processor 808 including one or more processing cores, and a power supply 809, etc. Those skilled in the art will understand that... Figure 9 The terminal structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0320] RF circuit 801 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and hands it over to one or more processors 808 for processing; additionally, it transmits uplink data to the base station. Typically, RF circuit 801 includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, RF circuit 801 can also communicate wirelessly with networks and other devices. Wireless communication can use any communication standard or protocol, including but not limited to GSM, GPRS, CDMA, WCDMA, LTE, email, and SMS.

[0321] The memory 802 can be used to store software programs and modules. The processor 808 executes various functional applications and information retrieval by running the software programs and modules stored in the memory 802. The memory 802 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the terminal (such as audio data, phone book, etc.). In addition, the memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 802 may also include a memory controller to provide access to the memory 802 for the processor 808 and the input unit 803.

[0322] Input unit 803 can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to object settings and function control. Specifically, in one embodiment, input unit 803 may include a touch-sensitive surface and other input devices. A touch-sensitive surface, also known as a touch display or touchpad, can collect touch operations on or near the object (e.g., operations performed by the object using a finger, stylus, or any suitable object or accessory on or near the touch-sensitive surface), and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface may include a touch detection device and a touch controller. The touch detection device detects the touch orientation of the object and the signal generated by the touch operation, transmitting the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 808, and can receive and execute commands from the processor 808. Furthermore, various types of touch-sensitive surfaces, such as resistive, capacitive, infrared, and surface acoustic wave, can be used. In addition to the touch-sensitive surface, input unit 803 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0323] The display unit 804 can be used to display information input by an object or information provided to an object, as well as various graphical object interfaces of the terminal. These graphical object interfaces can be composed of graphics, text, icons, video, and any combination thereof. The display unit 804 may include a display panel, optionally configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. Furthermore, a touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to the processor 808 to determine the type of touch event. Subsequently, the processor 808 provides corresponding visual output on the display panel according to the type of touch event.

[0324] The terminal may also include at least one sensor 805, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel according to the ambient light level, and the proximity sensor can turn off the display panel and / or backlight when the terminal is moved to the ear. As a type of motion sensor, a gravity acceleration sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, tapping), etc. Other sensors that the terminal may also be equipped with, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0325] Audio circuitry 806, a speaker, and a microphone provide an audio interface between the device and the terminal. Audio circuitry 806 converts received audio data into electrical signals, transmits them to the speaker, and the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 806, converted back into audio data, and processed by processor 808. The processed data is then transmitted via RF circuitry 801 to, for example, another terminal, or output to memory 802 for further processing. Audio circuitry 806 may also include an earphone jack to facilitate communication between a peripheral headset and the terminal.

[0326] WiFi is a short-range wireless transmission technology. A terminal using a WiFi module 807 can help users send and receive emails, browse web pages, and access streaming media, providing wireless broadband internet access. Although Figure 9 WiFi module 807 is shown, but it is understood that it is not a necessary component of the terminal and can be omitted as needed without changing the essence of the invention.

[0327] The processor 808 is the control center of the terminal, connecting various parts of the phone via various interfaces and lines. It executes software programs and / or modules stored in the memory 802, and calls data stored in the memory 802, to perform various functions and process data, thereby providing overall monitoring of the phone. Optionally, the processor 808 may include one or more processing cores; preferably, the processor 808 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 808.

[0328] The terminal also includes a power supply 809 (such as a battery) to power various components. Preferably, the power supply can be logically connected to the processor 808 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 809 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0329] Although not shown, the terminal may also include a camera, Bluetooth module, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 808 in the terminal loads the executable files corresponding to the processes of one or more applications into the memory 802 according to the following instructions, and the processor 808 runs the applications stored in the memory 802 to realize various functions:

[0330] Obtain the original image and text data, and expand the original text extracted from the original image and text data to obtain the expanded text;

[0331] The original text and the expanded text are identified as the target text;

[0332] Obtain the expanded image set, and match the target expanded image that matches the original image extracted from the original image and text data in the expanded image set;

[0333] The original image and the target augmented image are identified as the target image. The target image and the target text are then combined to obtain augmented image-text data, which is used to train the image-text recognition model.

[0334] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed description of the data processing method above, which will not be repeated here.

[0335] In this embodiment, the process involves acquiring original image and text data, augmenting the original text extracted from the original image and text data to obtain augmented text, determining the original text and augmented text as target text, acquiring an augmented image set, matching the target augmented image in the augmented image set with the original image extracted from the original image and text data, determining the original image and target augmented image as target image, and combining the target image and target text to obtain augmented image and text data, which is used for training the image and text recognition model. In this embodiment, the original text in the original image and text data is augmented to obtain augmented text. The augmented text and the original text are determined as the target text. The original image in the original image and text data is augmented to obtain the target augmented image. The target augmented image and the original image are determined as the target image. Finally, the target text and the target image are randomly combined to obtain augmented image and text data. The augmented image and text data is used as new image and text data. Since the augmented image and text data is augmented in a multimodal manner through text and images, it has a large amount of data and more types of image and text data, thereby increasing the amount and richness of image and text data, and thus improving the robustness of downstream training tasks.

[0336] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0337] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the data processing methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0338] Obtain the original image and text data, and expand the original text extracted from the original image and text data to obtain the expanded text;

[0339] The original text and the expanded text are identified as the target text;

[0340] Obtain the expanded image set, and match the target expanded image that matches the original image extracted from the original image and text data in the expanded image set;

[0341] The original image and the target augmented image are identified as the target image. The target image and the target text are then combined to obtain augmented image-text data, which is used to train the image-text recognition model.

[0342] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed description of the data processing method above, which will not be repeated here.

[0343] In this embodiment, the process involves acquiring original image and text data, augmenting the original text extracted from the original image and text data to obtain augmented text, determining the original text and augmented text as target text, acquiring an augmented image set, matching the target augmented image in the augmented image set with the original image extracted from the original image and text data, determining the original image and target augmented image as target image, and combining the target image and target text to obtain augmented image and text data, which is used for training the image and text recognition model. In this embodiment, the original text in the original image and text data is augmented to obtain augmented text. The augmented text and the original text are determined as the target text. The original image in the original image and text data is augmented to obtain the target augmented image. The target augmented image and the original image are determined as the target image. Finally, the target text and the target image are randomly combined to obtain augmented image and text data. The augmented image and text data is used as new image and text data. Since the augmented image and text data is augmented in a multimodal manner through text and images, it has a large amount of data and more types of image and text data, thereby increasing the amount and richness of image and text data, and thus improving the robustness of downstream training tasks.

[0344] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing methods provided in the various optional implementations of the above embodiments.

[0345] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0346] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0347] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the data processing methods provided in the embodiments of this application, the beneficial effects that any of the data processing methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0348] The data processing method, apparatus, computer equipment, and storage medium provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method, characterized in that, include: Obtain the original image and text data, and expand the original text extracted from the original image and text data to obtain expanded text; The original text and the expanded text are identified as the target text; Obtain an expanded image set, and match the target expanded image in the expanded image set with the original image extracted from the original image and text data; The original image and the target augmented image are identified as the target image, and the target image and the target text are combined to obtain augmented image-text data, which is used for training the image-text recognition model.

2. The data processing method according to claim 1, characterized in that, The step of determining the original text and the expanded text as the target text includes: The original text is split to obtain the split text; The split text is combined to obtain the combined text; The combined text, the original text, and the expanded text are identified as the target text.

3. The data processing method according to claim 2, characterized in that, The process of combining the split text to obtain combined text includes: Determine the limit on the number of split texts corresponding to the combined text; Within the specified limit on the number of split texts, a corresponding number of target split texts are selected from the split texts and combined to obtain the combined text.

4. The data processing method according to claim 1, characterized in that, The step of augmenting the original text extracted from the original image and text data to obtain augmented text includes: The identity identification text and the corresponding signature text are obtained from the original text extracted from the original image and text data. Generate corresponding text expansion instructions based on the identity identification text and the signature text; The text augmentation instructions are input into the trained text augmentation model, and the augmented text is output. The augmented text includes the augmented identity text and the augmented signature text.

5. The data processing method according to claim 1, characterized in that, The step of matching the target augmented image in the augmented image set with the original image extracted from the original image and text data includes: Obtain the first image feature of the expanded image from the expanded image set; Obtain the second image features of the original image extracted from the original image and text data; The first image feature and the second image feature are matched to obtain a similarity score between the augmented image and the original image; The augmented image whose similarity score is higher than a preset score threshold is identified as the target augmented image that matches the original image.

6. The data processing method according to claim 1, characterized in that, The step of determining the original image and the target augmented image as the target image includes: Determine the descriptive text corresponding to each image in the original image and the target augmented image, and generate the corresponding output image based on the descriptive text; The output image, the original image, and the target augmented image are determined as the target image.

7. The data processing method according to claim 6, characterized in that, The step of determining the descriptive text corresponding to each image in the original image and the target augmented image, and generating the corresponding output image based on the descriptive text, includes: Each image from the original image and the target augmented image is input into the trained text generation model, and the descriptive text corresponding to each image is output. The descriptive text is input into the trained image generation model, which outputs the corresponding image for each descriptive text.

8. The data processing method according to claim 1, characterized in that, The combination of the target image and the target text yields augmented image and text data, including: Determine at least one data type corresponding to the original graphic data; The target image and target text are filtered according to the data type to obtain the target image and target text corresponding to each data type; The target image and target text corresponding to each data type are combined to obtain the augmented image and text data corresponding to each data type.

9. The data processing method according to claim 1, characterized in that, The combination of the target image and the target text yields augmented image and text data, including: Determine the image category of the target image and the text category of the target text; Select any one of the target images in each image category as the selected target image, and select any one of the target texts in each text category as the selected target text; Expanded graphic data is generated by combining the selected target text and the selected target image.

10. The data processing method according to claim 9, characterized in that, The step of generating expanded graphic data based on the selected target text and the selected target image includes: Perform at least one of the following operations on the selected target text: text addition, text deletion, text replacement, and text back translation, to obtain enhanced target text; The selected target image is subjected to at least one of the following operations: image size adjustment, image color parameter adjustment, and image brightness parameter adjustment, to obtain an enhanced target image; The enhanced target image and the enhanced target text are combined to obtain augmented image and text data.

11. A data processing apparatus, characterized in that, include: The text expansion module is used to acquire the original image and text data and expand the original text extracted from the original image and text data to obtain expanded text. The determination module is used to determine the original text and the expanded text as the target text; An image augmentation module is used to acquire an augmented image set and match the target augmented image in the augmented image set with the original image extracted from the original image and text data. The combination module is used to determine the original image and the target augmented image as the target image, and combine the target image and the target text to obtain augmented image-text data, which is used for training the image-text recognition model.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the data processing method according to any one of claims 1 to 10.

13. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data processing method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the data processing method according to any one of claims 1 to 10.

Images