Training sample augmentation method, model training method, device and equipment

By amplifying the training samples of the visual text model, augmented triple samples are generated, which solves the problem of poor robustness caused by limited training samples and significantly improves the processing capability of the model.

CN118968211APending Publication Date: 2024-11-15BEIJING ZITIAO NETWORK TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410924316.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-10
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The robustness of the visual text model is poor, mainly due to the high time and labor cost of manually labeling training samples, resulting in limited training samples and the inability to effectively deal with minor changes in input instructions.

Method used

By amplifying the original instructions in the original triple sample, multiple amplification instructions are generated, and augmented triple samples are generated based on the target amplification instructions of semantic matching, realizing the automatic expansion of triple samples and enriching Training sample set.

Benefits of technology

It significantly improves the robustness of the visual text model, and fine-tuning the model through a rich training sample set, improving the processing ability of input instructions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968211A_ABST
    Figure CN118968211A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a training sample augmentation method, a model training method, devices and equipment, and the method comprises the steps: carrying out the augmentation of an original instruction in an original triple sample, and obtaining a plurality of augmentation instructions corresponding to the original instruction; determining a target amplification instruction matched with the semantics of the original instruction in the amplification instructions; and generating an augmented triple sample according to the original triple sample and the target amplification instruction. According to the method and the device, automatic expansion of the triple samples is realized, and the content and the quantity of the triple samples are greatly enriched. Furthermore, the training of the visual text model is performed based on a large number of triple samples with rich contents, so that the robustness of the visual text model can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a training sample augmentation method, a model training method, a device and equipment. Background Art

[0002] With the continuous development of artificial intelligence technology, visual text models have become well known to the public. Due to the high time and labor costs of manually annotating training samples, the training samples used in visual text models are limited, which leads to poor robustness of visual text models. A slight change in the input question may give an incorrect answer. Summary of the invention

[0003] In view of this, the present disclosure provides a training sample augmentation method, a model training method, an apparatus and a device to solve the problem of poor robustness of the visual text model.

[0004] In a first aspect, the present disclosure provides a training sample augmentation method, the method comprising:

[0005] Amplifying the original instruction in the original triple sample to obtain multiple amplified instructions corresponding to the original instruction;

[0006] Determine a target augmented instruction in the augmented instruction that matches the semantics of the original instruction;

[0007] Generate augmented triplet samples based on the original triplet samples and the target amplification instructions.

[0008] In the training sample augmentation method provided by the present disclosure, the original instructions in the original triple sample are augmented to obtain multiple augmented instructions corresponding to the original instructions, and the augmented triple sample is generated according to the target augmented instruction that matches the semantics of the original instruction and the original triple sample. In this way, the automatic expansion of the triple sample is achieved, which greatly enriches the content and quantity of the triple sample. Furthermore, the training of the visual text model based on the large number of triple samples with rich content can improve the robustness of the visual text model.

[0009] In a second aspect, the present disclosure provides a visual text model training method, the method comprising:

[0010] Acquire a target training sample from a training sample set; the training sample set includes an original triple sample and an augmented triple sample of the original triple sample; the augmented triple sample is obtained according to the training sample augmentation method provided in the first aspect;

[0011] The target training sample is used to fine-tune the pre-trained initial visual text model to obtain a target visual text model.

[0012] In the visual text model training method provided in this embodiment, a target visual text model is obtained by obtaining a target training sample from a training sample set including original triple samples and augmented triple samples of the original triple samples, and fine-tuning the pre-trained initial visual text model using the target training sample. Since the augmented triple samples in the training sample set are obtained by augmenting the original triple samples, the content and quantity of the triple samples are greatly enriched, and then, fine-tuning the visual text model based on the large number of triple samples with rich content can improve the robustness of the obtained target visual text model.

[0013] In a third aspect, the present disclosure provides a training sample augmentation device, the device comprising:

[0014] An amplification module, used to amplify the original instruction in the original triple sample to obtain multiple amplified instructions corresponding to the original instruction;

[0015] A determination module, used for determining a target augmented instruction in the augmented instruction that matches the semantics of the original instruction;

[0016] The generation module is used to generate augmented triplet samples according to the original triplet samples and the target amplification instructions.

[0017] In a fourth aspect, the present disclosure provides a visual text model training device, the device comprising:

[0018] An acquisition module, used to acquire a target training sample from a training sample set; the training sample set includes an original triple sample and an augmented triple sample of the original triple sample; the augmented triple sample is obtained according to the training sample augmentation method provided in the first aspect;

[0019] The fine-tuning module is used to fine-tune the pre-trained initial visual text model using the target training samples to obtain the target visual text model.

[0020] In a fifth aspect, the present disclosure provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the training sample augmentation method of the above-mentioned first aspect or any corresponding embodiment thereof, or executes the visual text model training method of the above-mentioned second aspect or any corresponding embodiment thereof by executing the computer instructions.

[0021] In a sixth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the training sample augmentation method of the above-mentioned first aspect or any corresponding embodiment thereof, or to execute the visual text model training method of the above-mentioned second aspect or any corresponding embodiment thereof.

[0022] In a seventh aspect, the present disclosure provides a computer program product comprising computer instructions, which are used to enable a computer to execute the training sample augmentation method of the above-mentioned first aspect or any corresponding embodiment thereof, or to execute the visual text model training method of the above-mentioned second aspect or any corresponding embodiment thereof.

[0023] In the present disclosure, the automatic expansion of triple samples is realized, which greatly enriches the content and quantity of triple samples. Furthermore, the training of the visual text model based on the large number of triple samples with rich content can improve the robustness of the visual text model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 is a flowchart of a training sample augmentation method according to an embodiment of the present disclosure;

[0026] Figure 2 is a flowchart of another training sample augmentation method according to an embodiment of the present disclosure;

[0027] Figure 3 is a flowchart of another training sample augmentation method according to an embodiment of the present disclosure;

[0028] Figure 4 is a schematic diagram of a training sample augmentation method according to an embodiment of the present disclosure;

[0029] Figure 5 is a flowchart of a visual text model training method according to an embodiment of the present disclosure;

[0030] Figure 6 is a structural block diagram of a training sample augmentation device according to an embodiment of the present disclosure;

[0031] Figure 7 is a structural block diagram of a visual text model training device according to an embodiment of the present disclosure;

[0032] Figure 8 It is a schematic diagram of the hardware structure of the computer device of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0034] At present, language models have made significant progress in various natural language processing tasks, which makes visual text models also widely use language models as base models for text output. In terms of structure, the visual text model can include a visual feature extraction module, a visual feature alignment module and a language model. In terms of training mechanism, the training of the visual text model usually includes pre-training and fine-tuning. Among them, pre-training can be performed on samples based on a large amount of "text-image" data, and fine-tuning can be performed based on "instruction-image-output" triple samples. After pre-training, the visual text model can process some instructions. However, due to the high time cost and labor cost of manually annotating training samples, the triple samples used by the visual text model during fine-tuning are limited, which leads to poor robustness of the visual text model. A slight change to the input instruction may give an incorrect answer. Based on this, the present disclosure provides a training sample augmentation method, by amplifying the original instruction in the original triple sample, a plurality of augmented instructions corresponding to the original instruction are obtained, and an augmented triple sample is generated according to the target augmented instruction matching the semantics of the original instruction and the original triple sample. Thus, the automatic expansion of triple samples is achieved, which greatly enriches the content and quantity of triple samples. Furthermore, the training of visual text model based on the large number of triple samples with rich content can improve the robustness of visual text model.

[0035] According to an embodiment of the present disclosure, an embodiment of a training sample augmentation method and a visual text model training method are provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0037] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0038] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0039] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0040] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0041] According to an embodiment of the present disclosure, an embodiment of a training sample augmentation method and a visual text model training method are provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] In this embodiment, a training sample augmentation method is provided, which can be used in the above-mentioned computer devices, such as mobile phones, tablet computers, desktop computers, portable notebooks, servers, etc. Figure 1 is a flowchart of a training sample augmentation method according to an embodiment of the present disclosure. Figure 1 As shown, the process includes the following steps:

[0043] Step S101 , amplify the original instruction in the original triple sample to obtain a plurality of amplified instructions corresponding to the original instruction.

[0044] In some embodiments, the computer device may include a human-computer interaction module, and accordingly, the user may operate the human-computer interaction module to upload or edit at least one original triple sample, and after the upload is successful or the editing is completed, operate the submission control to send a sample amplification request to the computer device. Accordingly, the computer device obtains the sample amplification request in response to the user's submission operation, obtains the original triple sample to be amplified from the sample amplification request, and performs amplification processing on the original instruction in the original triple sample to obtain multiple amplification instructions corresponding to the original instruction.

[0045] In other embodiments, the computer device may be in communication connection with the requesting device. Accordingly, the computer device may receive the sample amplification request sent by the requesting device, obtain the original triple sample to be amplified from the sample amplification request, and amplify the original instruction in the original triple sample to obtain multiple amplification instructions corresponding to the original instruction.

[0046] The original triplet sample can be used to fine-tune the initial visual text model obtained by pre-training. The original triplet sample includes the original instruction, the first image data corresponding to the original instruction, and the output data corresponding to the original instruction. The original instruction can be a question based on the first image data, the first image data can be any image data, and the output data corresponding to the original instruction can be the answer to the question corresponding to the original instruction. For example, the first image data includes two cats, the original instruction can be how many cats are there in the image, and the output data can be two.

[0047] Step S102, determining a target augmented instruction in the augmented instruction that matches the semantics of the original instruction.

[0048] After the original instruction is amplified, the obtained augmented instruction may have a different meaning from the original instruction. In order to enable the visual text model to effectively identify different instructions with the same meaning and output correct output data, in some embodiments of the present disclosure, after obtaining the augmented instruction corresponding to the original instruction, a target augmented instruction in the augmented instruction that matches the semantics of the original instruction is determined.

[0049] Step S103, generating an augmented triplet sample according to the original triplet sample and the target amplification instruction.

[0050] In the training sample augmentation method provided in this embodiment, the original instructions in the original triple sample are augmented to obtain multiple augmented instructions corresponding to the original instructions, and augmented triple samples are generated according to the target augmented instructions that match the semantics of the original instructions and the original triple sample. In this way, the automatic expansion of triple samples is achieved, which greatly enriches the content and quantity of triple samples. Furthermore, the training of the visual text model based on the large number of triple samples with rich content can improve the robustness of the visual text model.

[0051] In this embodiment, a training sample augmentation method is provided, which can be used in the above-mentioned computer devices, such as mobile phones, tablet computers, desktop computers, portable notebooks, servers, etc. Figure 2 is a flowchart of a training sample augmentation method according to an embodiment of the present disclosure. Figure 2 As shown, the process includes the following steps:

[0052] Step S201 , amplify the original instruction in the original triple sample to obtain a plurality of amplified instructions corresponding to the original instruction.

[0053] In some implementations, step S201 may include at least one of the following steps S2011 and S2012:

[0054] Step S2011, transforming the original syntax of the original instruction in the original triple sample to obtain a plurality of augmented instructions corresponding to the original instruction.

[0055] In some implementations, transforming the original syntax of the original instruction in the original triple sample may include at least one of the following operations:

[0056] Operation a1, determining a first target grammar matching the original instruction, and transforming the original grammar of the original instruction into the first target grammar; wherein the difficulty of understanding the first target grammar is greater than the difficulty of understanding the original grammar.

[0057] That is to say, compared with the difficulty of understanding the original instruction, the original instruction is transformed into an augmented instruction that is more difficult to understand. Thus, the pre-trained initial visual text model is fine-tuned using the augmented triple sample including the augmented instruction, so that the initial visual text model can learn the instruction that is more difficult to understand. As an example, the original grammar of the original instruction is "subject + predicate + object", and the first target grammar is "attributive + subject + adverbial + predicate + attributive + object".

[0058] Operation a2, determining a second target grammar that matches the original instruction, and transforming the original grammar of the original instruction into the second target grammar; the difficulty of understanding the second target grammar is lower than the difficulty of understanding the original grammar.

[0059] That is to say, compared with the difficulty of understanding the original instruction, the original instruction is transformed into an augmented instruction that is less difficult to understand. Thus, the pre-trained initial visual text model is fine-tuned using the augmented triple sample including the augmented instruction, so that the initial visual text model can learn the instruction that is less difficult to understand. As an example, the original grammar of the original instruction is "attributive + subject + adverbial + predicate + attributive + object", and the first target grammar is "subject + predicate + object".

[0060] Operation a3, determining a third target syntax matching the original instruction, and transforming the original syntax of the original instruction into the third target syntax; the instruction length corresponding to the third target syntax is greater than the instruction length of the original instruction.

[0061] That is to say, compared with the instruction length of the original instruction, the original instruction is transformed into an expanded instruction with a longer instruction length. Thus, the pre-trained initial visual text model is fine-tuned using the augmented triple sample including the expanded instruction, so that the initial visual text model can learn instructions with longer instruction length. The implementation method of step a3 is similar to the implementation method of step a1. The third target grammar can be determined by adding attributives, adverbials, complements, etc., and the original instruction is transformed into an expanded instruction with a longer instruction length according to the third target grammar.

[0062] Operation a4, determining a fourth target syntax matching the original instruction, and transforming the original syntax of the original instruction into the fourth target syntax; the instruction length corresponding to the fourth target syntax is smaller than the instruction length of the original instruction.

[0063] That is to say, compared with the instruction length of the original instruction, the original instruction is transformed into an augmented instruction with a shorter instruction length. Thus, the pre-trained initial visual text model is fine-tuned using the augmented triple sample including the augmented instruction, so that the initial visual text model can learn instructions with a shorter instruction length. The implementation method of step a4 is similar to the implementation method of step a2. The fourth target grammar can be determined by removing the attributive, adverbial, complement, etc., and the original instruction is transformed into an augmented instruction with a shorter instruction length according to the fourth target grammar.

[0064] Therefore, by transforming the syntax of the original instruction, different augmented instructions are obtained, which not only increases the number of instructions, but also increases the richness of the instruction syntax, thereby increasing the number of triple samples and enriching the content of the triple samples.

[0065] Step S2012, transforming the constituent elements of the original instruction in the original triple sample to obtain a plurality of augmented instructions corresponding to the original instruction.

[0066] In some implementations, transforming the constituent elements of the original instruction in the original triple sample may include at least one of the following operations:

[0067] Operation b1, determining a first element to be replaced among constituent elements of the original instruction and a second element having the same or similar meaning as the first element, and replacing the first element with the second element.

[0068] That is, the second element can be a synonymous element of the first element (such as a synonym, etc.), and by replacing the first element in the original instruction with the second element, an augmented instruction with unchanged instruction meaning but changed constituent elements is obtained. Thus, the pre-trained initial visual text model is fine-tuned using the augmented triplet sample including the augmented instruction, so that the initial visual text model can learn different instructions with the same meaning.

[0069] Operation b2, determining the third element to be replaced in the constituent elements of the original instruction and the fourth element which has a different meaning from the third element but a similar form, and replacing the third element with the fourth element.

[0070] That is, the third element and the fourth element are similar elements with different meanings (such as similar characters and words with different meanings). By replacing the third element in the original instruction with the fourth element, an augmented instruction with a changed meaning but similar form is obtained. Thus, the pre-trained initial visual text model is fine-tuned using the augmented triplet sample including the augmented instruction, so that the initial visual text model can learn similar instructions with different meanings.

[0071] Operation b3, determining the fifth element to be added to the original instruction, and adding the fifth element to the original instruction.

[0072] Among them, the fifth element can be an element related to the content of the original instruction, or an element unrelated to the content of the original instruction. The fifth element can be added to any position in the original instruction, such as to the end of the original instruction. In other words, by adding the fifth element to the original instruction, an augmented instruction with changed content and / or form can be obtained. Thus, by fine-tuning the pre-trained initial visual text model using the augmented triplet sample including the augmented instruction, the initial visual text model can learn different instructions.

[0073] Therefore, by transforming the constituent elements of the original instructions, different augmented instructions are obtained, which not only increases the number of instructions, but also increases the richness of the instruction composition, thereby increasing the number of triple samples and enriching the content of the triple samples.

[0074] In order to improve the efficiency of the augmentation process, in some embodiments, the original instructions may be augmented using a language processing model. That is, step S201 may include: augmenting the original instructions in the original triple sample using a language processing model to obtain multiple augmented instructions corresponding to the original instructions.

[0075] Specifically, an amplification request can be generated based on the original instructions in the original triple and the amplification principle, and the amplification request can be input into a pre-trained language processing model. The language processing model performs amplification processing on the original instructions according to the amplification principle in the amplification request. Among them, the amplification principle can be any one or more of the above-mentioned difficulty of understanding increases, difficulty of understanding decreases, instruction length increases, instruction length decreases, different meanings and similar forms, same or similar meanings and different forms, etc. The network structure and training method of the language processing model can be set according to needs in actual applications, and no specific limitation is made in this disclosure. The original instructions in the original triple sample are amplified by using the language processing model, which greatly improves the efficiency of the amplification processing.

[0076] Step S202 , determining a target augmented instruction in each augmented instruction that matches the semantics of the original instruction.

[0077] Step S203, generating an augmented triplet sample according to the original triplet sample and the target augmentation instruction.

[0078] For the specific implementation of step S202 and step S203, please refer to the relevant description below, and the repeated parts will not be repeated here.

[0079] In the training sample augmentation method provided in this embodiment, the original instruction in the original triple sample is augmented by at least one aspect of the syntax and constituent elements to obtain different augmented instructions, which not only increases the number of instructions, but also increases the richness of instructions. In turn, the number of triple samples is increased and the content of triple samples is enriched.

[0080] In this embodiment, a training sample augmentation method is provided, which can be used in the above-mentioned computer devices, such as mobile phones, tablet computers, desktop computers, portable notebooks, servers, etc. Figure 3 is a flowchart of a training sample augmentation method according to an embodiment of the present disclosure. Figure 3 As shown, the process includes the following steps:

[0081] Step S301, amplify the original instruction in the original triple sample to obtain multiple amplified instructions corresponding to the original instruction.

[0082] The implementation of step S301 can refer to the relevant description above, and the repeated parts will not be repeated here.

[0083] Step S302 , determining a target augmented instruction in the augmented instruction that matches the semantics of the original instruction.

[0084] In some implementations, step S302 may include the following steps S3021 to S3023:

[0085] Step S3021, extracting the first semantic feature of the original instruction.

[0086] In order to improve the accuracy of semantic feature extraction, in some embodiments, step S3021 may include: using a feature extraction model to extract the first semantic feature of the original instruction. Specifically, the original instruction is input into a pre-trained feature extraction model for feature extraction processing to obtain the first semantic feature of the original instruction. The network structure and training method of the feature extraction model can be set as needed in actual applications, and this is not specifically limited in this disclosure.

[0087] Step S3022: extract the second semantic feature of each augmentation instruction.

[0088] In some embodiments, step S3022 may include: using the above-mentioned feature extraction model to extract the second semantic feature of each amplification instruction. Specifically, the amplification instruction is input into a pre-trained feature extraction model for feature extraction processing to obtain the second semantic feature of the amplification instruction. Among them, each amplification instruction can be input into the pre-trained feature extraction model in sequence, or can be input into the pre-trained feature extraction model in batches. The feature extraction model is used to extract the semantic features of the original instruction and each amplification instruction, thereby improving the extraction rate and accuracy of the semantic features.

[0089] Step S3023: Determine, according to the first semantic feature and the second semantic feature, a target augmented instruction in each augmented instruction that matches the semantics of the original instruction.

[0090] Considering that in actual applications, there may be some augmented instructions that are quite different from the original instructions, and the augmented instructions are not very helpful in improving the robustness of the model. Based on this, in some embodiments, the target augmented instructions that match the semantics of the original instructions in each augmented instruction are determined according to the first semantic feature and the second semantic feature. Specifically, step S3023 may include: determining the similarity between the first semantic feature and each second semantic feature; determining the augmented instruction corresponding to the target similarity greater than the similarity threshold as the target augmented instruction that matches the semantics of the original instruction.

[0091] Specifically, for each second semantic feature, the similarity between the second semantic feature and the first semantic feature is calculated to obtain multiple similarities; each similarity is compared with a similarity threshold to obtain a target similarity greater than the similarity threshold; and the amplification instruction corresponding to the target similarity is determined as the target amplification instruction.

[0092] By extracting the first semantic feature of the original instruction and the second semantic feature of the augmented instruction, and determining the target augmented instruction in the augmented instruction that matches the semantics of the original instruction according to the first semantic feature and the second semantic feature, an augmented triple sample is generated according to the target augmented instruction. In this way, the triple sample is effectively expanded while ensuring that the augmented triple has a high quality.

[0093] Step S303: Generate an augmented triplet sample according to the original triplet sample and the target augmentation instruction.

[0094] Specifically, the triplet sample also includes the first image data corresponding to the original instruction and the output data corresponding to the original instruction. Based on this, the above step S303 includes: for each target augmentation instruction, generating an augmented triplet sample according to the target augmentation instruction, the first image data and the output data in the original triplet sample.

[0095] It is understandable that, in practical applications, the original instructions in multiple original triple samples can be amplified to obtain multiple amplified instructions corresponding to each original instruction. Accordingly, in step S303, for each target amplified instruction, the corresponding augmented triple sample can be generated according to the target amplified instruction and the first image data and output data in the original triple sample where the original instruction corresponding to the target amplified instruction is located. The augmented triple sample is generated according to the target amplified instruction and the original triple sample. The automatic expansion of the triple samples is achieved, which greatly enriches the content and quantity of the triple samples. Furthermore, the training of the visual text model based on the large number of triple samples with rich content can improve the robustness of the visual text model.

[0096] Furthermore, in order to make use of more high-quality augmented triple samples for fine-tuning when fine-tuning the initial visual text model, in some embodiments, after generating the augmented triple samples, it can also include: setting the selection probability of each augmented triple sample according to the target similarity corresponding to the augmented triple.

[0097] In some embodiments, the correspondence between the similarity interval and the selection probability can be pre-set. Accordingly, the selection probability of each augmented triple sample is set according to the target similarity corresponding to the augmented triple, which can include: for each target similarity, determining the target similarity interval where the target similarity is located; determining the selection probability corresponding to the target similarity interval as the selection probability of the corresponding augmented triple sample. The higher the target similarity, the greater the selection probability of the corresponding augmented triple; the greater the selection probability, the greater the possibility that the augmented triple is selected for fine-tuning the initial visual text model.

[0098] In the training sample augmentation method provided in this embodiment, the first semantic feature of the original instruction and the second semantic feature of the augmented instruction are extracted, and the target augmented instruction that matches the semantics of the original instruction in the augmented instruction is determined according to the first semantic feature and the second semantic feature; and the augmented triple sample is generated according to the target augmented instruction. Thus, the effective expansion of the triple sample is achieved while ensuring that the augmented triple has a high quality.

[0099] As one or more specific application embodiments of the present disclosure, Figure 4 As shown, for multiple original triple samples, the original instructions in each triple sample can first be amplified through a language processing model to obtain multiple augmented instructions corresponding to each original instruction. Then, the original instructions and augmented instructions are feature extracted through a feature extraction model to obtain corresponding first semantic features and second semantic features; and based on the first semantic features and the second semantic features, a target augmented instruction in the augmented instruction that matches the semantics of the original instruction is determined. Based on the target augmented instruction and the original triple sample, an augmented triple sample is generated. And, a training sample set is constructed based on the original triple sample and the augmented triple sample, and each sample in the training sample set is used to fine-tune the pre-trained initial visual text model to obtain a target visual text model.

[0100] It can be seen that by amplifying the original instructions in the original triple sample, multiple amplified instructions corresponding to the original instructions are obtained, and augmented triple samples are generated according to the target amplified instructions that match the semantics of the original instructions and the original triple sample. In this way, the automatic expansion of triple samples is achieved, which greatly enriches the content and quantity of triple samples. Furthermore, fine-tuning the visual text model based on the large number of triple samples with rich content can improve the robustness of the visual text model.

[0101] In this embodiment, a visual text model training method is provided, which can be used for the above-mentioned computer devices, such as mobile phones, tablet computers, desktop computers, portable notebooks, servers, etc. Figure 5is a flowchart of a visual text model training method according to an embodiment of the present disclosure. Figure 5 As shown, the process includes the following steps:

[0102] Step S501, obtaining a target training sample from a training sample set; the training sample set includes original triple samples and augmented triple samples of the original triple samples.

[0103] Specifically, a target training sample is obtained from a training sample set according to a selection probability corresponding to a target similarity; the target similarity is the similarity between a target augmented instruction in an augmented triplet sample and an original instruction corresponding to the target augmented instruction.

[0104] The augmented triplet samples are obtained according to the training sample augmentation method provided in any of the above embodiments. The greater the selection probability, the greater the possibility that the corresponding augmented triplet is selected for fine-tuning the initial visual text model, thereby ensuring that as many high-quality target training samples as possible are used for fine-tuning.

[0105] In some implementations, the process may further include before step S501:

[0106] The image-text pair is used to iteratively train the training network to obtain an initial visual text model; wherein the image-text pair includes the second image data and the text data, and the text data is used to describe the second image data. The specific training process of the initial visual text model is not specifically limited in this disclosure and can be set as needed in actual applications.

[0107] Step S502, fine-tuning the pre-trained initial visual text model using the target training sample to obtain a target visual text model.

[0108] The processing procedure of the fine-tuning process can be set according to actual needs in actual applications, and is not specifically limited in this disclosure.

[0109] In the visual text model training method provided in this embodiment, a target visual text model is obtained by obtaining a target training sample from a training sample set including original triple samples and augmented triple samples of the original triple samples, and fine-tuning the pre-trained initial visual text model using the target training sample. Since the augmented triple samples in the training sample set are obtained by augmenting the original triple samples, the content and quantity of the triple samples are greatly enriched, and then, fine-tuning the visual text model based on the large number of triple samples with rich content can improve the robustness of the obtained target visual text model.

[0110] In this embodiment, a training sample augmentation device is also provided, which is used to implement the above embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0111] This embodiment provides a training sample augmentation device, such as Figure 6 As shown, including:

[0112] The amplification module 601 is used to amplify the original instruction in the original triple sample to obtain multiple amplified instructions corresponding to the original instruction.

[0113] The determination module 602 is used to determine a target augmented instruction in the augmented instruction that matches the semantics of the original instruction.

[0114] The generating module 603 is used to generate an augmented triplet sample according to the original triplet sample and the target augmentation instruction.

[0115] In some optional embodiments, the amplification module 601 includes at least one of the following:

[0116] The first transformation unit is used to transform the original syntax of the original instruction.

[0117] The second transformation unit is used to transform the constituent elements of the original instruction.

[0118] In some optional implementations, the first transformation unit is configured to perform at least one of the following operations:

[0119] A first target grammar matching the original instruction is determined, and the original grammar of the original instruction is transformed into the first target grammar; the difficulty of understanding the first target grammar is greater than the difficulty of understanding the original grammar.

[0120] A second target grammar matching the original instruction is determined, and the original grammar of the original instruction is transformed into the second target grammar; the difficulty of understanding the second target grammar is less than the difficulty of understanding the original grammar.

[0121] A third target syntax matching the original instruction is determined, and the original syntax of the original instruction is transformed into the third target syntax; the instruction length corresponding to the third target syntax is greater than the instruction length of the original instruction.

[0122] A fourth target syntax matching the original instruction is determined, and the original syntax of the original instruction is transformed into the fourth target syntax; the instruction length corresponding to the fourth target syntax is smaller than the instruction length of the original instruction.

[0123] In some optional implementations, the second transform unit is configured to perform at least one of the following operations:

[0124] A first element to be replaced and a second element having the same or similar meaning as the first element are determined among constituent elements of the original instruction, and the first element is replaced with the second element.

[0125] A third element to be replaced among constituent elements of the original instruction and a fourth element having a different meaning from the third element but a similar form are determined, and the third element is replaced by the fourth element.

[0126] A fifth element to be added to the original instruction is determined, and the fifth element is added to the original instruction.

[0127] In some optional embodiments, the amplification module 601 is specifically used for:

[0128] The original instructions in the original triple sample are amplified using a language processing model to obtain a plurality of amplified instructions corresponding to the original instructions.

[0129] In some optional implementations, the determining module includes:

[0130] The first extraction unit is used to extract the first semantic feature of the original instruction.

[0131] The second extraction unit is used to extract the second semantic feature of each of the amplification instructions.

[0132] A determination unit is used to determine, according to the first semantic feature and the second semantic feature, a target augmented instruction in each of the augmented instructions that matches the semantics of the original instruction.

[0133] In some optional implementations, the determining unit is specifically configured to:

[0134] Determine the similarity between the first semantic feature and each of the second semantic features.

[0135] The augmented instruction corresponding to the target similarity greater than the similarity threshold is determined as the target augmented instruction that matches the semantics of the original instruction.

[0136] In some optional implementations, the first extraction unit is specifically configured to extract the first semantic feature of the original instruction by using a feature extraction model.

[0137] The first extraction unit is specifically configured to extract the second semantic feature of each of the augmentation instructions by using the feature extraction model.

[0138] In some optional implementations, the triplet sample further includes the first image data corresponding to the original instruction and the output data corresponding to the original instruction. Accordingly, the generation module is specifically configured to:

[0139] For each of the target augmentation instructions, an augmented triplet sample is generated according to the target augmentation instruction, the first image data and the output data.

[0140] The training sample augmentation device provided in this embodiment performs augmentation processing on the original instructions in the original triple sample to obtain multiple augmented instructions corresponding to the original instructions, and generates augmented triple samples according to the target augmented instructions that match the semantics of the original instructions and the original triple sample. In this way, the automatic expansion of triple samples is achieved, which greatly enriches the content and quantity of triple samples. Furthermore, the training of the visual text model based on the large number of triple samples with rich content can improve the robustness of the visual text model.

[0141] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0142] The training sample augmentation device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0143] In this embodiment, a visual text model training device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0144] This embodiment provides a visual text model training device, such as Figure 7 As shown, including:

[0145] The acquisition module 701 is used to acquire target training samples from a training sample set; the training sample set includes original triple samples and augmented triple samples of the original triple samples; the augmented triple samples are obtained according to the training sample augmentation method provided in any of the aforementioned embodiments.

[0146] The fine-tuning module 702 is used to fine-tune the pre-trained initial visual text model using the target training sample to obtain a target visual text model.

[0147] In some implementations, the acquisition module 701 is specifically used to:

[0148] According to the selection probability corresponding to the target similarity, a target training sample is obtained from the training sample set; the target similarity is the similarity between the target augmented instruction in the augmented triple sample and the original instruction corresponding to the target augmented instruction.

[0149] The visual text model training device provided in this embodiment obtains a target visual text model by obtaining a target training sample from a training sample set including original triple samples and augmented triple samples of the original triple samples, and fine-tuning the pre-trained initial visual text model using the target training sample. Since the augmented triple samples in the training sample set are obtained by augmenting the original triple samples, the content and quantity of the triple samples are greatly enriched, and then, the visual text model is fine-tuned based on the large number of triple samples with rich content, which can improve the robustness of the obtained target visual text model.

[0150] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0151] The training sample augmentation device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0152] The present disclosure also provides a computer device having the above Figure 6 The training sample augmentation device shown, or Figure 7 The visual text model training device shown.

[0153] See also Figure 8 , Figure 8 is a schematic diagram of a computer device provided by an optional embodiment of the present disclosure, such as Figure 8As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0154] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0155] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0156] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0157] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0158] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0159] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0160] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0161] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0162] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0163] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0164] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0165] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0166] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A training sample augmentation method, characterized in that: The method comprises: Amplifying the original instruction in the original triple sample to obtain a plurality of amplified instructions corresponding to the original instruction; Determine a target augmented instruction in the augmented instruction that matches the semantics of the original instruction; An augmented triplet sample is generated according to the original triplet sample and the target amplification instruction.

2. The method according to claim 1, characterized in that The amplification process of the original instruction in the original triple sample includes at least one of the following: Performing transformation processing on the original syntax of the original instruction; The constituent elements of the original instruction are transformed.

3. The method according to claim 2, characterized in that The transforming of the original syntax of the original instruction includes at least one of the following operations: Determine a first target grammar that matches the original instruction, and transform the original grammar of the original instruction into the first target grammar; the difficulty of understanding the first target grammar is greater than the difficulty of understanding the original grammar; Determine a second target grammar matching the original instruction, and transform the original grammar of the original instruction into the second target grammar; the difficulty of understanding the second target grammar is less than the difficulty of understanding the original grammar; Determine a third target syntax matching the original instruction, and transform the original syntax of the original instruction into the third target syntax; the instruction length corresponding to the third target syntax is greater than the instruction length of the original instruction; A fourth target syntax matching the original instruction is determined, and the original syntax of the original instruction is transformed into the fourth target syntax; the instruction length corresponding to the fourth target syntax is smaller than the instruction length of the original instruction.

4. The method according to claim 2, characterized in that: The transforming of the constituent elements of the original instruction includes at least one of the following operations: Determine a first element to be replaced among constituent elements of the original instruction, and a second element having the same or similar meaning as the first element, and replace the first element with the second element; Determine a third element to be replaced among the constituent elements of the original instruction, and a fourth element having a different meaning from the third element but a similar form, and replace the third element with the fourth element; A fifth element to be added to the original instruction is determined, and the fifth element is added to the original instruction.

5. The method according to any one of claims 1 to 4, characterized in that: The amplifying the original instruction in the original triple sample to obtain a plurality of amplified instructions corresponding to the original instruction includes: The original instructions in the original triple sample are amplified using a language processing model to obtain a plurality of amplified instructions corresponding to the original instructions.

6. The method according to claim 1, characterized in that The determining of a target augmented instruction in the augmented instruction that matches the semantics of the original instruction includes: Extracting a first semantic feature of the original instruction; Extracting a second semantic feature of each of the amplification instructions; According to the first semantic feature and the second semantic feature, a target augmented instruction in each of the augmented instructions that matches the semantics of the original instruction is determined.

7. The method according to claim 6, characterized in that The step of determining, according to the first semantic feature and the second semantic feature, a target augmented instruction in each augmented instruction that matches the semantics of the original instruction includes: Determining a similarity between the first semantic feature and each of the second semantic features; The augmented instruction corresponding to the target similarity greater than the similarity threshold is determined as the target augmented instruction that matches the semantics of the original instruction.

8. The method according to claim 6, characterized in that The extracting the first semantic feature of the original instruction includes: Using a feature extraction model, extracting a first semantic feature of the original instruction; The extracting the second semantic feature of each of the amplification instructions includes: The feature extraction model is used to extract the second semantic feature of each of the augmentation instructions.

9. The method according to claim 1, characterized in that: The triple sample also includes the first image data corresponding to the original instruction and the output data corresponding to the original instruction; and generating the augmented triple sample according to the original triple sample and the target augmentation instruction includes: For each of the target augmentation instructions, an augmented triplet sample is generated according to the target augmentation instruction, the first image data and the output data.

10. A visual text model training method, characterized in that: The method comprises: Acquire a target training sample from a training sample set; the training sample set includes an original triple sample and an augmented triple sample of the original triple sample; the augmented triple sample is obtained according to the training sample augmentation method according to any one of claims 1 to 9; The target training sample is used to fine-tune the pre-trained initial visual text model to obtain a target visual text model.

11. The method according to claim 10, characterized in that The step of obtaining a target training sample from a training sample set includes: According to the selection probability corresponding to the target similarity, a target training sample is obtained from the training sample set; the target similarity is the similarity between the target augmented instruction in the augmented triple sample and the original instruction corresponding to the target augmented instruction.

12. A training sample augmentation device, characterized in that: The device comprises: An amplification module, used to amplify the original instruction in the original triple sample to obtain a plurality of amplified instructions corresponding to the original instruction; A determination module, used for determining a target augmented instruction in the augmented instruction that matches the semantics of the original instruction; A generating module is used to generate an augmented triplet sample according to the original triplet sample and the target amplification instruction.

13. A visual text model training device, characterized in that: The device comprises: An acquisition module, used for acquiring a target training sample from a training sample set; the training sample set includes an original triple sample and an augmented triple sample of the original triple sample; the augmented triple sample is obtained according to the training sample augmentation method according to any one of claims 1 to 9; The fine-tuning module is used to use the target training sample to fine-tune the pre-trained initial visual text model to obtain a target visual text model.

14. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the training sample augmentation method described in any one of claims 1 to 9, or executes the visual text model training method described in any one of claims 10 to 11 by executing the computer instructions.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the training sample augmentation method described in any one of claims 1 to 9, or to execute the visual text model training method described in any one of claims 10 to 11.

16. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to enable a computer to execute the training sample augmentation method described in any one of claims 1 to 9, or to execute the visual text model training method described in any one of claims 10 to 11.