A processing method and device of a visual language training model and an electronic device

Abstract

The present disclosure provides a processing method and device of a visual language training model and an electronic device. The method comprises: obtaining a training sample, wherein the training sample comprises a training image and a training text used to describe the training image; splitting the training text in units of phrases, and selecting a covered phrase from the split training text; selecting a first region covered in the training image and a second region adjacent to the first region; and training the visual language training model according to the covered phrase, the covered training text, the training image, the first region and the second region.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of model processing, and more particularly, to a processing method of a visual language training model, a processing device of a visual language training model, an electronic device, and a computer-readable storage medium. BACKGROUND

[0002] After a pre-training model is trained on a large-scale dataset, a downstream task can achieve good results with only a small amount of labeled data or even without any labeled data. At present, both academia and industry pay high attention to pre-training models.

[0003] Processing the text part on the visual language training model will refer to the bert model to randomly cover a certain word, and the image part will cover a certain area. An image caption data set is used.

[0004] Image caption is a relatively comprehensive task combining the fields of CV (Computer Vision) and NLP (Neuro-Linguistic Programming). The input is an image, and the output is a piece of text describing the image. This task requires the model to identify objects in the picture, understand the relationship between objects, and express it in a natural language.

[0005] A certain image area may correspond to multiple words, but in the process of covering the words in the text, some strongly related words may be split. Training the visual language training model on the text with some strongly related words split and covered may result in poor performance of the visual language training model.

[0006] In the process of training the visual language training model, other regions in the image may interfere with the covered region, which may also result in poor performance of the visual language training model. SUMMARY

[0007] An object of the present disclosure is to provide a new technical solution for processing a visual language training model.

[0008] According to a first aspect of the present disclosure, a processing method of a visual language training model is provided, comprising:

[0009] obtaining a training sample, wherein the training sample comprises a training image and training text for describing the training image;

[0010] split the training text in units of phrases, and select a covered phrase from the split training text; select a first region covered in the training image, and a second region adjacent to the first region;

[0011] train the visual language training model according to the covered phrase, the covered training text, the training image, the first region and the second region.

[0012] Optionally, the splitting the training text in units of phrases comprises:

[0013] construct a phrase structure tree of the training text; wherein the phrase structure tree is used to describe the syntax structure of the training text;

[0014] split the training text according to the phrase structure tree.

[0015] Optionally, the selecting a covered phrase from the split training text comprises:

[0016] traverse the split training text in units of phrases to generate a random number corresponding to each phrase;

[0017] select a phrase with a random number in a set range as the covered phrase.

[0018] Optionally, the training the visual language training model according to the covered phrase, the covered training text, the training image, the first region and the second region comprises:

[0019] perform a first training task of the visual language training model according to the covered phrase, the covered training text and the training image;

[0020] perform a second training task of the visual language training model according to the training image, the first region and the second region.

[0021] Optionally, the performing the first training task of the visual language training model according to the covered phrase, the covered training text and the training image comprises:

[0022] obtain a word order of each phrase in the training text and a preset covering vector used to represent the covered phrase;

[0023] encode the word order of each phrase, the uncovered phrase and the training image according to a preset neural network to obtain a word order vector, an uncovered phrase vector and a training image vector respectively;

[0024] Taking the undetermined parameter of the visual language training model as a variable, an accuracy rate of predicting each covered phrase according to the covering vector, the syntax vector, the uncovered phrase vector and the training image vector is obtained;

[0025] According to the accuracy rate of the covered phrase, the undetermined parameter is solved.

[0026] Optionally, the second training task of the visual language training model according to the training image, the first region and the second region comprises:

[0027] According to a preset neural network, the training image, the first region and the second region are respectively encoded to obtain a training image vector, a first region vector and a second region vector;

[0028] Based on the value of the undetermined parameter of the visual language training model obtained through the first training task, the undetermined parameter is taken as a variable, and the training image vector and the second region vector are used to predict a predicted region vector of the first region;

[0029] According to the predicted region vector and the first region vector, the undetermined parameter is solved.

[0030] Optionally, the solving of the undetermined parameter according to the predicted region vector and the first region vector comprises:

[0031] The distance between the predicted region vector and the first region vector is calculated;

[0032] The value of the undetermined parameter at the minimum distance is solved.

[0033] Optionally, the solving of the undetermined parameter according to the predicted region vector and the first region vector comprises:

[0034] The value of the undetermined parameter when the predicted region vector and the first region vector belong to a preset same category is solved.

[0035] Optionally, the method further comprises:

[0036] A setting entry for setting an application scenario of the visual language training model is provided;

[0037] An application scenario input through the setting entry is obtained;

[0038] The step of obtaining a training sample comprises: obtaining a corresponding training sample according to the input application scenario.

[0039] Optionally, after the training of the visual language training model, the method further comprises:

[0040] determine an application scenario to which the visual language training model is applicable;

[0041] find an application project matched with the application scenario;

[0042] input the visual language training model into the application project.

[0043] Optionally, after the visual language training model is trained, the method further includes:

[0044] obtain to-be-processed image-text information; wherein, the image-text information includes an image and / or text;

[0045] process the to-be-processed image-text information by using the visual language training model to obtain a corresponding processing result.

[0046] According to a second aspect of the present disclosure, a processing apparatus of a visual language training model is provided, including:

[0047] a sample obtaining module, configured to obtain a training sample, wherein the training sample includes a training image and a training text used to describe the training image;

[0048] a text covering module, configured to split the training text in units of phrases, and select a covered phrase from the split training text;

[0049] an image covering module, configured to select a first region to be covered in the training image, and a second region adjacent to the first region;

[0050] a model training module, configured to train a visual language training model according to the covered phrase, the covered training text, the training image, the first region and the second region.

[0051] According to a third aspect of the present disclosure, an electronic device is provided, including:

[0052] a processor and a memory, the memory being configured to store instructions for controlling the processor to execute the method according to the first aspect of the present disclosure.

[0053] According to a fourth aspect of the present disclosure, a computer readable storage medium is provided, which stores a computer program, the computer program being configured to implement the method according to the first aspect of the present disclosure when executed by a processor.

[0054] In the embodiments of the present specification, by splitting the training text into phrases for masking, some strongly related words can be avoided from being split, and the phrases and image regions can be better aligned. By selecting a second region adjacent to the first region to be masked, the difficulty of the model in predicting the region category can be reduced, and the relationship between the first region and the second region can be better established. In this way, the effect of the visual language training model can be further improved.

[0055] Other features of the present disclosure, and their advantages, will become apparent from the following detailed description of exemplary embodiments of the present disclosure, with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0056] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0057] Figure 1a is a schematic diagram of a hardware structure of an electronic device according to a first embodiment of the present disclosure.

[0058] Figure 1b is a schematic diagram of a hardware structure of an electronic device according to a second embodiment of the present disclosure.

[0059] Figure 2 is a schematic diagram of an application scenario of a visual language training model according to an embodiment of the present specification.

[0060] Figure 3 is a flowchart showing a processing method of a visual language training model according to an embodiment of the present disclosure.

[0061] Figure 4 is a schematic diagram showing an example of a phrase structure tree according to an embodiment of the present disclosure.

[0062] Figure 5 is a schematic diagram showing a principle block diagram of a processing device of a visual language training model according to an embodiment of the present disclosure.

[0063] Figure 6 is a principle block diagram of an electronic device according to a third embodiment of the present disclosure. DETAILED DESCRIPTION

[0064] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. Note that the relative arrangement, numerical expressions, and numerical values of components and steps set forth in these embodiments are not limiting to the scope of the present disclosure unless otherwise specifically stated.

[0065] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way limiting to the scope of the present disclosure and its applications or uses.

[0066] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail herein. However, where appropriate, such techniques, methods, and devices can be considered part of the present disclosure.

[0067] In all of the compositions and methods disclosed and discussed herein, any particular value should be construed as merely exemplary, and not as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0068] It should be noted that like reference numerals and letters in the various figures indicate similar items, and thus, once any particular item is defined in one figure, it is not necessarily discussed further in subsequent figures.

[0069] <Hardware Configuration>

[0070] Figure 1a and Figure 1b is a block diagram of a hardware configuration of an electronic device 1000 that can be used to implement a method of any embodiment of the present disclosure.

[0071] In one embodiment, as shown in Figure 1a , the electronic device 1000 can be a server 1100.

[0072] The server 1100 is a computer that provides processing, database, communications facilities, etc. The server 1100 can be a monolithic server or a distributed server across multiple computers or computer data centers. The server can be various types, such as, but not limited to, a web server, a news server, a mail server, a messaging server, an advertisement server, a file server, an application server, an interaction server, a database server, or a proxy server. In some embodiments, each server can include hardware, software, or embedded logic components or a combination of two or more such components for performing suitable functions supported or implemented by the server. For example, a server such as a blade server, a cloud server, etc., or a server group composed of multiple servers, can include one or more of the above types of servers, etc.

[0073] In this embodiment, the server 1100 can include a processor 1110, a memory 1120, an interface device 1130, a communication device 1140, a display device 1150, and an input device 1160, as shown in Figure 1a .

[0074] In this embodiment, the server 1100 can also include a speaker, a microphone, etc., which are not limited herein.

[0075] The processor 1110 can be a dedicated server processor, or a desktop processor, a mobile processor, etc. that meets performance requirements, which is not limited herein. The memory 1120 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a nonvolatile memory such as a hard disk, etc. The interface device 1130 includes, for example, various bus interfaces, such as a serial bus interface (including a USB interface), a parallel bus interface, etc. The communication device 1140 is capable of wired or wireless communication, for example. The display device 1150 is, for example, a liquid crystal display screen, an LED display screen, a touch display screen, etc. The input device 1160 can include, for example, a touch screen, a keyboard, etc.

[0076] In this embodiment, the memory 1120 of the server 1100 is configured to store instructions for controlling the processor 1110 to operate to perform at least the method according to any embodiment of the present disclosure. The skilled person can design the instructions according to the disclosed solutions of the present disclosure. How the instructions control the processor to operate is well known in the art, and thus is not described in detail herein.

[0077] Although a plurality of devices of the server 1100 are shown in Figure 1a , the present disclosure can only involve part of the devices, for example, the server 1100 only involves the memory 1120 and the processor 1110.

[0078] In one embodiment, the electronic device 1000 can be a terminal device 1200 such as a PC, a notebook computer, etc. used by an operator, which is not limited herein.

[0079] In this embodiment, referring to Figure 1b , the terminal device 1200 can include a processor 1210, a memory 1220, an interface device 1230, a communication device 1240, a display device 1250, an input device 1260, a loudspeaker 1270, a microphone 1280, etc.

[0080] The processor 1210 can be a mobile version processor. The memory 1220 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a nonvolatile memory such as a hard disk, and the like. The interface device 1230 includes, for example, a USB interface, a headphone interface, and the like. The communication device 1240 is capable of wired or wireless communication, for example, and can include a short-range communication device, such as any device that performs short-range wireless communication based on a Hilink protocol, a WiFi (IEEE 802.11 protocol), a Mesh, a Bluetooth, a ZigBee, a Thread, a Z-Wave, an NFC, a UWB, a LiFi, and the like, and can include a long-range communication device, such as any device that performs WLAN, GPRS, 2G / 3G / 4G / 5G long-range communication. The display device 1250 is, for example, a liquid crystal display, a touch display, and the like. The input device 1260 can include, for example, a touch screen, a keyboard, and the like. A user can input / output voice information through the speaker 1270 and the microphone 1280.

[0081] In this embodiment, the memory 1220 of the terminal device 1200 is configured to store instructions for controlling the processor 1210 to operate to perform at least the method according to any embodiment of the present disclosure. The skilled person can design the instructions according to the disclosed solutions of the present disclosure. How the instructions control the processor to operate is well known in the art, and thus will not be described in detail here.

[0082] Although a plurality of devices of the terminal device 1200 are shown in Figure 1b , the present disclosure can only involve part of the devices, for example, the terminal device 1200 only involves the memory 1220, the processor 1210, and the display device 1250.

[0083] <application scenarios>

[0084] Figure 2 An application scenario diagram of the visual language training model according to the embodiments of the present disclosure.

[0085] Specifically, as shown in Figure 2 , the visual language training model can process the input image and the question text to obtain the corresponding reply text. The visual language training model can process the input image to obtain the text for describing the image. The visual language training model can process the input text to obtain the image described by the text.

[0086] The visual language training model obtained by the embodiments of the present specification can be used to process image-text information (including images and / or texts), and can be specifically used in VQA (visual question answering), image-to-text search, text-to-image search, and the like. Further, the visual language training model can also be used in the e-commerce field, and can be specifically applied to the scenarios of selling point generation, search recommendation, title generation, and the like.

[0087] In the selling point generation scenario, the visual language training model can process an input image to obtain a text used to describe the image, and the text can reflect the selling point of a product contained in the input image.

[0088] In the search recommendation scenario, the visual language training model can process an input text to obtain an image described by the text, and then recommend a product contained in the image to a user.

[0089] In the title generation scenario, the visual language pre-training model can process an input image to obtain a text used to describe the image, as a title of the image, and the text can reflect the theme of the input image.

[0090] Further, the visual language training model obtained by the embodiments of the present specification can also be used in the e-commerce field. Specifically, it can be used in the scenarios of selling point generation, search recommendation, title generation, and recommendation question answering in the e-commerce field.

[0091] In the selling point generation scenario in the e-commerce field, the visual language training model can process an input image containing a product to obtain a text used to describe the selling point of the product.

[0092] In the search recommendation scenario in the e-commerce field, the visual language training model can process an input text describing a product to obtain an image described by the text, and then recommend a product contained in the image to a user.

[0093] In the title generation scenario in the e-commerce field, the visual language pre-training model can process an input image containing a product to obtain a text used to describe the product contained in the image, as a title of a corresponding link of the product.

[0094] In the recommendation question answering scenario in the e-commerce field, the visual language pre-training model can process an input image containing a product input by a user to obtain a text used to describe the product contained in the image, and feed back the text to the user; or the visual language pre-training model can process a text describing a product input by a user to obtain an image described by the text, and feed back the image to the user, to realize a dialogue with the user.

[0095] In the process of training the visual language training model, the embodiment of the present specification can avoid splitting some strongly related words by splitting the training text into phrases for covering, and can better align the phrases and image regions. By selecting a second region adjacent to the first region covered, the difficulty of the model in predicting the region category can be reduced, and the relationship between the first region and the second region can be better established. In this way, the effect of the visual language training model can be further improved.

[0096] <Method embodiment>

[0097] In the present embodiment, a processing method of a visual language training model is provided. The method can be implemented by an electronic device. The electronic device can be a server 1100 as shown in Figure 1a , or a terminal device 1200 as shown in Figure 1b .

[0098] According to Figure 3 , the processing method of the visual language training model of the present embodiment can include the following steps S3100-S3400:

[0099] Step S3100, obtaining a training sample, wherein the training sample includes a training image and a training text used to describe the training image.

[0100] In the embodiment of the present specification, the number of training samples used to train the visual language training model can be one or more, and the embodiments of the present disclosure do not limit this. The following will only be described by taking training the visual language training model according to one training sample as an example.

[0101] Each training sample in the present embodiment includes a training image and a training text used to describe the training image. Specifically, one training sample can include one training image and one corresponding training text.

[0102] Step S3200, splitting the training text in units of phrases, and selecting a covered phrase from the split training text.

[0103] In an embodiment of the present specification, a phrase dictionary can be set in advance according to application scenarios or specific needs, wherein a large number of phrases are recorded in the phrase dictionary. Based on the phrase dictionary, the training text can be split in units of phrases.

[0104] In another embodiment of the present specification, splitting the training text in units of phrases can include the following steps S3210-S3220:

[0105] Step S3210, constructing a phrase structure tree of the training text.

[0106] In this embodiment, a constituent tree can be used to describe the syntactic structure of the training text. The constituent tree has only leaf nodes associated with the words in the input sentence, and other internal nodes are labeled phrase constituents. Nouns used to label phrase constituents can include ROOT, IP, NP, VP, PU, LCP, PP, CP, DNP, ADVP, ADJP, DP, QP, NN, NR, NT, PN, VV, VC, CC, VE, VA, AS, VRD, CD, DT, EX, FW, IN, JJ, JJR, JJS, LS, MD, PDT, POS, PRP, RB, RBR, RBS, RP, SYM, TO, WDT, WP, WP$, WRB, etc.

[0107] wherein, ROOT represents a sentence to be processed; IP represents a simple sentence; NP represents a noun phrase; VP represents a verb phrase; PU represents a punctuation, usually a period, a question mark, an exclamation mark, and the like; LCP represents a locative phrase; PP represents a prepositional phrase; CP represents a phrase composed of's' indicating a modifying relationship; DNP represents a phrase composed of's' indicating a belonging relationship; ADVP represents an adverbial phrase; ADJP represents an adjective phrase; DP represents a determiner phrase; QP represents a quantifier phrase; NN represents a common noun; NR represents a proper noun; NT represents a time noun; PN represents a pronoun; VV represents a verb; VC represents is; CC represents a conjunction; VE represents has; VA represents a predicate adjective; AS represents a content marker (e.g., indicates that); VRD represents a verb-adverb compound; CD represents a cardinal number; DT represents a determiner; EX represents an existential there; FW represents a foreign word; IN represents a preposition or a subordinating conjunction; JJ represents an adjective or an ordinal numeral; JJR represents an adjective, comparative; JJS represents an adjective, superlative; LS represents a list item marker; MD represents a modal auxiliary; PDT represents a pre-determiner; POS represents a genitive marker; PRP represents a personal pronoun; RB represents an adverb; RBR represents an adverb, comparative; RBS represents an adverb, superlative; RP represents a particle; SYM represents a symbol; TO represents "to" as a preposition or an infinitive marker; WDT represents a WH-determiner; WP represents a WH-pronoun; WP$ represents a WH-pronoun, possessive; and WRB represents a Wh-adverb.

[0108] Taking the training text "The girl took apples from a bag" as an example, the phrase structure tree of this training text could be as follows: Figure 4 As shown.

[0109] Step S3220: Split the training text according to the phrase structure tree.

[0110] In this embodiment, all words corresponding to the NP node can be merged into a single phrase, and the words corresponding to other bottom-level nodes can be combined into a single phrase.

[0111] In such Figure 4 In the illustrated embodiment, the training text "The girl took apples from a bag" includes 7 words. This embodiment combines the words "The" and "girl" corresponding to the NP node into a single phrase "The girl," and the words "a" and "bag" corresponding to the NP node into a single phrase "a bag." The words "took," "apples," and "from" corresponding to the other bottom-level nodes are also treated as phrases. Therefore, the resulting phrases can include "The girl," "took," "apples," "from," and "a bag." This reduces the number of phrases in the training text to 5, effectively reducing the sentence length from 7 to 5.

[0112] In this embodiment, the sentence length of the training text can be reduced by merging all the words corresponding to the NP node into a single phrase.

[0113] In this embodiment, the training text is split into phrases, and the split training text may include multiple phrases.

[0114] In embodiments of this specification, the step of selecting masked phrases from the segmented training text may include steps S3230 to S3240 as shown below:

[0115] Step S3230: Iterate through the split training text in terms of phrases and generate a random number corresponding to each phrase.

[0116] In this embodiment, the range of random numbers can be preset according to the application scenario or specific needs. For example, the range of random numbers can be [0,1] or [0,100].

[0117] In one embodiment of this specification, the method may further include: providing a range input box for a value range; obtaining a value range through the range input box to generate a random number within the value range.

[0118] In one example, the range input box can include an input box for inputting a complete value range, and in the case that the value range is [0, 1], the user can input [0, 1] through the range input box.

[0119] In another example, the range input box can further include a first input box for inputting a minimum value of the value range and a second input box for inputting a maximum value of the value range. In the case that the value range is [0, 1], the user can input 0 through the first input box and input 1 through the second input box.

[0120] In step S3240, a phrase with a random number in a set range is selected as the covered phrase.

[0121] The set range can be set in advance according to application scenarios or specific requirements, and the set range can be smaller than the value range of the random number. For example, the set range can be [0, 0.15]. Then, the phrase with a random number less than or equal to 0.15 can be selected from the phrases split from the training text as the covered phrase.

[0122] For the training text in any training sample, there can be zero covered phrase or one or more covered phrases.

[0123] In step S3300, a first region to be covered is selected in the training image, and a second region adjacent to the first region is selected.

[0124] In the embodiment, the second region adjacent to the first region includes at least one of the following cases: the second region is completely non-intersected with the first region, but no other region is included between the first region and the second region; and the first region and the second region at least partially intersect. In the case that the first region and the second region at least partially intersect, the first region can be completely included in the second region, the second region can be completely included in the first region, a part of the boundary of the first region and the second region can coincide, or the first region and the second region can partially intersect.

[0125] In one embodiment of the present disclosure, the training image can be split according to a preset splitting rule in advance to obtain a plurality of regions, and the first region to be covered and the second region adjacent to the first region are selected from the plurality of regions. The splitting rule can be set in advance according to application scenarios or specific requirements, for example, the training image can be split according to a set number or a set size.

[0126] In another embodiment of the present disclosure, the training image can be subjected to target detection to obtain at least one target region, and the first region to be covered and the second region adjacent to the first region are selected from the at least one target region.

[0127] The target region is a minimum region completely containing the corresponding target.

[0128] In this embodiment, the target detection can be performed on the training image to obtain the corresponding target region. The target to be detected can include a person, a cat, a dog, a football, a mountain, a cup, a shirt, a shoe, a building, a door, a bag, a street, water, a flower, grass, snow, rain, a car, and the like.

[0129] In one embodiment of the present disclosure, the category of the target to be detected in the training image can be set in advance according to the application scenario or specific requirements. Specifically, a category input box for setting the category of the target to be detected can be provided, and the category of the target to be detected is obtained through the category input box, so as to perform target detection on the training image according to the category to obtain at least one target region.

[0130] If a training image includes multiple targets of the same category, target detection can be performed on the training image to obtain the target region of each target of the category. For example, in the case where the training image includes 5 persons and 1 football, the target region of each person and the target region of the football can be obtained, that is, 6 target regions are obtained.

[0131] In one embodiment of the present disclosure, the manner of selecting the first region to be covered from the at least one target region can be similar to the manner of selecting the covered phrase from the at least one phrase, including: traversing the at least one target region, generating another random number corresponding to each target region; selecting the target region in which the other random number is within another set range as the first region to be covered. In this embodiment, the setting manner of the value range of the other random number can refer to the setting manner of the value range of the random number of the phrase in the foregoing embodiment, which will not be described here.

[0132] The other set range in this embodiment can be set in advance according to the value range of the other random number, and the other set range is smaller than the value range of the other random number. For example, in the case where the value range of the other random number is [0, 1], the other set range can be [0.85, 1].

[0133] In one embodiment of the present disclosure, the first region to be covered can include zero target regions, or one or more target regions.

[0134] In this embodiment, the second region at least partially intersecting with the first region to be covered has at least partially overlapping regions with the at least one first region, that is, each second region at least partially intersects with the at least one first region.

[0135] At step S3400, the visual language training model is trained according to the masked phrases, the masked training texts, the training images, the first regions, and the second regions.

[0136] In an embodiment, the visual language training model can also be a visual language pre-training model.

[0137] The masked training texts are training texts after the masked phrases are masked. The masked training texts include at least one unmasked phrase.

[0138] In an embodiment of the present disclosure, the step of training the visual language training model according to the masked phrases, the masked training texts, the training images, the first regions, and the second regions can include steps S3410-S3420 as shown below:

[0139] At step S3410, a first training task of the visual language training model is performed according to the masked phrases, the masked training texts, and the training images.

[0140] In the present embodiment, the first training task can be a masked language modeling with visual clues (MLM) task.

[0141] Specifically, the first training task of the visual language training model can include steps S3411-S3414 as shown below:

[0142] At step S3411, the order of each phrase in the training texts and a preset masking vector for representing the masked phrases are obtained.

[0143] In the present embodiment, the phrases include the masked phrases and the unmasked phrases in the training texts. The order of each phrase in the training texts can be represented by a number, and the smaller the order of the phrase in the training texts, the smaller the number representing the order.

[0144] In an embodiment of the present disclosure, the masked phrases can be represented by the masking vectors. In the case where a masked phrase includes one word, one masking vector can be used to represent the masked phrase; in the case where a masked phrase includes N (where N is an integer greater than 1) words, N masking vectors can be used to represent each word in the masked phrase as the masking vector of the masked phrase.

[0145] In the example of the training text being "The girl took apples from a bag", if the phrase "the girl" is covered, the sentence after covering can be "[MASK] took apples from a bag". In this embodiment, the order of the covered phrase "The girl" in the training text can be 2, and the orders of the uncovered phrases "took", "apples", "from", and "a bag" in the training text can be 3-6 in sequence.

[0146] In this embodiment, in the case that the training text includes M phrases, the orders of the M phrases can be 2-M+1 respectively. On this basis, the order of [CLS] is usually set to 1 in advance, and the order of [SEP] is set to M+1+1, which are respectively used to represent the beginning and end of the training text.

[0147] In another embodiment of the present disclosure, the covered phrase can be represented by a covering vector. In the case that the covered phrase includes one word, one covering vector can be used to represent the covered phrase; in the case that the covered phrase includes N (where N is an integer greater than 1) words, N covering vectors can be used to represent each word in the covered phrase respectively as the covering vector of the covered phrase.

[0148] In the example of the training text being "The girl took apples from a bag", if the phrase "the girl" is covered, the sentence after covering can be "[MASK] took apples from a bag". In this embodiment, the order of the covered phrase in the training text can include the orders of the words "The" and "girl" in the covered phrase in the training text, which can be 2 and 3 respectively, and the order of the uncovered phrase "took" in the training text can be 4 in sequence, the order of the uncovered phrase "apples" in the training text can be 5 in sequence, the order of the uncovered phrase "from" in the training text can be 6 in sequence, and the orders of the words "a" and "bag" in the uncovered phrase "a bag" in the training text can be 7 and 8 respectively, then the order of the uncovered phrase "a bag" in the training text can be 7 and 8 respectively.

[0149] In this embodiment, in the case that the training text includes M words, the orders of the M words can be 2-M+1 respectively. On this basis, the order of [CLS] is usually set to 1 in advance, and the order of [SEP] is set to M+1+1, which are respectively used to represent the beginning and end of the training text.

[0150] Step S3412, encode the syntax, the uncovered phrase and the training image of each phrase according to the preset neural network, to obtain the syntax vector, the uncovered phrase vector and the training image vector respectively.

[0151] The neural network in this embodiment can be a way of converting discrete variables into continuous vectors, which is neural network Embedding.

[0152] Specifically, the syntax vector corresponding to each syntax can be obtained by encoding each syntax, the uncovered phrase vector corresponding to each uncovered phrase can be obtained by encoding each uncovered phrase, and the training image vector can be obtained by encoding the training image.

[0153] Step S3413, taking the undetermined parameters of the visual language training model as variables, obtaining the accuracy rate of each covered phrase according to the covering vector, the syntax vector, the uncovered phrase vector and the training image vector.

[0154] Specifically, the accuracy rate of the covered phrase can be the probability that the covered phrase is predicted correctly.

[0155] For example, the covered phrase of the training text of the xth training sample can include N (where N is a positive integer) words, and the accuracy rate of the ith (where i is a positive integer less than or equal to N) word in the covered phrase of the training text of the xth training sample can be represented as p(x i , θ). Where θ can be an undetermined parameter. In this embodiment, the initial value of the undetermined parameter θ can be randomly set when performing the training task.

[0156] Step S3414, solving the undetermined parameter according to the accuracy rate of the covered phrase.

[0157] In the case where the training sample includes one, the formula for solving the undetermined parameter according to the accuracy rate of the covered phrase can be:

[0158]

[0159] The training target of this embodiment is to solve the value of the undetermined parameter θ when L1 x is the minimum.

[0160] In the case where the training sample includes multiple, the formula for solving the undetermined parameter according to the accuracy rate of the covered phrase can be:

[0161] L1=E x∈D (L1 x )

[0162]

[0163] wherein E denotes expectation, D denotes a set of training samples, x i represents the i-th word in the masked phrase of the training text of the x-th training sample, p(x i , θ) represents the accuracy of the i-th word in the masked phrase of the training text of the x-th training sample.

[0164] The training target of the embodiment is to solve the value of the undetermined parameter θ when L1 is minimum.

[0165] Step S3420, performing a second training task of the visual language training model according to the training image, the first region and the second region.

[0166] In the embodiment, the second training task can be Masked Region Classification with Linguistic Clues (MRC) using text information.

[0167] In one embodiment of the present specification, performing a second training task of the visual language training model according to the training image, the first region and the second region can include steps S3421-S3423 as shown below:

[0168] Step S3421, encoding the training image, the first region and the second region respectively according to a preset neural network to obtain a training image vector, a first region vector and a second region vector.

[0169] The neural network in the embodiment can be a neural network Embedding which converts discrete variables into continuous vectors.

[0170] Specifically, the training image can be encoded to obtain a training image vector, the first region can be encoded to obtain a first region vector, and the second region can be encoded to obtain a second region vector.

[0171] Step S3422, predicting a predicted region vector of the first region based on the value of the undetermined parameter obtained through the first training task and taking the undetermined parameter as a variable according to the training image vector and the second region vector.

[0172] The predicted region vector of the first region can be represented as y(x’ j , θ), wherein x’ jwherein x' represents the x'th training sample, j represents the jth first region in the training image of the x'th training sample, and represents the undetermined parameter. In the embodiment, the value of the undetermined parameter obtained by the first training task can be used as the initial value of the undetermined parameter in the second training task. That is, the second training task is performed based on the undetermined parameter obtained by the first training task to further correct the undetermined parameter.

[0173] In step S3423, the undetermined parameter is solved according to the prediction region vector and the first region vector.

[0174] In one embodiment of the present specification, solving the undetermined parameter according to the prediction region vector and the first region vector can include: calculating the distance between the prediction region vector and the corresponding first region vector; and solving the value of the undetermined parameter when the distance is the smallest.

[0175] In the case that the training sample for performing the training of the visual language training model is one, and the training image includes at least one covered first region, the average value of the distance between the prediction region vector and the first region vector of each first region can be determined, and the value of the undetermined parameter when the average value of the distance is the smallest can be solved.

[0176] In the case that the training sample for performing the training of the visual language training model is multiple, and the training image includes at least one covered first region, for each training sample, the average value of the distance between the prediction region vector and the first region vector of each first region can be determined, the expectation of the average value of the distance of each training sample can be determined, and the value of the undetermined parameter when the expectation is the smallest can be solved.

[0177] In another embodiment of the present specification, solving the undetermined parameter according to the prediction region vector and the first region vector can include: solving the value of the undetermined parameter when the prediction region vector and the first region vector belong to the same preset category.

[0178] In the case that the training sample for performing the training of the visual language training model is one, and the training image includes at least one covered first region, the value of the undetermined parameter when the prediction region vector and the first region vector of each first region belong to the same preset category can be solved.

[0179] In the case that the training sample for performing the training of the visual language training model is multiple, and the training image includes at least one covered first region, the value of the undetermined parameter when the prediction region vector and the first region vector of the first region belong to the same preset category in more than a set percentage of training samples can be solved. The set percentage can be set in advance according to the application scenario or specific requirements. For example, the set percentage can be 90%.

[0180] In the embodiments of the present specification, by splitting the training text into phrases for masking, some strongly related words can be avoided from being split, and the phrases and image regions can be better aligned. By selecting a second region adjacent to the first region to be masked, the difficulty of the model in predicting the region category can be reduced, and the relationship between the first region and the second region can be better established. In this way, the effect of the visual language training model can be further improved.

[0181] In an embodiment of the present specification, the method can further include:

[0182] A setting entry for setting an application scenario of the visual language training model is provided, and the application scenario input through the setting entry is obtained. The step of obtaining the training sample in step S3100 can include: obtaining the corresponding training sample according to the input application scenario.

[0183] The application scenario in the embodiment can include VQA (visual question answering), image-to-text search, text-to-image search, and the like.

[0184] In an embodiment of the present specification, after performing the aforementioned step S3400, the method can further include: determining an application scenario to which the visual language training model is applicable; finding an application item matching the application scenario; and inputting the visual language training model to the application item.

[0185] The application scenario in the embodiment can include VQA (visual question answering), image-to-text search, text-to-image search, and the like.

[0186] In an embodiment of the present specification, after performing the aforementioned step S3400, the method can further include:

[0187] Obtaining image-text information to be processed; and processing the image-text information to be processed by using the visual language training model to obtain a corresponding processing result. The image-text information can include an image and / or text.

[0188] Specifically, in the case where the image-text information includes an image and question text, the visual language training model can process the input image and question text to obtain corresponding answer text. In the case where the image-text information includes an image, the visual language training model can process the input image to obtain text for describing the image. In the case where the image-text information includes text, the visual language training model can process the input text to obtain an image described by the text.

[0189] In an embodiment of the present specification, based on the processing result of the image-text information, the method can further include: displaying the processing result of the image-text information.

[0190] Further, in the case that the processing result of the graphic-text information includes text, the method can further include: playing the text content in the processing result.

[0191] <Device Embodiment>

[0192] In this embodiment, a processing device 5000 for visual language training model is provided, as shown in the figure, comprising a sample acquisition module 5100, a text covering module 5200, an image covering module 5300 and a model training module 5400. The sample acquisition module 5100 is configured to acquire a training sample, wherein the training sample comprises a training image and training text for describing the training image; the text covering module 5200 is configured to split the training text in units of phrases, and select a covered phrase from the split training text; the image covering module 5300 is configured to select a first region to be covered in the training image, and a second region adjacent to the first region; and the model training module 5400 is configured to train a visual language training model according to the covered phrase, the covered training text, the training image, the first region and the second region. Figure 5

[0193] In one embodiment of the present disclosure, splitting the training text in units of phrases comprises:

[0194] constructing a phrase structure tree of the training text; wherein the phrase structure tree is used to describe the syntax structure of the training text;

[0195] splitting the training text according to the phrase structure tree.

[0196] In one embodiment of the present disclosure, selecting a covered phrase from the split training text comprises:

[0197] traversing the split training text in units of phrases to generate a random number corresponding to each phrase;

[0198] selecting a phrase with a random number within a set range as a covered phrase.

[0199] In one embodiment of the present disclosure, the model training module 5400 can be further configured to:

[0200] performing a first training task of the visual language training model according to the covered phrase, the covered training text and the training image;

[0201] performing a second training task of the visual language training model according to the training image, the first region and the second region.

[0202] In one embodiment of the present disclosure, performing a first training task of the visual language training model according to the covered phrase, the covered training text and the training image comprises:​

[0203] obtaining the order of each phrase in the training text and a preset covering vector for representing the covered phrase;

[0204] encoding the order of each phrase, the uncovered phrase and the training image according to the preset neural network, to obtain an order vector, an uncovered phrase vector and a training image vector respectively;

[0205] taking the undetermined parameters of the visual language training model as variables, obtaining the accuracy rate of predicting each covered phrase according to the covering vector, the order vector, the uncovered phrase vector and the training image vector;

[0206] solving the undetermined parameters according to the accuracy rate of the covered phrase.

[0207] In an embodiment of the present disclosure, the second training task of the visual language training model according to the training image, the first region and the second region comprises:

[0208] encoding the training image, the first region and the second region according to the preset neural network to obtain a training image vector, a first region vector and a second region vector;

[0209] based on the value of the undetermined parameters of the visual language training model obtained through the first training task, taking the undetermined parameters as variables, predicting a predicted region vector of the first region according to the training image vector and the second region vector;

[0210] solving the undetermined parameters according to the predicted region vector and the first region vector.

[0211] In an embodiment of the present disclosure, solving the undetermined parameters according to the predicted region vector and the first region vector comprises:

[0212] calculating the distance between the predicted region vector and the first region vector;

[0213] solving the value of the undetermined parameters at the minimum distance.

[0214] In an embodiment of the present disclosure, solving the undetermined parameters according to the predicted region vector and the first region vector comprises:

[0215] solving the value of the undetermined parameters when the predicted region vector and the first region vector belong to the preset same category.

[0216] In an embodiment of the present disclosure, the processing device 5000 can further comprise:

[0217] a module for providing a setting entry for setting the application scenario of the visual language training model;

[0218] a module configured to acquire an application scenario input through a setting portal;

[0219] The sample acquisition module is configured to acquire a corresponding training sample according to the input application scenario.

[0220] In an embodiment of the present disclosure, the processing apparatus 5000 can further include:

[0221] a module configured to determine an application scenario to which the visual language training model is applicable;

[0222] a module configured to find an application project matched with the application scenario;

[0223] a module configured to input the visual language training model into the application project.

[0224] In an embodiment of the present disclosure, after the training of the visual language training model is performed, the processing apparatus 5000 can further include:

[0225] a module configured to acquire image-text information to be processed; wherein the image-text information includes an image and / or text;

[0226] a module configured to process the image-text information to be processed by using the visual language training model to obtain a corresponding processing result.

[0227] Those skilled in the art should understand that the processing apparatus 5000 of the visual language training model can be implemented in various ways. For example, the processing apparatus 5000 of the visual language training model can be implemented by configuring a processor with instructions. For example, the instructions can be stored in a ROM, and when the device is started, the instructions are read from the ROM to a programmable device to implement the processing apparatus 5000 of the visual language training model. For example, the processing apparatus 5000 of the visual language training model can be fixed in a special device (such as an ASIC). The processing apparatus 5000 of the visual language training model can be divided into independent units, or they can be combined together to implement. The processing apparatus 5000 of the visual language training model can be implemented by one of the above-mentioned various implementation ways, or can be implemented by a combination of two or more of the above-mentioned various implementation ways.

[0228] In the present embodiment, the processing apparatus 5000 of the visual language training model can have various implementation forms. For example, the processing apparatus 5000 of the visual language training model can be a functional module running in any software product or application program providing model management services, or a peripheral embedded part, plug-in, patch, etc. of these software products or application programs, and can also be the software products or application programs themselves.

[0229] In the embodiments of the present specification, by splitting the training text into phrases for covering, some strongly related words can be avoided from being split, and the phrases and image regions can be better aligned. By selecting a second region adjacent to the first region to be covered, the difficulty of the model in predicting the region category can be reduced, and the relationship between the first region and the second region can be better established. In this way, the effect of the visual language training model can be further improved.

[0230] <Electronic device>

[0231] In the embodiments of the present specification, an electronic device 1000 is also provided. The electronic device 1000 can be a server 1100 as shown in FIG. 1 or a terminal device 1200 as shown in FIG. 2. Figure 1a Figure 1b

[0232] As shown in FIG. 1, the electronic device 1000 can further include a processor 1300 and a memory 1400 for storing executable instructions; the processor 1300 is configured to control the electronic device 1000 to perform the processing method of the visual language training model according to any embodiment of the present disclosure according to the instructions. Figure 6

[0233] For example, the electronic device 1000 can be a computer cluster capable of providing management services for the visual language training model.

[0234] In the embodiments of the present specification, by splitting the training text into phrases for covering, some strongly related words can be avoided from being split, and the phrases and image regions can be better aligned. By selecting a second region adjacent to the first region to be covered, the difficulty of the model in predicting the region category can be reduced, and the relationship between the first region and the second region can be better established. In this way, the effect of the visual language training model can be further improved.

[0235] <Computer readable storage medium>

[0236] In the embodiments of the present specification, a computer readable storage medium having a computer program stored thereon is also provided, and the computer program, when executed by a processor, implements the processing method of the visual language training model according to any embodiment of the present disclosure.

[0237] The present disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions embodied therewith, and the computer readable program instructions are used to cause a processor to implement various aspects of the present disclosure.

[0238] ​​​Computer readable storage media can be tangible storage media which can retain and store instructions for use by an instruction execution device. Computer readable storage media can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer readable storage media include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0239] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0240] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0241] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0242] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other data storage device. When the computer readable program instructions are loaded into the computer and other programmable data processing apparatus, a series of operational steps are implemented that provide processes such that the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0243] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0244] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0245] Embodiments of the present disclosure have been described above, with examples of the description being exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of words in this document is intended to best explain the principles of the embodiments, practical application, or improvement to the art in the market, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein. The scope of the present disclosure is defined by the appended claims.

Claims

1. A processing method of a visual language training model, comprising: obtaining a training sample, wherein the training sample comprises a training image and a training text used to describe the training image; splitting the training text in units of phrases, and selecting a covered phrase from the split training text; selecting a first region covered in the training image, and a second region adjacent to the first region; performing a first training task of the visual language training model according to the covered phrase, the covered training text and the training image, solving a to-be-determined parameter of the visual language training model based on a value of the to-be-determined parameter, taking the to-be-determined parameter as a variable, predicting a predicted region vector of the first region according to a covered training image vector obtained by encoding the covered training image, and a second region vector obtained by encoding the second region, and solving the to-be-determined parameter according to the predicted region vector and a first region vector obtained by encoding the first region.

2. The method of claim 1, wherein the splitting the training text in units of phrases comprises: constructing a phrase structure tree of the training text, wherein the phrase structure tree is used to describe a syntax structure of the training text; splitting the training text according to the phrase structure tree.

3. The method of claim 1, wherein the selecting a covered phrase from the split training text comprises: traversing the split training text in units of phrases to generate a random number corresponding to each phrase; selecting a phrase with a random number within a set range as the covered phrase.

4. The method of claim 1, wherein the performing the first training task of the visual language training model according to the covered phrase, the covered training text and the training image comprises: obtaining a word order of each phrase in the training text and a preset covering vector used to represent the covered phrase; encoding the word order of each phrase, an uncovered phrase and the training image according to a preset neural network to obtain a word order vector, an uncovered phrase vector and a training image vector, respectively; taking the to-be-determined parameter of the visual language training model as a variable, obtaining an accuracy rate of each covered phrase predicted according to the covering vector, the word order vector, the uncovered phrase vector and the training image vector; solving the to-be-determined parameter according to the accuracy rate of the covered phrase.

5. The method of claim 1, further comprising: encoding the covered training image, the first region and the second region according to a preset neural network to obtain the covered training image vector, the first region vector and the second region vector, respectively.

6. The method of claim 1, wherein the solving the to-be-determined parameter according to the predicted region vector and the first region vector obtained by encoding the first region comprises: calculating a distance between the predicted region vector and the first region vector; solving a value of the to-be-determined parameter when the distance is smallest.

7. The method of claim 1, wherein solving the pending parameter based on the predicted region vector and a first region vector obtained by encoding the first region comprises: solving a value of the pending parameter when the predicted region vector and the first region vector belong to a preset same category.

8. The method of claim 1, wherein after the training of the visual language training model, the method further comprises: obtaining to-be-processed image-text information, wherein the image-text information comprises an image and / or text; processing the to-be-processed image-text information using the visual language training model to obtain a corresponding processing result.

9. A processing apparatus of a visual language training model, comprising: a sample obtaining module configured to obtain training samples, wherein the training samples comprise training images and training texts used to describe the training images; a text covering module configured to split the training texts in units of phrases, and select covered phrases from the split training texts; an image covering module configured to select a first region to be covered in the training images, and a second region adjacent to the first region; and a model training module configured to perform a first training task of a visual language training model based on the covered phrases, covered training texts, and the training images, determine a value of a pending parameter of the visual language training model obtained through the first training task, predict a predicted region vector of the first region based on the pending parameter as a variable, a covered training image vector obtained by encoding the covered training images, and a second region vector obtained by encoding the second region, and solve the pending parameter based on the predicted region vector and a first region vector obtained by encoding the first region.

10. An electronic device, comprising: a processor and a memory configured to store instructions for controlling the processor to perform the method of any one of claims 1-8.

11. A computer-readable storage medium having stored thereon a computer program, the computer program, when executed by a processor, implementing the method of any one of claims 1-8. ​ ​ ​ ​ ​ ​ ​ ​

Images