Text processing method and device, equipment and medium

Abstract

The invention provides a text processing method and device, equipment and a medium. The method comprises the following steps: performing text coding on a second language sample text through an initial text coder to obtain a second text coding feature, and performing text coding on a first language sample text through a reference text coder to obtain a first text coding feature; performing parameter optimization on parameters except the frozen training parameters in the initial text encoder through the first text encoding feature, the second text encoding feature and the frozen training parameters to obtain a parameter-optimized text encoder, and performing text encoding on the second language sample text to obtain optimized text encoding features, performing image coding on the sample image through a reference image coder to obtain image coding features; and performing iterative training on the text encoder after parameter optimization through the image encoding features and the optimized text encoding features to obtain a target text encoder. The training cost and the training difficulty can be reduced.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a text processing method, apparatus, device and medium. Background Technology

[0002] Currently, image-text models (such as CLIP (Contrastive Language Image Pre-training) models) can be trained for cross-modal matching of images and text. For example, in text-to-image search scenarios, an image-text model can be used to obtain an image that matches text A. Similarly, in image-to-text search scenarios, an image-text model can be used to obtain text that matches image B.

[0003] One approach is to train the image-text model from scratch using a large number of sample texts and corresponding matching sample images. However, the large amount of sample data results in a long training time, increasing training costs. Furthermore, the large number of parameters in the entire image-text model makes training difficult. Summary of the Invention

[0004] This application provides a text processing method, apparatus, device, and medium that can reduce training costs and difficulty.

[0005] On one hand, embodiments of this application provide a text processing method, which includes:

[0006] Obtain the first language sample text used to train the initial text encoder, the second language sample text translated from the first language sample text, and the sample image corresponding to the first language sample text; the parameters in the initial text encoder include frozen training parameters, which are derived from the reference text encoder associated with the first language sample text.

[0007] When text encoding the second language sample text through the initial text encoder to obtain the second text encoding features, the first text encoding features obtained by text encoding the first language sample text through the reference text encoder are also obtained.

[0008] By using the first text encoding feature, the second text encoding feature, and the frozen training parameters, the parameters in the initial text encoder other than the frozen training parameters are optimized to obtain the optimized text encoder.

[0009] When the second language sample text is encoded by the optimized text encoder to obtain the optimized text encoding features, the image encoding features obtained by encoding the sample image by the reference image encoder associated with the reference text encoder are also obtained.

[0010] The text encoder with optimized parameters is iteratively trained using image encoding features and optimized text encoding features to obtain the target text encoder. The target text encoder is used to predict at least the target image corresponding to the output target language text, and the target language text has the same language type as the second language sample text.

[0011] On one hand, embodiments of this application provide a text processing apparatus, the apparatus comprising:

[0012] The sample acquisition module is used to acquire the first language sample text for training the initial text encoder, the second language sample text translated from the first language sample text, and the sample image corresponding to the first language sample text; the parameters in the initial text encoder include frozen training parameters, which are derived from the reference text encoder associated with the first language sample text.

[0013] The feature encoding module is used to obtain the first text encoding features obtained by text encoding the first language sample text through the reference text encoder when the second text encoding features are obtained by text encoding the second language sample text through the initial text encoder.

[0014] The model training module is used to optimize the parameters of the initial text encoder, excluding the frozen training parameters, using the first text encoding features, the second text encoding features, and the frozen training parameters, to obtain the optimized text encoder.

[0015] The feature encoding module is also used to obtain the image encoding features obtained by encoding the sample image through the reference image encoder associated with the reference text encoder when the second language sample text is encoded by the parameter-optimized text encoder to obtain optimized text encoding features.

[0016] The model training module is also used to iteratively train the optimized text encoder using image encoding features and optimized text encoding features to obtain the target text encoder. The target text encoder is used to predict at least the target image corresponding to the output target language text, and the target language text has the same language type as the second language sample text.

[0017] One aspect of this application provides a computer device, including a memory and a processor. The memory is connected to the processor, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method provided in any of the above aspects of this application.

[0018] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having a processor performs the method provided in any of the above aspects of this application.

[0019] According to one aspect of this application, a computer program product is provided, the computer program product including a computer program that, when executed by a processor, implements the methods provided in any of the above aspects of the embodiments of this application.

[0020] In this embodiment, existing reference text encoders and reference image encoders can be used as references to train the initial text encoder, thereby obtaining a target text encoder for the language type of the second language sample text. Through knowledge distillation, the target text encoder can quickly learn the capabilities of the reference text encoder and adapt to the reference image encoder. During training, the output of the reference text encoder is first used as a supervision signal to train the parameters of the initial text encoder, excluding the frozen training parameters. Then, the output of the reference image encoder is used as a supervision signal to further train the initial text encoder, achieving phased model training. This reduces the training difficulty. By inheriting some parameters from the reference text encoder during training (i.e., freezing the training parameters), the number of parameters trained can be reduced. Compared to training the initial text encoder from scratch, this reduces the amount of sample data used, effectively reducing training costs and difficulty while improving the training effect of the target text encoder. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A schematic diagram of a network architecture provided for an embodiment of this application;

[0023] Figure 2 This is a schematic diagram of a data processing procedure provided in an embodiment of this application;

[0024] Figure 3 A flowchart illustrating a text processing method provided in an embodiment of this application;

[0025] Figure 4 A schematic diagram illustrating the training process of a text encoder provided in an embodiment of this application;

[0026] Figure 5 A schematic diagram illustrating the training process of another text encoder provided in this application embodiment;

[0027] Figure 6 A schematic diagram of a text-image matching scenario provided in an embodiment of this application;

[0028] Figure 7 A schematic diagram of another image-text matching scenario provided for an embodiment of this application;

[0029] Figure 8 A flowchart illustrating a text processing method provided in an embodiment of this application;

[0030] Figure 9 A schematic diagram illustrating a training sample acquisition scenario provided in an embodiment of this application;

[0031] Figure 10 A schematic diagram of an optimized training scenario provided in an embodiment of this application;

[0032] Figure 11 A schematic diagram illustrating a model training process provided in an embodiment of this application;

[0033] Figure 12 This is a schematic diagram of the structure of a text processing device provided in an embodiment of this application;

[0034] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0036] Please see Figure 1 , Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application. For example... Figure 1 As shown, the system architecture may include a business server 100 and a business terminal cluster. The business terminal cluster may include one or more business terminals (e.g., user terminals). The number of business terminals in the business terminal cluster is not limited here. Figure 1As shown, the multiple service terminals in the service terminal cluster may specifically include: service terminal 200a, service terminal 200b, ..., service terminal 200n. Communication connections can exist between the service terminals in the cluster; for example, there is a communication connection between service terminal 200a and service terminal 200b, and between service terminal 200a and service terminal 200n. Simultaneously, any service terminal in the service terminal cluster can have a communication connection with the service server 100, so that each service terminal in the cluster can interact with the service server 100 through this communication connection. For example, there is a communication connection between service terminal 200a and service server 100. The above communication connection is not limited to a specific method; it can be established directly or indirectly through wired communication, wireless communication, or other methods. This application does not impose any restrictions on this method.

[0037] Among them, such as Figure 1 Each business terminal in the shown business terminal cluster can be equipped with an application client for data processing. When the application client runs on each business terminal, it can interact with the aforementioned... Figure 1 The business servers 100 shown interact with each other. The application client can be any type of client, such as a social networking client, image processing client, instant messaging client (e.g., conferencing client), entertainment client (e.g., game client, live streaming client), multimedia client (e.g., video client), information client (e.g., news client), shopping client, in-vehicle client, multimedia client, application download client (a client used to provide users with various downloadable application resources), etc., clients with the ability to display text, images, audio, and video data. The specific type of application client is not limited here.

[0038] For example, an application client refers to a client that can send and receive Internet messages in real time and has information search functions. Business object X can upload target language text in the application client on business terminal 200a. Business server 100 can predict and output the target image corresponding to the target language text through the target text encoder trained by the technical solution of this application, and return the target image corresponding to the target language text to business terminal 200a.

[0039] In this application, the computer device involved in the embodiments may be a server (e.g., Figure 1 The business server 100 shown can also be a terminal (e.g., Figure 1(Any one of the business terminals in the business terminal cluster shown). The server can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smart TV, smartwatch, vehicle terminal, aircraft, etc., but is not limited to these. This application's embodiments can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0040] in, Figure 1 This is merely an example to characterize the possible network architectures of the technical solutions in this application, and does not limit the specific architecture of the technical solutions in this application. That is, the technical solutions in this application can also provide other forms of network architecture.

[0041] Please see below. Figure 2 , Figure 2 This is a schematic diagram of a data processing procedure provided in an embodiment of this application. The process involves acquiring training samples for training an initial text encoder. These training samples may include first language sample text 21a, second language sample text 21b translated from the first language sample text, and sample images 21c (sample images matching the first language sample text) corresponding to the first language sample text.

[0042] The parameters in the initial text encoder 22a include frozen training parameters (parameters that remain fixed during training), which (partially or entirely) come from a reference text encoder associated with the first language sample text.

[0043] Specifically, the first language sample text can be input into the reference text encoder 22b, and the reference text encoder performs text encoding on the first language sample text to obtain the first text encoding feature 23a; the second language sample text can be input into the initial text encoder, and the reference text encoder performs text encoding on the second language sample text to obtain the second text encoding feature 23b.

[0044] At this point, the parameters (i.e., the parameters to be trained) in the initial text encoder, excluding the frozen training parameters, can be optimized using the first text encoding feature, the second text encoding feature, and the frozen training parameters, resulting in the optimized text encoder 24a.

[0045] The next stage of training can be carried out, such as inputting the second language sample text into the optimized text encoder, which encodes the second language sample text to obtain optimized text encoding features 25a; or inputting the sample image into the reference image encoder 22c associated with the reference text encoder, which encodes the sample image to obtain image encoding features 25b.

[0046] At this point, the optimized text encoder can be iteratively trained using image encoding features and optimized text encoding features to obtain the target text encoder 26a. This target text encoder can not only learn the performance of the reference text encoder, but also adapt to the image encoder, enabling the target text encoder to match language text and images of the same language type as the second language sample text with the image encoder.

[0047] In this way, by learning a reference text encoder and image encoder for the language type of first-language sample text, a target text encoder for the language type of second-language sample text can be trained (which can significantly reduce the number of parameters required for training compared to training the entire image-text processing model). A text encoder for one language type can be extended to text encoders for different language types, enabling matching between text and images in different language types.

[0048] The aforementioned target text encoder is used at least to predict the target image corresponding to the output target language text, and the target language text and the second language sample text have the same language type.

[0049] For example, the target language text is input into the target text encoder, and the target image to be matched is input into the reference image encoder to obtain the target image corresponding to the target language text (the target image that matches the target language text).

[0050] Optionally, the computer device can execute this text processing method to improve the text-image matching effect according to actual business needs. Specifically, a text encoder adapted to other language types can be trained by using a well-performing reference text encoder and a corresponding reference image encoder as references.

[0051] Meanwhile, by using knowledge distillation, inheriting the parameters of the learned reference text encoder to freeze some of the inherited training parameters, and conducting phased training, the trained text encoders for other language types can achieve similar training results to the reference text encoder while reducing training costs, thus enabling accurate text-image matching in any language type.

[0052] The image-text matching scenarios applied in this application's technical solution may include image-to-text search scenarios, text-to-text search scenarios, text-to-image search scenarios, and so on.

[0053] For example, in a text-to-image search scenario, by inputting target language text, the target text encoder and the reference image encoder can search for the target image corresponding to the target language text.

[0054] For example, in the image-to-text search scenario, by inputting a text library and a target image, the target text encoder and the reference image encoder can search the text library to obtain the target language text corresponding to the target image.

[0055] Optionally, zero-shot classification tasks (such as determining the category of an object contained in a target image by using target language text describing the category) and text-to-image generation tasks (such as generating a corresponding image by using target language text as a given description) can also be achieved through a target text encoder and a reference image encoder. The application scenarios are not limited here.

[0056] It should be noted that when the computer device in this application embodiment acquires relevant data, such as target language text uploaded by a business object (user), it may display a prompt interface or pop-up window. The prompt interface or pop-up window is used to prompt the user that the aforementioned image and other data are being acquired. The data acquisition steps will only begin after the user confirms the prompt interface or pop-up window; otherwise, the process will end.

[0057] It is understood that in the specific implementation of this application, user data (e.g., the input target language text) may be involved. When the above embodiments of this application are applied to specific products or technologies, the user's permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0058] It is understood that the above scenarios are merely examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of this application. The technical solutions of this application can also be applied to other scenarios. For example, as those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0059] For further details, please see Figure 3 , Figure 3 This is a flowchart illustrating a text processing method provided in an embodiment of this application, such as... Figure 3 As shown, the method can be executed by the aforementioned computer devices, such as... Figure 1For any of the service terminals or service servers shown, the method may specifically include the following steps S101-S105:

[0060] S101. Obtain the first language sample text used to train the initial text encoder, the second language sample text translated from the first language sample text, and the sample image corresponding to the first language sample text.

[0061] One first-language sample text corresponds to one second-language sample text and one sample image.

[0062] For example, a first-language sample text (such as an English sentence) of a first-language type (such as English) can be translated to obtain a translated text (such as a corresponding Chinese sentence) of another language type (such as Chinese), which can then be used as a second-language sample text. The language type of this second-language sample text is the second-language type.

[0063] The sample image corresponding to the first language sample text is an image that matches the first language sample text. For example, it can be an image obtained by performing a text search on the first language sample text, or the first language sample text and its corresponding sample image can be an open-source image-text data pair.

[0064] The first language sample text corresponds to a text processing model (such as the CLIP model). The reference text encoder associated with the first language sample text can be a text encoder in a pre-trained text processing model. The reference text encoder is associated with a reference image encoder, and the reference image encoder can be an image encoder in a pre-trained text processing model.

[0065] The parameters in the initial text encoder include frozen training parameters, which are derived from the reference text encoder associated with the first language sample text (the frozen training parameters are inherited from the reference text encoder, further reducing the number of parameters that need to be trained in the initial text encoder).

[0066] Here, the frozen training parameters are those parameters that are frozen and fixed during training. The reference parameters in the initial text encoder, excluding the frozen training parameters, are the parameters to be trained, i.e., the parameters that need to be optimized during training.

[0067] For example, a well-trained image-text processing model can be obtained, and an initial text encoder to be trained can be obtained through the text encoder in the image-text processing model (such as using a reference text encoder as the initial text encoder). The initial text encoder can then be trained using the image-text processing model as a reference. In this way, the target text encoder obtained can be combined with the image encoder in the image-text processing model to construct an image-text processing model for the second language type.

[0068] S102. When the second language sample text is encoded by the initial text encoder to obtain the second text encoding features, the first text encoding features obtained by encoding the first language sample text by the reference text encoder are acquired.

[0069] The reference text encoder corresponds to a first text segmenter (such as an English segmenter) for the first language type. The first text segmenter can be used to segment the first language sample text to obtain the segmentation information of the first language sample text, and then input into the reference text encoder for text encoding to obtain the first text encoding feature.

[0070] The initial text encoder corresponds to a second text segmenter for the second language type, such as a Chinese segmenter. Alternatively, if you want to train a text encoder primarily for a certain language type, you can choose to use a text segmenter for that language type (or if you want to train a text encoder for multiple language types, you can choose the text segmenter corresponding to the language type with the most linguistic information).

[0071] For example, for a text encoder that can perform bilingual Chinese and English, with Chinese as the main language (or with more information in Chinese than in English), then a Chinese word segmenter or a word segmenter that can perform bilingual Chinese word segmentation can be selected, without any restrictions here.

[0072] Specifically, the second text segmenter can be used to segment the second language sample text to obtain the segmentation information of the second language sample text, and then input into the initial text encoder for text encoding to obtain the second text encoding features.

[0073] S103. Using the first text encoding feature, the second text encoding feature, and the frozen training parameters, optimize the parameters of the initial text encoder except for the frozen training parameters to obtain the optimized text encoder.

[0074] This method involves keeping the frozen training parameters fixed, optimizing the training parameters using the first and second text encoding features (optimization is performed after initializing the training parameters), and obtaining the optimized text encoder when the initial text encoder converges. The parameters in this optimized text encoder still include the frozen training parameters.

[0075] The above approach uses the first text encoding feature (the output of the reference text encoder) as a supervision signal (to achieve knowledge distillation of the reference text encoder, allowing its performance to be transferred to the initial text encoder), and combines the feature deviation between the first and second text encoding features to train the initial text encoder (i.e., optimize the parameters to be trained), thus obtaining a parameter-optimized text encoder. For example, the MSE (mean-square error) loss function can be used for parameter optimization.

[0076] Optionally, the initial text encoder includes an initial feature embedding layer and an initial feature encoding layer. The reference text encoder includes a reference feature embedding layer and a reference feature encoding layer. Optionally, the parameters in the initial feature encoding layer may be partially or entirely derived from the reference text encoder.

[0077] The parameters to be trained can be some or all of the parameters in the initial feature embedding layer and the initial feature encoding layer. For example, the parameters to be trained can include all the parameters in the initial feature embedding layer and some of the parameters in the initial feature encoding layer. If the initial feature encoding layer includes multiple feature encoding sub-layers, the parameters can be some or all of the parameters in those multiple feature encoding sub-layers, such as the parameters in the first number of feature encoding sub-layers (e.g., if there are 6 feature encoding sub-layers, the parameters can be the parameters in the first 3 or the first 4 feature encoding sub-layers, which is not limited here).

[0078] Optionally, the initial feature embedding layer and the initial feature encoding layer can be trained in stages. For example, the frozen training parameters in the initial text encoder can be the first frozen training parameters.

[0079] In this process, the parameters of the initial text encoder, excluding the frozen training parameters, are optimized to obtain the optimized text encoder. This can be achieved by using the parameters in the initial feature encoding layer as the first frozen training parameters; keeping the first frozen training parameters fixed, optimizing the parameters in the initial feature embedding layer using the first and second text encoding features (with the first text encoding features as the supervision signal) to obtain the transition feature embedding layer; and using the initial text encoder containing the transition feature embedding layer and the initial feature encoding layer as the transition text encoder.

[0080] At this point, after initializing the parameters in the initial feature embedding layer, the initial feature embedding layer is trained. This allows the trained transitional feature embedding layer to learn the performance of the reference feature embedding layer.

[0081] Since the text structures of different languages are different, the tokenization information obtained by the corresponding tokenizers for texts representing the same concept is different (e.g., the English "a cat" corresponds to the Chinese "一种猫". The English "a cat" may be split into two tokens, while the Chinese "一种猫" may be split into three tokens).

[0082] In this way, during training, when the tokenization information of the second-language sample text is encoded in the feature embedding layer of the initial text encoder, it can be fused to achieve the effect of the tokenization information of the first-language sample text being encoded in the feature embedding layer of the reference text encoder. This makes the output of the initial feature embedding layer for the second-language sample text similar to the output of the reference feature embedding layer for the first-language sample text with the same meaning. Thus, a preliminary alignment between the initial text encoder and the reference text encoder can be achieved. Since only a small part of the parameters in the initial text encoder need to be trained, the training and convergence processes are accelerated.

[0083] Then, even if the language types are different, when inputting language texts with the same meaning, the output of the transitional feature embedding layer can be similar to the output of the reference feature embedding layer, that is, the initial text encoder implicitly learns the performance of the reference text encoder preliminarily.

[0084] Among them, when encoding the second-language sample text through the transitional text encoder to obtain the transitional text encoding features, the parameters in the transitional text encoder can be optimized by using the first text encoding features and the transitional text encoding features to obtain a text encoder with optimized parameters. For example, all or some of the parameters in the transitional text encoder can be optimized.

[0085] Among them, the initial feature encoding layer can also be trained to achieve a further alignment of the initial text encoder implicitly with the reference text encoder.

[0086] Optionally, the parameters in the transitional text encoder include frozen training parameters, and the frozen training parameters in the transitional text encoder are the second frozen training parameters (i.e., the first frozen training parameters in the transitional text encoder include the second frozen training parameters).

[0087] For example, the second frozen training parameters can be the parameters in the initial feature encoding layer. At this time, the first frozen training parameters include the second frozen training parameters, and the parameters in the first frozen training parameters other than the second frozen training parameters are the parameters to be trained at this stage.

[0088] Therefore, when the second language sample text is encoded by the transitional text encoder to obtain the transitional text encoding features, the parameters of the transitional text encoder other than the second frozen training parameters are optimized using the first text encoding features, the transitional text encoding features, and the second frozen training parameters to obtain the optimized text encoder.

[0089] For example, if the initial feature coding layer includes J coding layers (feature coding sub-layers), then the reference feature coding layer also includes the same number of feature coding sub-layers. The parameters in the feature coding sub-layers other than the first j feature coding sub-layers in the initial feature coding layer can be the second frozen training parameters (the parameters in the feature coding sub-layers other than the first j feature coding sub-layers in the initial feature coding layer come from the feature coding sub-layers other than the first j feature coding sub-layers in the reference feature coding layer).

[0090] Therefore, the second frozen training parameters can be kept fixed, and the parameters in the transition feature embedding layer and the parameters of the first j feature encoding sub-layers in the initial feature encoding layer can be optimized using the first text encoding features and the transition text encoding features (with the first text encoding features as the supervision signal) to obtain the optimized feature embedding layer and the optimized feature encoding layer; the transition text encoder containing the optimized feature embedding layer and the optimized feature encoding layer is used as the text encoder after parameter optimization.

[0091] Where J is a positive integer, and j is a positive integer less than or equal to J. The ratio of j to J is a preset value (such as 1 / 2 or 1 / 3, etc.), such as training the first half of the feature encoding sub-layers in the initial feature encoding layer (freezing the remaining feature encoding sub-layers), or training the first 1 / 3 of the feature encoding sub-layers in the initial feature encoding layer (freezing the remaining feature encoding sub-layers). There is no limitation here, and it can be set according to the specific scenario.

[0092] Compared to training all parameters of the J feature coding sub-layers, training only the parameters of a portion of the coding layers can reduce the training workload and improve the training convergence speed.

[0093] At this point, during training, all parameters in the initial feature encoding layer except for the second frozen training parameters are initialized. Then, the transition feature embedding layer and the initialized initial feature encoding layer are trained. This allows the trained transition feature embedding layer to further learn the performance of the reference feature embedding layer, and the first j feature encoding sub-layers in the initial feature encoding layer to learn the performance of the first j feature encoding sub-layers in the reference feature encoding layer.

[0094] In this way, during training, the word segmentation information of the second language sample text can be encoded in the feature encoding layer of the initial text encoder until the output of the j-th feature encoding sub-layer, so that it can be fused into the word segmentation information of the first language sample text with the same meaning in the feature encoding layer (j-th feature encoding sub-layer) of the reference text encoder, thus achieving precise alignment between the initial text encoder and the reference text encoder.

[0095] Therefore, even if the language types are different, when inputting language text with the same meaning, the output of the j-th feature encoding sublayer of the optimized feature encoding layer can be similar to the output of the j-th feature encoding sublayer of the reference feature encoding layer. This allows the output of the finally parameter-optimized text encoder to be similar to the output of the reference text encoder, thus enabling the reference text encoder to further transfer its performance to the parameter-optimized text encoder.

[0096] By structurally decomposing the initial text encoder to achieve phased training (the first phase freezes the first set of training parameters, and the second phase freezes the second set of training parameters, which are the parameters from the first set), the training difficulty can be reduced, and the alignment between the parameter-optimized text encoder and the reference text encoder can be improved. Furthermore, after initial alignment of the initial feature embedding layer, aligning the transition feature embedding layer with a portion of the initial feature encoding layer reduces the amount of sample data required, while preventing overfitting and gradient explosion. This significantly reduces training time and computational cost compared to full-parameter training.

[0097] For example, such as Figure 4 As shown, Figure 4 This is a schematic diagram of the training process of a text encoder provided in an embodiment of this application; wherein, a first language sample text 40a is input to a first word segmenter 41a to obtain the word segmentation information of the first language sample text, and is then input to a reference text encoder 42a to obtain a first text encoding feature 43a. A second language sample text 40b is input to a second word segmenter 41b to obtain the word segmentation information of the second language sample text, and is then input to an initial text encoder 42b to obtain a second text encoding feature 43b.

[0098] For example, an initial text encoder may include an initial feature embedding layer and an initial feature encoding layer, which includes multiple feature encoding sub-layers in series (e.g., 4). Each feature encoding sub-layer may be a transform layer (a type of neural network).

[0099] In the first stage of training, the parameters in the initial feature embedding layer of the initial text encoder are the parameters to be trained, and the parameters in the initial feature encoding layer of the initial text encoder are the first frozen training parameters.

[0100] At this point, the first frozen training parameters are frozen, and the parameters to be trained in the initial feature embedding layer are trained (parameter optimization) using the first text encoding features and the second text encoding features to obtain the transition feature embedding layer. The initial text encoder containing the transition feature embedding layer and the initial feature encoding layer is used as the transition text encoder 44a.

[0101] The word segmentation information of the second language sample text obtained by the second word segmenter is input into the transition text encoder to obtain the transition text encoding feature 45a.

[0102] In the second stage of training, the parameters in the transition feature embedding layer of the transition text encoder and a portion of the encoding layers in the initial feature encoding layer (such as the first two feature encoding sub-layers) are the parameters to be trained, while the parameters in another portion of the encoding layers in the initial feature encoding layer (such as the last two feature encoding sub-layers) are the second frozen training parameters for the second stage.

[0103] At this point, the second set of training parameters can be frozen. The parameters to be trained in the transitional feature embedding layer and the initial feature encoding layer of the transitional text encoder are then trained (parameter optimization) using the first text encoding features and the transitional text encoding features until convergence. The trained transitional feature embedding layer becomes the optimized feature embedding layer, and the trained initial feature encoding layer becomes the optimized feature encoding layer. The transitional text encoder containing the aforementioned optimized feature embedding layer and optimized feature encoding layer serves as the parameter-optimized text encoder 44b.

[0104] S104. When the second language sample text is encoded by the optimized text encoder to obtain optimized text encoding features, the image encoding features obtained by encoding the sample image by the reference image encoder associated with the reference text encoder are acquired.

[0105] Among them, the word segmentation information of the second language sample text can be input into the optimized text encoder to obtain optimized text encoding features.

[0106] This process involves preprocessing the sample images (such as extracting the content of the sample images), inputting the image content information of the sample images into the reference image encoder associated with the reference text encoder, and then performing image encoding to obtain image encoding features.

[0107] S105. Through image encoding features and optimized text encoding features, the text encoder with optimized parameters is iteratively trained to obtain the target text encoder.

[0108] Specifically, image encoding features can be used as supervision information to train a parameter-optimized text encoder (with the reference image encoder frozen) together with optimized text encoding features until convergence, so that the target text encoder can be aligned with the reference image encoder. This allows for image-text matching for second language types using the target text encoder and the reference image encoder.

[0109] For example, a contrastive learning approach can be used to train a text encoder with optimized parameters.

[0110] Since the alignment reference text encoder stage has already achieved good convergence, the alignment reference image encoder stage can achieve good results with only a small amount of image and text data for fine-tuning. Compared to full-parameter training, this effectively reduces training time and computational costs.

[0111] For example, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the training process of a text encoder provided in an embodiment of this application; wherein, after the training of the first stage and the second stage described above, a third stage of training is performed, such as inputting the word segmentation information of the second language sample text 51b obtained by the second word segmenter 51a into the optimized text encoder 52a to obtain optimized text encoding features 53a.

[0112] The image content information of the preprocessed sample image 54a is input into the reference image encoder 52b to obtain the image coding features 54b of the sample image.

[0113] In this way, the optimized text encoder can be iteratively trained using image encoding features and optimized text encoding features until convergence, resulting in the target text encoder 55a, which is aligned with the reference image encoder.

[0114] This method enables matching between the output of a second language sample text with the same meaning as the first language sample text, obtained through a target text encoder, and the output of a sample image that matches the first language sample text, obtained through a reference image encoder. This allows the second language sample text to be matched with the sample image that matches the first language sample text.

[0115] It is understandable that different training methods and loss functions are used at different stages. This allows for more targeted training of the initial text encoder, which can improve training results and better align with and learn performance from the reference text encoder and reference image encoder.

[0116] The target text encoder is a text encoder for the second language type of the second language sample text. For example, it can be used to predict the target image (or the target language text corresponding to the target image) that corresponds to the output target language text. This target language text has the same language type as the second language sample text.

[0117] For example, in a text search image scenario, the target language text to be searched can be obtained; when the target language text is encoded by the target text encoder to obtain the target text encoding features, the image encoding features obtained by encoding the target image in the image library by the reference image encoder can be obtained; based on the feature similarity between the target text encoding features and the image encoding features of the target image, the target image corresponding to the target language text can be selected from the image library.

[0118] For example, in an image-to-text search scenario, the target image to be searched and a text library can be obtained; when the target language text in the text library is encoded by the target text encoder to obtain the target text encoding features, the image encoding features obtained by encoding the target image by the reference image encoder can be obtained; based on the feature similarity between the target text encoding features and the image encoding features of the target image, the target language text corresponding to the target image can be selected from the text library.

[0119] For example, such as Figures 6-7 As shown, Figures 6-7 This application provides a schematic diagram of a text-image matching scenario; wherein, in Figure 6 In this context, text-based or image-based image search tasks can be performed. For example, the target page 60a can display an input area for entering the target language text to be searched (e.g., "Enter the text to be searched") or an upload area for uploading the target image to be searched (e.g., "Upload the image to be searched").

[0120] In the text-to-image search task, users can enter text (such as "a smiling boy") in the input area for the target language text to be searched. After confirming the search, one or more target images corresponding to the target language text matched from the image library can be displayed in the search results display area.

[0121] For example, the encoding results of text encoding of target language text by a target text encoder and the image encoding results of target images in an image library by a reference image encoder can be matched to obtain one or more target images corresponding to the target language text. Alternatively, a specified number of target images with the highest feature similarity to the target language text can be selected as one or more target images corresponding to the target language text.

[0122] Alternatively, in an image search task, after uploading an image in the upload area for uploading the target image to be searched, and then confirming the search, one or more target images corresponding to the uploaded image matched from the image library can be displayed in the search results display area.

[0123] For example, the encoding result of the uploaded image encoded by the reference image encoder can be matched with the encoding result of the target image in the image library encoded by the reference image encoder to obtain one or more target images corresponding to the uploaded image. Alternatively, a specified number of target images with the highest feature similarity to the uploaded image can be used as one or more target images corresponding to the uploaded image.

[0124] exist Figure 7 In this system, users can perform image-to-text or text-to-text search tasks. For example, the target page 70a can display an input area for entering the target language text to be searched (e.g., "Enter text to search"), an upload area for uploading the target image to be searched (e.g., "Upload image to search"), and a text upload area for uploading a searchable text library (e.g., "Upload text library").

[0125] In the image-to-text search task, images are uploaded in the upload area for uploading the target image to be searched, and text libraries (which may include one or more target language texts, such as "mobile phone", "pet", "face cream", "barbecue", "skiing", "library", "bookstore", "audio", "food", "eel", "grass", "wild boar", "sheep", "person", "speaker", "computer", "dog", "cat", "candle", etc.) are uploaded in the text upload area for uploading the searchable text library.

[0126] After confirming the search, one or more target language texts corresponding to the target image matched from the text library can be displayed in the search results display area.

[0127] For example, the encoding result of the target language text (such as "mobile phone") in the text library by the target text encoder is matched with the encoding result of the target image by the reference image encoder to obtain one or more target language texts (such as "mobile phone", "barbecue", "grass", "person") corresponding to the target image in the text library. Alternatively, a specified number of target language texts in the text library with the highest feature similarity to the target image can be used as one or more target language texts corresponding to the target image.

[0128] Alternatively, in a text search task, after entering the text in the input area for inputting the target language text to be searched, and in the text upload area for uploading a searchable text library, the text library can be uploaded.

[0129] The search results display area can show one or more target language texts that match the input text from the text library.

[0130] For example, the encoding result of the input text encoded by the target text encoder is matched with the encoding result of the target language text in the text library encoded by the target text encoder to obtain one or more target language texts corresponding to the input text in the text library. Alternatively, a specified number of target language texts in the text library with the highest feature similarity to the input text can be used as one or more target language texts corresponding to the input text.

[0131] In this embodiment, existing reference text encoders and reference image encoders can be used as references to train the initial text encoder, thereby obtaining a target text encoder for the language type of the second language sample text. Through knowledge distillation, the target text encoder can quickly learn the capabilities of the reference text encoder and adapt to the reference image encoder. During training, the output of the reference text encoder is first used as a supervision signal to train the parameters of the initial text encoder, excluding the frozen training parameters. Then, the output of the reference image encoder is used as a supervision signal to further train the initial text encoder, achieving phased model training. This reduces the training difficulty. By inheriting some parameters from the reference text encoder during training (i.e., freezing the training parameters), the number of parameters trained can be reduced. Compared to training the initial text encoder from scratch, this reduces the amount of sample data used, effectively reducing training costs and difficulty while improving the training effect of the target text encoder.

[0132] For further details, please see Figure 8 , Figure 8 This is a flowchart illustrating a text processing method provided in an embodiment of this application, such as... Figure 8 As shown, the method can be executed by the aforementioned computer devices, such as... Figure 1 For any of the service terminals or service servers shown, the method may specifically include the following steps S201-S208:

[0133] S201. Determine the initial text encoder based on the reference feature embedding layer and reference feature encoding layer in the reference text encoder.

[0134] The initial text encoder can be obtained by referencing the text encoder. For example, the reference feature embedding layer can be used as the initial feature embedding layer in the initial text encoder, and the reference feature encoding layer can be used as the initial feature encoding layer in the initial text encoder.

[0135] Alternatively, the parameters of the reference feature embedding layer can be adjusted to obtain a parameter-adjusted reference feature embedding layer, which can then be used as the initial feature embedding layer. In this case, the initial text encoder inherits the overall structure of the reference text encoder.

[0136] Optionally, the first language sample text is segmented by the first text segmenter corresponding to the reference text encoder and then encoded by the initial text encoder; the second language sample text is segmented by the second text segmenter corresponding to the initial text encoder and then encoded by the initial text encoder.

[0137] The initial text encoder can also be determined based on the first text segmenter and the second text segmenter. The first text segmenter refers to a segmenter adapted to the information content of the first language type corresponding to the reference text encoder.

[0138] The vocabulary information contained in the first text segmenter is adapted to the parameter count of the reference feature embedding layer. If the parameter count of the reference feature embedding layer is greater than or equal to the vocabulary information contained in the first text segmenter, the vocabulary corresponding to the first text segmenter is used for text segmentation.

[0139] Therefore, the second text segmenter is also a segmenter adapted to the information content of the second language type corresponding to the initial text encoder. Thus, the number of parameters required for the initial feature embedding layer can be determined based on the vocabulary information content of the second text segmenter.

[0140] For example, when the vocabulary information of the first text segmenter is less than that of the second text segmenter, it indicates that the information content of the second language type is greater than that of the first language type.

[0141] At this point, the number of parameters in the initial feature embedding layer of the initial text encoder needs to be greater than or equal to the vocabulary information of the second text segmenter. Therefore, the number of parameters in the initial feature embedding layer should be greater than the number of parameters in the reference feature embedding layer.

[0142] Therefore, the reference feature embedding layer can be parameter-expanded to obtain a parameter-expanded reference feature embedding layer; the number of parameters in this parameter-expanded reference feature embedding layer is greater than the number of parameters in the original reference feature embedding layer; the initial feature embedding layer is determined based on the parameter-expanded reference feature embedding layer, and the initial feature coding layer is determined based on the reference feature coding layer. For example, the parameter-expanded reference feature embedding layer can be used as the initial feature embedding layer.

[0143] For example, the amount of information in English is usually less than that in Chinese. Therefore, the vocabulary information of an English word segmenter is less than that of a Chinese word segmenter. If a Chinese (or bilingual) text encoder is trained by an English text encoder (using an English word segmenter), which uses a Chinese word segmenter with more information, then the feature embedding layer in the English text encoder can be parameter-extended. This extended feature embedding layer can then be used as the feature embedding layer in the Chinese (or bilingual) text encoder.

[0144] The parameter expansion can be achieved by replacing a specific network layer in the reference feature embedding layer with a more complex network layer (e.g., replacing the last or several fully connected layers in the reference feature embedding layer with a more complex fully connected layer), or by adding new network layers to the reference feature embedding layer (e.g., increasing the number of fully connected layers in the reference feature embedding layer). No specific method of parameter expansion is limited here.

[0145] Optionally, when the vocabulary information of the first text segmenter is greater than that of the second text segmenter, parameter compression is performed on the reference feature embedding layer to obtain a parameter-compressed reference feature embedding layer, which is then used as the initial feature embedding layer. The number of parameters in the parameter-compressed reference feature embedding layer is less than the number of parameters in the reference feature embedding layer, but greater than or equal to the vocabulary information of the second text segmenter.

[0146] Alternatively, when the vocabulary information of the first text segmenter is greater than or equal to that of the second text segmenter, the reference feature embedding layer can be directly used as the initial feature embedding layer.

[0147] Parameter compression can be achieved by replacing specific network layers in the reference feature embedding layer with simpler network layers (e.g., replacing the last or several fully connected layers in the reference feature embedding layer with simpler fully connected layers), or by reducing the number of network layers in the reference feature embedding layer (e.g., reducing the number of fully connected layers in the reference feature embedding layer), and so on. No specific method of parameter compression is limited here.

[0148] S202, Obtain the first language sample text used to train the initial text encoder, the second language sample text translated from the first language sample text, and the sample image corresponding to the first language sample text.

[0149] The first language sample text is of the first language type, and the second language sample text is of the second language type. The first and second language types can be any language type. The second language type can be different from the first language type; for example, the first language type could be one language (e.g., English), and the second language type could be another language (e.g., Chinese).

[0150] Alternatively, the second language type can be the same as the first language type. For example, the first language type is a language type (such as English), and the second language type is the aforementioned language type (such as English), or it may include the aforementioned language type (such as English) as well as other language types (such as Chinese). There are no restrictions here.

[0151] Therefore, the trained target text encoder can perform text encoding on target language text of second language type (such as supporting Chinese, or supporting Chinese and English bilingual).

[0152] For example, second language sample text is obtained by translating first language sample text, such as translating English sample text into Chinese sample text. In this case, a text encoder for Chinese can be trained.

[0153] Alternatively, if you want to implement a text encoder for multiple language types, such as a Chinese-English bilingual text encoder, you can replace some of the second language sample texts with the corresponding first language sample texts during training.

[0154] For example, the text to be replaced is sampled from the second language sample text (e.g., 1 / 2 or 1 / 3 of the second language sample text is sampled). When the text to be replaced, which is the second language sample text, is input into the initial text encoder, it is replaced with the corresponding replacement language sample text (e.g., the corresponding first language sample text) and input into the initial text encoder for training. This allows the second language type of the sample text used to train the initial text encoder to also include the first language type of the first language sample text, thereby obtaining a bilingual text encoder.

[0155] S203. When the second language sample text is encoded by the initial text encoder to obtain the second text encoding features, the first text encoding features obtained by encoding the first language sample text by the reference text encoder are acquired.

[0156] Optionally, as described above, the second language sample text in the training samples can be used to sample the text to be replaced. For example, a random sampler can be set up to sample the second language sample text when the second language sample text is encoded by the initial text encoder to obtain the second text encoding features, in order to determine whether replacement is needed.

[0157] In the case where the second language sample text is the sampled text to be replaced, the text to be replaced is processed based on the first language sample text to obtain the replacement language sample text. At this time, the replacement language sample text can be encoded by the initial text encoder to obtain the replacement text encoding features, and the replacement text encoding features are used as the second text encoding features.

[0158] Alternatively, when the second language sample text is not the text to be replaced, the text encoding features obtained by text encoding the second language sample text through the initial text encoder can be used as the second text encoding features.

[0159] For example, if there are multiple second language sample texts used to train the initial text encoder, a portion of the second language sample texts can be randomly sampled from these multiple second language sample texts, and text replacement processing can be performed on this portion of the second language sample texts.

[0160] Alternatively, the second language sample text and the first language sample text can be input into a random sampler, which then samples from the second language sample text and the first language sample text. When the sampled text is the first language sample text, it means that the input second language sample text is the text to be replaced. In this case, the first language sample text can be used to replace the second language sample text.

[0161] When the sampled text is a second language sample text, it means that the input second language sample text is not the text to be replaced. In this case, it is necessary to perform text encoding on the second language sample text.

[0162] Optionally, the entire text content of the text to be replaced can be replaced, or only a portion of the text content can be replaced. For example, taking a second language sample text as an example, a portion of the text content in the second language sample text can be replaced with the corresponding first language sample text.

[0163] For example, after performing text segmentation on the first language sample text and the second language sample text, the text corresponding to a specified text (such as a noun) in the first language sample text is matched in the second language sample text, and the corresponding text in the second language sample text is replaced with the specified text in the first language sample text.

[0164] Specifically, it can be replacing a specified word (such as a noun) in the second - language sample text with the corresponding specified word (such as a noun) with the same meaning in the first - language sample text. For example, if the second - language sample text is "一只猫" and the first - language sample text is "a cat", at this time, the noun "猫" in the second - language sample text can be replaced with the corresponding content ("cat") in the first - language sample text "a cat", resulting in the updated second - language sample text "一只cat".

[0165] At this time, the second - language types of the second - language sample text include multiple types (that is, the second - language sample text includes multiple language types) to train a text encoder for multiple language types. Among them, optionally, the sample text input to the initial text encoder includes pure - Chinese sample text, pure - English sample text, and Chinese - English mixed sample text. At this time, the second - language types include Chinese and English, and a Chinese - English bilingual sample text is trained in this way.

[0166] Among them, when performing text replacement processing, the second - language sample text for full - text content replacement and / or the second - language sample text for partial - text content replacement can be sampled by a random sampler. For example, half of the text to be replaced is for full - text content replacement, and the other half of the text to be replaced is for partial - text content replacement.

[0167] Among them, when the text replacement processing is for full - content replacement, that is, the first - language sample text and the corresponding second - language sample text are input to the random sampler, and it samples in the first - language sample text and the second - language sample text. When the sampled text is the first - language sample text, the sampled first - language sample text is used as the replacement language sample text for text encoding; when the sampled text is the second - language sample text, the second - language sample text is encoded.

[0168] For example, as Figure 9 shown Figure 9 is a schematic diagram of an acquisition scenario of training samples provided by an embodiment of the present application; among them, the training samples for training the initial text encoder can include the first - language sample text (such as "a cat") and the second - language sample text (such as "一只猫") translated from the first - language sample text.

[0169] When encoding the second - language sample text, the second - language sample text and the first - language sample text are input to the random sampler. When the sampled text is the first - language sample text, the second - language sample text can be processed for text replacement by the first - language sample text, such as using the first - language sample text as the corresponding replacement language sample text.

[0170] At this point, after inputting the replacement language sample text into the second segmenter corresponding to the initial text encoder, the segmentation information output by the second segmenter is input into the initial text encoder to obtain the second text encoding feature of the second language sample text. This second text encoding feature is the replacement text encoding feature of the replacement language sample text.

[0171] When the sampled text is a second language sample text, the second language sample text is input into the second word segmenter corresponding to the initial text encoder, and the output of the second word segmenter is input into the initial text encoder to obtain the second text encoding feature of the second language sample text. This second text encoding feature is the encoding feature of the second language sample text.

[0172] S204. Using the first text encoding features, the second text encoding features, and the frozen training parameters, optimize the parameters in the initial text encoder except for the frozen training parameters to obtain the first optimized text encoder.

[0173] In this process, the parameters of the initial text encoder, excluding the frozen training parameters, can be optimized in stages (see the relevant description in the above embodiments for details). The first stage trains the initial feature embedding layer, and the second stage trains the parameters of the transition text encoder, excluding the second frozen training parameters.

[0174] At this point, text replacement processing can be used not only in the first stage but also in the second stage. For example, in the second stage, when the transitional text encoder trained in the first stage encodes the second language sample text and obtains the transitional text encoding features, text replacement processing can be performed on the second language sample text input to the transitional text encoder.

[0175] Optionally, for training samples used to train the initial text encoder, in order to improve the training effect, training samples with higher quality can be selected. The quality of these samples is determined by the feature similarity between the first language sample text and the second language sample text.

[0176] For example, the text encoding features of the second language sample text can be determined by a pre-trained text encoder (such as an open-source text encoder) for the second language sample text, so as to determine the sample quality between the first language sample text and the second language sample text together with the first text encoding features of the first language sample text.

[0177] At this point, no text replacement processing is performed when determining the sample quality (text replacement processing is only performed when the second language sample text needs to be input into the corresponding text encoder and the corresponding text encoder is trained by the output encoding result).

[0178] At this point, training samples with quality greater than a quality threshold can be selected from the training samples as high-quality training samples to train the initial text encoder.

[0179] For example, first language sample texts 1-10 are obtained, and each of them is translated to obtain second language sample texts 1-10. The first text encoding features of the first language sample texts 1-10 are determined using a reference text encoder, and the text encoding features to be processed in the second language sample texts 1-10 are determined using an open-source text encoder. The sample quality of the second language sample texts 1-10 is determined using the first text encoding features of the first language sample texts 1-10 and the text encoding features to be processed in the second language sample texts 1-10. Training samples are then selected based on sample quality; for example, second language sample texts 1-5 are selected. At this point, the first language sample texts 1-5, the second language sample texts 1-5, and the corresponding sample images 1-5 of the first language sample texts 1-5 can be used as high-quality training samples to train the initial text encoder (during the initial text encoder training, the aforementioned text replacement processing can be performed on the second language sample texts 1-5).

[0180] Alternatively, when optimizing the parameters of the initial text encoder (excluding the frozen training parameters) using the first text encoding features, the second text encoding features, and the frozen training parameters to obtain the optimized text encoder, the optimized text encoder can be used to select training samples.

[0181] For example, after training the initial text encoder with all training samples to obtain the optimized text encoder, the optimized text encoder and the reference text encoder can be used to filter the training samples, and the filtered training samples can be used to train the optimized text encoder again (the first and second training stages mentioned above) to obtain the final text encoder that is aligned with the reference text encoder.

[0182] For example, the first execution of steps S203-S204 yields a text encoder with optimized parameters, which is then called the first optimized text encoder. The first optimized text encoder and the reference text encoder can then be used to select training samples.

[0183] For example, by using the first optimized text encoder to encode the second language sample text to obtain optimized text encoding features (such as the first optimized text encoding feature, without performing the above text replacement process, directly encoding the second language sample text), the first optimized text encoding feature and the first text encoding feature can be used to select training samples.

[0184] S205. When encoding the second language sample text using the first optimized text encoding feature to obtain the first optimized text encoding feature, the text quality of the second language sample text corresponding to the first optimized text encoding feature is determined based on the feature similarity between the first optimized text encoding feature and the first text encoding feature.

[0185] Specifically, the feature similarity between the first optimized text encoding feature and the first text encoding feature can be determined as the text quality (sample quality) of the second language sample text corresponding to the first optimized text encoding feature.

[0186] S206. Based on text quality, select filtered language sample texts from the second language sample texts. Through the filtered language sample texts and the corresponding first language sample texts, iteratively train the first optimized text encoder to obtain the second optimized text encoder.

[0187] Among them, there are multiple second language sample texts. The U second language sample texts with the highest text quality (U is a positive integer greater than 1) (or the second language sample texts with text quality greater than a specified quality threshold) can be selected from the multiple second language sample texts as the filter language sample texts.

[0188] Specifically, the sample image of the first language sample text corresponding to the selected language sample text can be used as the sample image associated with the selected language sample text.

[0189] At this point, by filtering language sample texts and the first language sample texts corresponding to the filtered language sample texts, the first optimized text encoder can be iteratively trained to obtain the second optimized text encoder (the text encoder with optimized parameters in the second round).

[0190] That is, by filtering the language sample text and the first language sample text corresponding to the filtered language sample text, the above steps S203-S204 are executed again.

[0191] Specifically, when text encoding the selected language sample text through the first optimized text encoder to obtain the text encoding features of the selected language sample text (optionally, the text replacement processing mentioned above can also be performed at this time), the parameters in the first optimized text encoder other than the frozen training parameters are optimized by using the first text encoding features of the first language sample text corresponding to the selected language sample text, the text encoding features of the selected language sample text, and the frozen training parameters, to obtain the second optimized text encoder.

[0192] Similarly, the first optimized text encoder can be trained in stages. For example, in the first stage, the feature embedding layer in the first optimized text encoder is trained (at this time, the training parameters are frozen as the feature encoding layer in the first optimized text encoder, and the parameters to be trained are the feature embedding layer in the first optimized text encoder).

[0193] For example, in the second stage, the feature embedding layer and some feature coding layers in the first optimized text encoder are trained (at this time, the training parameters are frozen as the coding layers in the feature coding layer of the first optimized text encoder except for the first j feature coding sub-layers, and the parameters to be trained are the feature embedding layer and the feature coding layer in the first optimized text encoder except for the first j feature coding sub-layers).

[0194] Optionally, at this point, the second optimized text encoder obtained from the above training can be used to filter and retrain the training samples again, or the alignment with the reference image encoder can be selected.

[0195] S207. When the second optimized text encoder is used to encode the selected language sample text to obtain the second optimized text encoding features of the selected language sample text, the image encoding features are obtained by using the reference image encoder associated with the reference text encoder to encode the sample image associated with the selected language sample text.

[0196] Specifically, when training the second optimized text encoder using the reference image encoder, the optimized text encoding features (second optimized text encoding features) of the selected language sample text can be determined by the second optimized text encoder, and the image encoding features of the sample image (sample image of the first language sample text corresponding to the selected language sample text) associated with the selected language sample text can be determined by the reference image encoder.

[0197] Optionally, when determining the optimized text encoding features, the text replacement process described above can be performed on the selected language sample text, or it can be omitted.

[0198] S208. Using image encoding features and second optimized text encoding features, iteratively train the second optimized text encoding features to obtain the target text encoder.

[0199] Among them, image coding features can be used as supervision signals, and the second optimized text coding features can be iteratively trained.

[0200] Optionally, since the reference text encoder and the second optimized text encoder are aligned, a small number of training samples can be selected for the alignment of the reference image encoder and the second optimized text encoder, thereby reducing the amount of sample data used during training.

[0201] For example, select the V (V is a positive integer less than U) second language sample texts (or second language sample texts with text quality greater than a specified quality threshold) with the highest text quality (optionally, it can refer to the text quality determined by the second optimized text encoder; or it can refer to the text quality re-determined by the second optimized text encoder, in which case the above text replacement process will not be performed) among the language sample texts as the target language sample texts.

[0202] At this point, the second optimized text encoding features of the target filtered language sample text can be obtained solely through the second optimized text encoder, and the image encoding features of the sample image associated with the target filtered language sample text can be obtained through the reference image encoder.

[0203] Therefore, the target text encoder can be obtained by iteratively training the second optimized text encoder using the image encoding features of the sample images associated with the target selected language sample text and the second optimized text encoding features of the target selected language sample text.

[0204] Optionally, other open-source datasets (such as sample texts of the second language type and their corresponding sample images, optionally without performing text replacement processing) can be obtained together with the target language sample text to iteratively train the second optimized text encoder, thereby enriching the data source of the training samples.

[0205] For example, such as Figure 10 As shown, Figure 10 This is a schematic diagram of an optimized training scenario provided by an embodiment of this application; wherein, the initial text encoder 101a includes an initial feature embedding layer and an initial feature encoding layer; the reference text encoder 102a includes a reference feature embedding layer and a reference feature encoding layer. The reference feature embedding layer is parameter-adjusted (e.g., parameter expansion) to obtain a parameter-adjusted reference feature embedding layer (e.g., a parameter-expanded reference feature embedding layer), which is used as the initial feature embedding layer, and the reference feature encoding layer is used as the initial feature encoding layer.

[0206] The training samples are obtained, including first language sample text 103a (such as text a11, text a12, ..., text a1n) and corresponding translated second language sample text 103b (such as text a21, text a22, ..., text a2n, one first language sample text corresponds to one second language sample text). The initial text encoder is trained using the first language sample text 103a and the second language sample text 103b to obtain the first optimized text encoder 101b.

[0207] During training, a random sampler can be used to perform text replacement on the second language sample text.

[0208] At this point, the first optimized text encoding features of the second language sample text (such as text a21) can be determined by the first optimized text encoder, and the text quality of text a21 can be determined by the feature similarity between the first text encoding features of the first language sample text (such as text a11 corresponding to text a21) obtained by the reference text encoder and the first optimized text encoding features, until the text quality of each second language sample text is obtained.

[0209] Specifically, based on text quality, selected language sample texts 104a (such as text a21, text a22, ..., text a210) are selected from the second language sample texts 103b. The first optimized text encoder is trained using the selected language sample texts and the corresponding first language sample texts 104b (such as text a11, text a12, ..., text a110) to obtain the second optimized text encoder 101c.

[0210] During training, a random sampler can be used to perform text replacement on the selected language sample text.

[0211] Specifically, target language sample texts 105a (such as text a21, text a22, ..., text a25) can be selected from the selected language sample texts 104a based on text quality. The second optimized text encoder 101c is trained using the target selected language sample texts 105a and the sample images 105b associated with the target selected language sample texts 105a (such as sample image b11 of text a11 corresponding to text a21, sample image b12 of text a12 corresponding to text a22, ..., sample image b15 of text a15 corresponding to text a25), resulting in the target text encoder 101d.

[0212] During training, a random sampler can be used to perform text replacement on the target language sample text.

[0213] The target text encoder is used to predict at least the target image corresponding to the output target language text, and the target language text and the second language sample text have the same language type.

[0214] For example, a target text encoder can encode the target language text. The second language type of this second language sample text includes Chinese and English, so the target language text can be purely Chinese, purely English, or a mixture of Chinese and English.

[0215] For example, such as Figure 11 As shown, Figure 11A schematic diagram of the process of model training provided by an embodiment of this application; among them, the first language sample text 111a (such as "a smiling boy") is obtained, and the translated text obtained by translating the first language sample text is the second language sample text 111b corresponding to the first language sample text (such as "一个笑着的男孩"), the first language sample text is input into the first tokenizer (such as the English tokenizer 112a), and the text information of the first language sample text is input into the reference text encoder (such as the English encoder 113a), and the encoded feature 114a (such as the first text encoded feature of the first language sample text) is output.

[0216] Among them, the second language sample text is input into the second tokenizer (such as the Chinese tokenizer 112b), and the text information of the second language sample text is input into the initial text encoder (such as the Chinese encoder 113b), and the encoded feature 114b (such as the second text encoded feature of the second language sample text) is output.

[0217] Among them, the initial feature embedding layer u21 in the initial text encoder refers to the feature embedding layer obtained by parameter expansion of the reference feature embedding layer u11 in the reference text encoder. The initial feature encoding layer u22 in the initial text encoder inherits the parameters of the reference feature encoding layer u12 in the reference text encoder.

[0218] Optionally, when determining the second text encoded feature, before the second language sample text is input into the second tokenizer, the text to be replaced can be determined by a random sampler, and text replacement processing is performed on the text to be replaced to obtain a replacement language sample text, so as to be input into the second tokenizer, and the replacement text encoded feature output by the initial text encoder is used as the second text encoded feature.

[0219] For example, the first language sample text and the second language sample text are input into the random sampler 115a, and the sampled text is input into the second tokenizer, and then input into the initial text encoder, and the encoded feature output by the initial text encoder is used as the second text encoded feature.

[0220] Among them, the initial text encoder can be trained by the first text encoded feature and the second text encoded feature.

[0221] Among them, the first stage is to train the initial feature embedding layer to obtain a transitional feature embedding layer, and at this time, the parameters in the initial feature encoding layer are frozen. The second stage is to train a part of the encoding layers in the transitional feature embedding layer and the initial feature encoding layer to obtain a transitional feature embedding layer, and at this time, the parameters of another part of the encoding layers in the initial feature encoding layer are still frozen.

[0222] In the process of obtaining the first optimized text encoder through the first and second stages of training, a similarity determination network can be used to filter second language sample texts. This similarity determination network includes a reference text encoder and the first optimized text encoder.

[0223] The second language sample text can be input into the first optimized text encoder to output encoding features 114b (such as the first optimized text encoding features of the second language sample text). The text quality (i.e. translation quality, translation accuracy can be used to measure sample quality) of the second language sample text can be determined by the feature similarity between the first optimized text encoding features and the first text encoding features of the first language sample text obtained by the reference text encoder, so as to select the selected language sample text.

[0224] Specifically, a second optimized text encoder can be obtained by filtering language sample texts and the first language sample texts corresponding to the filtered language sample texts and then retraining the first optimized text encoder (by repeating the training of the first and second stages mentioned above).

[0225] Optionally, a random sampler can also be used when training the first optimized text encoder. For example, before the selected language sample text is input into the second segmenter of the second optimized text encoder, the text to be replaced can be determined by a random sampler, and the text to be replaced can be processed to obtain the replaced language sample text, which is then input into the second segmenter. The replaced text encoding features output by the second optimized text encoder are used as the second optimized text encoding features of the selected language sample text. Thus, the first optimized text encoder can be trained using the second optimized text encoding features and the first text encoding features of the first language sample text corresponding to the selected language sample text.

[0226] Specifically, a target language sample text can be selected from the selected language sample text based on text quality, and a second optimized text encoder can be trained using the sample images of the target language sample text and the first language sample text corresponding to the target language sample text.

[0227] For example, the word segmentation information of the target language sample text is input into the second optimized text encoder (using a random sampler) to output encoding features 114b (such as the second optimized text encoding features of the target language sample text), and the image content information obtained by image preprocessing the sample image 111c of the first language sample text corresponding to the target language sample text is input into the reference image encoder to obtain image encoding features 114c. The second optimized text encoder is trained using the second optimized text encoding features and the image encoding features to obtain the target text encoder.

[0228] Among them, the parameters in the reference text encoder and the reference image encoder are frozen.

[0229] The reference text encoder and the target text encoder (or the second optimized text encoder) can be used as a new similarity determination network to determine the similarity between two texts.

[0230] For example, given a text (Text 1) in the first language and a text (Text 2) translated into the second language, we can input the word segmentation information of Text 1 into the reference text encoder of the similarity determination network to obtain the text encoding features of Text 1. Similarly, we can input the word segmentation information of Text 2 into the target text encoder of the similarity determination network to obtain the text encoding features of Text 2. At this point, we can determine the feature similarity between the text encoding features of Text 1 and Text 2. This feature similarity can be considered as the semantic similarity between Text 1 and Text 2, that is, the translation quality (translation accuracy) of Text 2 relative to Text 1.

[0231] This invention relates to a contrastive language-image pre-trained model (specifically, training the text encoder within it) for image and text matching. This model generates a Chinese CLIP model through three-stage training, for example, based on the English CLIP model and fine-tuned on billions of data points. The model not only supports Chinese but also enables bilingual (Chinese and English) support, and demonstrates significant performance improvements on multiple business datasets.

[0232] The trained target text encoder can be combined with the reference image encoder to form a second language type image-text processing model. This second language type image-text processing model can be used for image and second language type text matching. The target text encoder is trained in three stages, based on a high-performance open-source image-text processing model, and fine-tuned on a certain training sample to obtain a text encoder that can be used for multiple language types.

[0233] The technical solution of this application adopts an independent image encoder and text encoder structure, combined with knowledge distillation and a phased training strategy. For example, in the first and second stages, only the reference text encoder is used for supervised training, and then in the third stage, the reference image encoder is used for supervision and fine-tuning to further improve performance. This can reduce the amount of training sample data, save a lot of disk space occupied by image and text data, and reduce training costs.

[0234] The trained target text encoder structurally inherits from the reference text encoder and inherits some of its parameters (achieving fast convergence). Optionally, the reference feature embedding layer (which increases the number of parameters) can be extended to adapt the initial feature embedding layer to a Chinese word segmenter.

[0235] Among them, training tests were conducted on different versions (such as basic version, large-scale version, and massive-scale version) of the initial text encoder. It was found that when using certain training resources and without enabling any acceleration methods, the training cost and training time can be effectively reduced compared to full-parameter training.

[0236] Furthermore, by testing the results of various evaluation tasks (such as cross-modal image and text detection for Chinese, zero-shot classification for Chinese, and zero-shot classification for English) using the image and text processing model obtained by the target text encoder, it was found that the model outperforms other open-source image and text processing models on various evaluation tasks, and shows significant performance improvement on multiple business datasets.

[0237] For example, the task accuracy (in %) results for different evaluation tasks are as follows:

[0238]

[0239] Table 1

[0240] In this application, Model 11 represents the base version of the open-source Chinese image and text processing model, Model 12 represents the large-scale version of the open-source Chinese image and text processing model, and Model 13 represents the huge-scale version of the open-source Chinese image and text processing model; Model 21 represents the base version of the Chinese image and text processing model in this application, Model 22 represents the large-scale version of the Chinese image and text processing model in this application, and Model 23 represents the huge-scale version of the Chinese image and text processing model in this application; Model 31 represents the base version of the Chinese-English bilingual image and text processing model in this application, Model 32 represents the large-scale version of the Chinese-English bilingual image and text processing model in this application, and Model 33 represents the huge-scale version of the Chinese-English bilingual image and text processing model in this application.

[0241] In this embodiment, knowledge distillation can be used to quickly enable the target text encoder to learn the capabilities of the reference text encoder and adapt to the reference image encoding. Furthermore, during training, phased model training can reduce the training difficulty. By inheriting some parameters from the reference text encoder (i.e., freezing training parameters) during training, the amount of sample data used can be reduced. In addition, the text encoder obtained from the first training can be used to select samples, and the selected high-quality samples can be used to train the text encoder again, which can effectively reduce training costs and further improve training results.

[0242] For further details, please see Figure 12 , Figure 12 This is a schematic diagram of the structure of a text processing device provided in an embodiment of this application. Figure 12 As shown, the text processing device 1 can be applied to a computer device. It should be understood that the text processing device 1 can be a computer program (including program code) running on the computer device; for example, the text processing device 1 can be an application software. It is understood that the text processing device 1 can be used to perform the corresponding steps in the methods provided in the embodiments of this application. Figure 12 As shown, the text processing device 1 may include: a sample acquisition module 11, a feature encoding module 12, and a model training module 13. Wherein:

[0243] The sample acquisition module 11 is used to acquire the first language sample text for training the initial text encoder, the second language sample text translated from the first language sample text, and the sample image corresponding to the first language sample text; the parameters in the initial text encoder include frozen training parameters, which are derived from the reference text encoder associated with the first language sample text.

[0244] The feature encoding module 12 is used to obtain the first text encoding features obtained by text encoding the first language sample text through the reference text encoder when the second text encoding features are obtained by text encoding the second language sample text through the initial text encoder.

[0245] The model training module 13 is used to optimize the parameters of the initial text encoder, excluding the frozen training parameters, using the first text encoding features, the second text encoding features, and the frozen training parameters, to obtain the optimized text encoder.

[0246] The feature encoding module 12 is also used to obtain the image encoding features obtained by encoding the sample image through the reference image encoder associated with the reference text encoder when the second language sample text is encoded by the text encoder with optimized parameters to obtain optimized text encoding features.

[0247] The model training module 13 is also used to iteratively train the optimized text encoder using image encoding features and optimized text encoding features to obtain the target text encoder; the target text encoder is used at least to predict the target image corresponding to the output target language text, and the target language text has the same language type as the second language sample text.

[0248] The initial text encoder includes an initial feature embedding layer and an initial feature encoding layer; the parameters in the initial feature encoding layer are derived from the reference text encoder; the frozen training parameters in the initial text encoder are the first frozen training parameters.

[0249] Model training module 13 includes:

[0250] The parameter freezing determination unit 131 is used to use the parameters in the initial feature encoding layer as the first frozen training parameters;

[0251] The first parameter optimization unit 132 is used to keep the first frozen training parameters fixed and optimize the parameters in the initial feature embedding layer through the first text encoding feature and the second text encoding feature to obtain the transition feature embedding layer.

[0252] The first parameter optimization unit 132 is also used to use the initial text encoder containing the transition feature embedding layer and the initial feature encoding layer as the transition text encoder.

[0253] The second reference optimization unit 133 is used to optimize the parameters in the transition text encoder by using the first text encoding feature and the transition text encoding feature when the second language sample text is encoded by the transition text encoder to obtain the transition text encoding feature, so as to obtain the optimized text encoder.

[0254] Wherein, the first frozen training parameters include the second frozen training parameters; the initial feature encoding layer includes J feature encoding sub-layers; the parameters of the feature encoding sub-layers other than the first j feature encoding sub-layers are the second frozen training parameters; J is a positive integer greater than 1, and j is a positive integer less than J;

[0255] The second reference optimization unit 133 is specifically used for:

[0256] Keeping the second frozen training parameters fixed, the parameters in the transition feature embedding layer and the parameters of the first j feature encoding sub-layers in the initial feature encoding layer are optimized using the first text encoding features and the transition text encoding features to obtain the optimized feature embedding layer and the optimized feature encoding layer.

[0257] The text encoder is optimized by using the transitional text encoder, which includes an optimized feature embedding layer and an optimized feature encoding layer, as parameters.

[0258] The first language sample text is segmented by the first text segmenter corresponding to the reference text encoder and then encoded by the initial text encoder; the second language sample text is segmented by the second text segmenter corresponding to the initial text encoder and then encoded by the initial text encoder; the initial text encoder includes an initial feature embedding layer and an initial feature encoding layer; the reference text encoder includes a reference feature embedding layer and a reference feature encoding layer.

[0259] Feature encoding module 12 also includes:

[0260] The encoder determining unit 121 is used to perform parameter expansion on the reference feature embedding layer when the vocabulary information of the first text segmenter is less than that of the second text segmenter, to obtain a parameter-expanded reference feature embedding layer; the number of parameters in the parameter-expanded reference feature embedding layer is greater than the number of parameters in the reference feature embedding layer.

[0261] The encoder determination unit 121 is also used to determine the initial feature embedding layer based on the reference feature embedding layer after parameter expansion, and to determine the initial feature coding layer based on the reference feature coding layer.

[0262] The feature encoding module 12 includes:

[0263] The text replacement unit 122 is used to perform text replacement processing on the text to be replaced based on the first language sample text when the second language sample text is the text to be replaced, so as to obtain the replacement language sample text.

[0264] The text replacement unit 122 is also used to perform text encoding on the replacement language sample text through the initial text encoder to obtain the replacement text encoding features, and to use the replacement text encoding features as the second text encoding features;

[0265] The text encoding unit 123 is used to use the text encoding features obtained by text encoding the second language sample text through the initial text encoder as the second text encoding features when the second language sample text is not the text to be replaced.

[0266] Among them, the text encoder with optimized parameters is the first optimized text encoder; the optimized text encoding features obtained by passing the second language sample text through the first optimized text encoder are the first optimized text encoding features;

[0267] Feature encoding module 12 includes:

[0268] The text filtering module 124 is used to determine the text quality of the second language sample text corresponding to the first optimized text encoding feature based on the feature similarity between the first optimized text encoding feature and the first text encoding feature;

[0269] The text filtering module 124 is also used to select the U second language sample texts with the highest text quality from the second language sample texts as the filtered language sample texts, and to use the sample image of the first language sample text corresponding to the filtered language sample text as the sample image associated with the filtered language sample text; U is a positive integer greater than 1;

[0270] The model training unit 125 is used to iteratively train the first optimized text encoder by filtering language sample text and the first language sample text corresponding to the filtered language sample text to obtain the second optimized text encoder.

[0271] The text encoding unit 123 is also used to obtain the image encoding features of the sample image associated with the selected language sample text by performing image encoding on the sample image associated with the selected language sample text through the reference image encoder associated with the reference text encoder when the selected language sample text is text encoded by the second optimized text encoder to obtain the second optimized text encoding features of the selected language sample text.

[0272] Among them, the V second language sample texts with the highest text quality in the selected language sample texts are the target selected language sample texts; V is a positive integer less than U;

[0273] Model training module 13 is specifically used for:

[0274] The second optimized text encoder is iteratively trained using the image encoding features of the sample images associated with the target selected language sample text and the second optimized text encoding features of the target selected language sample text to obtain the target text encoder.

[0275] The text processing device 1 further includes:

[0276] Image and text retrieval module 14 is used to obtain the target language text to be retrieved;

[0277] The image and text retrieval module 14 is also used to obtain the image encoding features obtained by encoding the target language text in the image library by the reference image encoder when the target text encoding features are obtained by text encoding the target language text by the target text encoder.

[0278] The image retrieval module 14 is also used to select the target image corresponding to the target language text from the image library based on the feature similarity between the target text encoding features and the image encoding features of the target image.

[0279] The text processing device 1 further includes:

[0280] The image and text retrieval module 14 is also used to obtain the target image and text library to be retrieved;

[0281] The image and text retrieval module 14 is also used to obtain the image encoding features obtained by encoding the target image by the reference image encoder when the target language text in the text library is encoded by the target text encoder to obtain the target text encoding features.

[0282] The image and text retrieval module 14 is also used to select the target language text corresponding to the target image from the text library based on the feature similarity between the target text encoding features and the image encoding features of the target image.

[0283] The specific implementation methods of the sample acquisition module 11, feature encoding module 12, model training module 13, and image / text retrieval module 14 can be found in the relevant descriptions in the above embodiments, and will not be repeated here. It should be understood that the beneficial effects obtained using the same method will also not be repeated here.

[0284] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0285] Further, please see Figure 13 , Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 13 As shown, the computer device 1400 can be a business terminal or a server; this is not a limitation. For ease of understanding, this application takes a computer device as a server as an example. The computer device 1400 may include: a processor 1401, a network interface 1404, and a memory 1405. Furthermore, the computer device 1400 may also include: a user interface 1403, and at least one communication bus 1402. The communication bus 1402 is used to implement communication between these components. The user interface 1403 may also include a standard wired interface and a wireless interface. The network interface 1404 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface). The memory 1405 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 1405 may also be at least one storage device located remotely from the aforementioned processor 1401. Figure 13 As shown, the memory 1405, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0286] The network interface 1404 in the computer device 1400 can also provide network data interaction functionality. Figure 13 In the computer device 1400 shown, the network interface 1404 provides network data interaction functionality; the user interface 1403 is mainly used to provide an input interface for the user; and the processor 1401 can be used to call the device control application stored in the memory 1405 to execute the above-mentioned functions. Figure 3 , Figure 8 The text processing method described in the corresponding embodiments can also be executed as described above. Figure 12 The description of the text processing device 1 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated here.

[0287] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium, which stores a computer program executed by the aforementioned text processing device 1. The computer program includes a computer program that, when executed by a processor, can execute the aforementioned text... Figure 3 , Figure 8 The description of the text processing method in the corresponding embodiments is already provided and will not be repeated here. Similarly, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application. As an example, a computer program can be deployed and executed on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network. These multiple computing devices distributed across multiple locations and interconnected via a communication network can constitute a blockchain system.

[0288] Furthermore, it should be noted that this application also provides a computer program product, which includes a computer program that can be stored in a computer-readable storage medium. The processor of a computer device reads the computer program from the computer-readable storage medium, and the processor can execute the computer program, causing the computer device to perform the aforementioned... Figure 3 , Figure 8 The description of the text processing method in the corresponding embodiments is already provided and will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer program product embodiments related to this application, please refer to the description of the method embodiments of this application.

[0289] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0290] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0291] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0292] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0293] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A text processing method, characterized in that, The method includes: The method involves acquiring a first language sample text for training an initial text encoder, a second language sample text translated from the first language sample text, and a sample image corresponding to the first language sample text. The parameters in the initial text encoder include frozen training parameters, which are derived from a reference text encoder associated with the first language sample text. When the second language sample text is encoded by the initial text encoder to obtain the second text encoding features, the first text encoding features obtained by encoding the first language sample text by the reference text encoder are also acquired. Using the first text encoding feature, the second text encoding feature, and the frozen training parameters, the parameters in the initial text encoder other than the frozen training parameters are optimized to obtain the optimized text encoder. When the second language sample text is text encoded by the optimized text encoder using the parameters, and the optimized text encoding features are obtained, the image encoding features obtained by the reference image encoder associated with the reference text encoder are acquired. The optimized text encoder is iteratively trained using the image encoding features and the optimized text encoding features to obtain the target text encoder. The target text encoder is used at least to predict the target image corresponding to the output target language text, and the target language text has the same language type as the second language sample text.

2. The method according to claim 1, characterized in that, The initial text encoder includes an initial feature embedding layer and an initial feature encoding layer; the parameters in the initial feature encoding layer are derived from the reference text encoder; the frozen training parameters in the initial text encoder are the first frozen training parameters; The step of optimizing the parameters of the initial text encoder, excluding the frozen training parameters, using the first text encoding features, the second text encoding features, and the frozen training parameters to obtain a parameter-optimized text encoder includes: The parameters in the initial feature encoding layer are used as the first frozen training parameters; Keeping the first frozen training parameters fixed, the parameters in the initial feature embedding layer are optimized using the first text encoding feature and the second text encoding feature to obtain the transition feature embedding layer; The initial text encoder containing the transition feature embedding layer and the initial feature encoding layer is used as the transition text encoder; When the second language sample text is encoded by the transition text encoder to obtain transition text encoding features, the parameters in the transition text encoder are optimized using the first text encoding features and the transition text encoding features to obtain a parameter-optimized text encoder.

3. The method according to claim 2, characterized in that, The first frozen training parameters include the second frozen training parameters; the initial feature encoding layer includes J feature encoding sub-layers; the parameters of the feature encoding sub-layers other than the first j feature encoding sub-layers are the second frozen training parameters; J is a positive integer greater than 1, and j is a positive integer less than J; The step of optimizing the parameters in the transition text encoder using the first text encoding feature and the transition text encoding feature to obtain a parameter-optimized text encoder includes: Keeping the second frozen training parameters fixed, the parameters in the transition feature embedding layer and the parameters of the first j feature encoding sub-layers in the initial feature encoding layer are optimized using the first text encoding feature and the transition text encoding feature to obtain the optimized feature embedding layer and the optimized feature encoding layer. The transitional text encoder containing the optimized feature embedding layer and the optimized feature encoding layer is used as the text encoder optimized by the parameters.

4. The method according to claim 1, characterized in that, The first language sample text is segmented by the first text segmenter corresponding to the reference text encoder and then encoded by the initial text encoder; the second language sample text is segmented by the second text segmenter corresponding to the initial text encoder and then encoded by the initial text encoder; the initial text encoder includes an initial feature embedding layer and an initial feature encoding layer; The reference text encoder includes a reference feature embedding layer and a reference feature encoding layer; The method further includes: When the vocabulary information of the first text segmenter is less than that of the second text segmenter, the reference feature embedding layer is extended to obtain a parameter-extended reference feature embedding layer; the number of parameters in the parameter-extended reference feature embedding layer is greater than the number of parameters in the reference feature embedding layer. The initial feature embedding layer is determined based on the reference feature embedding layer after parameter expansion, and the initial feature coding layer is determined based on the reference feature coding layer.

5. The method according to claim 1, characterized in that, The step of encoding the second language sample text using the initial text encoder to obtain the second text encoding features includes: When the second language sample text is the text to be replaced, the text to be replaced is replaced based on the first language sample text to obtain the replacement language sample text; The replacement language sample text is text encoded by the initial text encoder to obtain replacement text encoding features, and the replacement text encoding features are used as the second text encoding features; When the second language sample text is not the text to be replaced, the text encoding feature obtained by text encoding the second language sample text through the initial text encoder is used as the second text encoding feature.

6. The method according to claim 1, characterized in that, The optimized text encoder is the first optimized text encoder; the optimized text encoding features obtained by passing the second language sample text through the first optimized text encoder are the first optimized text encoding features. The step of obtaining the image encoding features obtained by encoding the sample image through the reference image encoder associated with the reference text encoder includes: The text quality of the second language sample text corresponding to the first optimized text encoding feature is determined based on the feature similarity between the first optimized text encoding feature and the first text encoding feature. The U second language sample texts with the highest text quality are selected from the second language sample texts and used as the filter language sample texts. The sample images of the first language sample texts corresponding to the filter language sample texts are used as the sample images associated with the filter language sample texts; U is a positive integer greater than 1. The first optimized text encoder is iteratively trained using the filtered language sample text and the first language sample text corresponding to the filtered language sample text to obtain the second optimized text encoder. When the selected language sample text is text encoded by the second optimized text encoder to obtain the second optimized text encoding features of the selected language sample text, the image encoding features are obtained by image encoding of the sample image associated with the selected language sample text through the reference image encoder associated with the reference text encoder.

7. The method according to claim 6, characterized in that, The V second language sample texts with the highest text quality among the selected language sample texts are the target selected language sample texts; V is a positive integer less than U; The step of iteratively training the optimized text encoder using the image encoding features and the optimized text encoding features to obtain the target text encoder includes: The second optimized text encoder is iteratively trained using the image encoding features of the sample image associated with the target selected language sample text and the second optimized text encoding features of the target selected language sample text to obtain the target text encoder.

8. The method according to claim 1, characterized in that, The method further includes: Retrieve the target language text to be searched; When the target language text is encoded by the target text encoder to obtain the target text encoding features, the image encoding features obtained by encoding the target image in the image library by the reference image encoder are also acquired. Based on the feature similarity between the target text encoding features and the image encoding features of the target image, the target image corresponding to the target language text is selected from the image library.

9. The method according to claim 1, characterized in that, The method further includes: Obtain the target image and text database to be searched; When the target language text in the text library is encoded by the target text encoder to obtain the target text encoding features, the image encoding features obtained by encoding the target image by the reference image encoder are also acquired. Based on the feature similarity between the target text encoding features and the image encoding features of the target image, the target language text corresponding to the target image is selected from the text library.

10. A text processing device, characterized in that, The device includes: The sample acquisition module is used to acquire a first language sample text for training the initial text encoder, a second language sample text translated from the first language sample text, and a sample image corresponding to the first language sample text; the parameters in the initial text encoder include frozen training parameters, which are derived from a reference text encoder associated with the first language sample text; The feature encoding module is used to obtain the first text encoding feature obtained by text encoding the first language sample text through the reference text encoder when the second language sample text is text encoded by the initial text encoder to obtain the second text encoding feature; The model training module is used to optimize the parameters of the initial text encoder, excluding the frozen training parameters, using the first text encoding features, the second text encoding features, and the frozen training parameters, to obtain a parameter-optimized text encoder. The feature encoding module is further configured to, when performing text encoding on the second language sample text through the text encoder optimized by the parameters to obtain optimized text encoding features, acquire the image encoding features obtained by performing image encoding on the sample image through the reference image encoder associated with the reference text encoder; The model training module is further configured to iteratively train the parameter-optimized text encoder using the image encoding features and the optimized text encoding features to obtain a target text encoder; the target text encoder is at least used to predict the target image corresponding to the output target language text, and the target language text has the same language type as the second language sample text.

11. A computer device, characterized in that, Including memory and processor; The memory is connected to the processor, the memory is used to store computer programs, and the processor is used to invoke the computer programs so that the computer device performs the method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1-9.

13. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.

Images