Model pre-training method and device for multi-language task

Abstract

The invention provides a model pre-training method and device for a multi-language task, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a multi-modal training data set which comprises a plurality of pieces of sample text data and a plurality of pieces of sample image-text pair data with multi-language contents, and covers a universal field and a preset business field; based on a plurality of sample image-text pair data, performing comparative learning training of alignment of visual features and text features on the initial model, freezing model parameters of a decoding module and adjusting model parameters of a visual encoder and a projection module in the training process until a first ending condition is met; based on the multiple pieces of sample image-text pair data and the multiple pieces of sample text data, constraint training of content understanding is carried out on the initial model meeting a first ending condition, and model parameters of a visual encoder, a projection module and a decoding module are adjusted in the training process until a second ending condition is met to obtain a target model; the information extraction capability of the model in the specific field can be remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model pre-training method and device for multi-language tasks. Background Art

[0002] The advancement of globalization and digitalization has made multilingual and multimodal tasks increasingly important. Although pre-training models have achieved outstanding results in the field of natural language processing, there are still many unresolved problems in multilingual and multimodal tasks. At present, in the field of multimodal pre-training, data support is mainly concentrated in English, resulting in poor adaptability of existing multimodal models in the field of visual text understanding that includes multiple languages ​​(such as Chinese, German, French, Japanese, etc.). This challenge is more prominent in specific business areas in different scenarios. It not only involves multilingual issues, but also has to deal with the differences between document images and natural images, making multimodal models unable to adapt to task processing in the field. Summary of the invention

[0003] The present application provides a model pre-training method and device for multilingual tasks, which can significantly improve the model pre-training effect for multilingual tasks and improve the task processing effect of the model in a preset business field.

[0004] On the one hand, the present application provides a model pre-training method for multilingual tasks, the method comprising: Acquire a multimodal training data set and an initial model, wherein the training data set includes a plurality of sample text data and a plurality of sample image-text pair data, wherein the plurality of sample image-text pair data and the plurality of sample text data include content in multiple languages, and the plurality of sample image-text pair data include sample image-text pair data in a general field and sample image-text pair data in a preset business field in a target scene, wherein the plurality of sample text data include text data in the preset business field, and the initial model includes a visual encoder, a projection module, and a decoding module connected in sequence, wherein the decoding module is constructed based on a large language model; Based on the plurality of sample image-text pair data, performing comparative learning training of aligning visual features and text features on the initial model, freezing the model parameters of the decoding module and adjusting the model parameters of the visual encoder and the projection module during the training process until a first end condition is met; Based on the multiple sample image-text pair data and the multiple sample text data, the initial model that meets the first end condition is subjected to constrained training for content understanding, and during the training process, the model parameters of the visual encoder, the projection module, and the decoding module are adjusted until the second end condition is met; The initial model that meets the second end condition is determined as the target model; during the training process, the text in the sample image-text data and the sample text data of the preset business field are segmented in combination with the target vocabulary corresponding to the preset business field to serve as the input of the projection module.

[0005] On the other hand, a model pre-training device for multi-language tasks is provided, the device comprising: Acquisition module: used to acquire a multimodal training data set and an initial model, wherein the training data set includes a plurality of sample text data and a plurality of sample image-text pair data, wherein the plurality of sample image-text pair data and the plurality of sample text data include content in multiple languages, and the plurality of sample image-text pair data include sample image-text pair data in a general field and sample image-text pair data in a preset business field in a target scene, wherein the plurality of sample text data include text data in the preset business field, and the initial model includes a visual encoder, a projection module and a decoding module connected in sequence, wherein the decoding module is constructed based on a large language model; A first training module: used for performing comparative learning training of aligning visual features and text features on the initial model based on the plurality of sample image-text pair data, freezing the model parameters of the decoding module and adjusting the model parameters of the visual encoder and the projection module during the training process until a first end condition is met; A second training module: used for performing content understanding constraint training on the initial model that meets the first end condition based on the multiple sample image-text pair data and the multiple sample text data, and adjusting the model parameters of the visual encoder, the projection module and the decoding module during the training process until the second end condition is met; Model generation module: used to determine the initial model that meets the second end condition as the target model; during the training process, the text and sample text data in the sample image and text data of the preset business field are segmented in combination with the target vocabulary corresponding to the preset business field to serve as the input of the projection module.

[0006] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the model pre-training method for multilingual tasks as described above.

[0007] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the model pre-training method for multilingual tasks as described above.

[0008] On the other hand, a server is provided, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the model pre-training method for multilingual tasks as described above.

[0009] On the other hand, a terminal is provided, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the model pre-training method for multilingual tasks as described above.

[0010] On the other hand, a computer program product or a computer program is provided, which includes computer instructions, which, when executed by a processor, implement the model pre-training method for multi-language tasks as described above.

[0011] The model pre-training method, apparatus, device, storage medium, server, terminal, computer program and computer program product for multilingual tasks provided in this application have the following technical effects: The multimodal training data set used in this application includes multiple sample text data and multiple sample image-text pair data, the multiple sample image-text pair data and multiple sample text data include content in multiple languages, and the multiple sample image-text pair data include sample image-text pair data in general fields and sample image-text pair data in preset business fields in target scenarios, and the multiple sample text data include text data in preset business fields, so as to provide multilingual image-text data and pure text data covering preset business fields, improve the adaptability of multimodal models to multilingual tasks in specific business fields, and have the ability to understand both multimodal data and text data. In addition, in the pre-training process, a two-stage training method is adopted, and visual feature and text feature alignment training and content understanding constraint training are performed respectively, so as to realize the multimodal capability migration from high-resource language to low-resource language through multilingual and multimodal pre-training, reduce data requirements, shorten training time, and improve model performance and versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application; Figure 2 It is a flowchart of a model pre-training method for multilingual tasks provided in an embodiment of the present application; Figure 3 It is a schematic diagram of a framework of a model pre-training device for multilingual tasks provided in an embodiment of the present application; Figure 4 This is a hardware structure block diagram of an electronic device that performs a model pre-training method for multi-language tasks provided in an embodiment of the present application. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0015] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server comprising a series of steps or submodules is not necessarily limited to those steps or submodules clearly listed, but may include other steps or submodules that are not clearly listed or inherent to these processes, methods, products or devices.

[0016] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0017] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0018] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, pre-trained models are also called large models and basic models. After fine-tuning, they can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0019] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision that uses cameras and computers to replace human eyes to identify, track and measure targets, and further performs image processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Large model technology has brought important changes to the development of computer vision technology. Pre-trained models in the visual field such as swin-transformer, ViT, V-MOE, and MAE can be quickly and widely applied to downstream specific tasks after fine tuning. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0020] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing involves natural language, which is the language people use in daily life and is closely related to linguistic research; it also involves computer science and mathematics, and is an important technology for model training in the field of artificial intelligence. The pre-trained model is developed from the large language model in the field of NLP. After fine-tuning, the large language model can be widely used in downstream tasks. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0021] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role. The solution provided in the embodiment of this application involves artificial intelligence machine learning / deep learning, computer vision technology and natural language processing technology, which are specifically described by the following embodiments.

[0022] See also Figure 1 , Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application, such as Figure 1 As shown, the application environment may include a terminal 01 and a server 02. In actual applications, the terminal 01 and the server 02 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0023] The server 02 in the embodiment of the present application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0024] Specifically, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing. Cloud technology can be applied to various fields, such as medical cloud, cloud Internet of Things, cloud security, cloud education, cloud conference, artificial intelligence cloud service, cloud application, cloud call and cloud social networking. Cloud technology is based on the cloud computing business model application. It distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, expanded at any time, and paid for by use. As a basic capability provider of cloud computing, a cloud computing resource pool (referred to as cloud platform, generally referred to as IaaS (Infrastructure as a Service)) platform will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0025] According to the logical function division, the PaaS (Platform as a Service) layer can be deployed on the IaaS layer, and the SaaS (Software as a Service) layer can be deployed on the PaaS layer, or SaaS can be directly deployed on IaaS. PaaS is a platform for software operation, such as databases, web containers, etc. SaaS is a variety of business software, such as web portals, SMS mass senders, etc. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0026] Specifically, the server 02 mentioned above may include a physical device, which may specifically include a network communication submodule, a processor, a memory, etc., and may also include software running in the physical device, which may specifically include an application program, etc.

[0027] Specifically, terminal 01 may include physical devices such as smart phones, desktop computers, tablet computers, laptops, digital assistants, augmented reality (AR) / virtual reality (VR) devices, intelligent voice interaction devices, smart home appliances, smart wearable devices, and vehicle-mounted terminal devices, and may also include software running in physical devices, such as applications.

[0028] In an embodiment of the present application, terminal 01 can be used to obtain a multimodal training data set and an initial model, and send them to server 02, so that server 02 performs comparative learning training of visual feature and text feature alignment on the initial model based on the multiple sample image-text pair data, and performs constraint training of content understanding on the initial model that meets the first end condition based on the multiple sample image-text pair data and the multiple sample text data, to obtain a target model.

[0029] Specifically, the model pre-training method for multilingual tasks of the present application can be applied to image content understanding in preset business fields under target scenarios to perform tasks such as image classification and image content description, such as payment content review business, transaction contract review, or visual question answering (VQA), image captioning, target detection, positioning and image classification.

[0030] Furthermore, it is understandable that Figure 1 What is shown is merely an application environment of a model pre-training method for multi-language tasks. The application environment may include more or fewer nodes, and this application does not impose any limitation thereto.

[0031] The application environment involved in the embodiment of the present application, or the terminal 01 and the server 02 in the application environment, etc., can be a distributed system formed by connecting a client and multiple nodes (any form of computing devices in the access network, such as a server, a user terminal) through network communication. The distributed system can be a blockchain system, which can provide the above-mentioned model pre-training service for multilingual tasks, neural network training service and data storage service, etc.

[0032] The following is an introduction to the technical solution of this application based on the above application environment. The embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. Please refer to Figure 2 , Figure 2 It is a flowchart of a model pre-training method for multi-language tasks provided in an embodiment of the present application. This specification provides method operation steps such as the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or server product is executed, it can be executed in the order of the method shown in the embodiment or the figure or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, Figure 2 As shown, the method may include the following steps S201-S207: S201: Obtain a multimodal training dataset and an initial model.

[0033] Specifically, the training data set includes a plurality of sample text data and a plurality of sample image-text pair data, and the plurality of sample image-text pair data and the plurality of sample text data include content in multiple languages. Sample image-text pair data refers to sample data including images and texts, and sample text data refers to plain text data. The training data set may contain sample image-text pair data and sample text data in multiple languages, and a single sample image-text pair data may also include image content or text in multiple languages, and a single sample text data may also include text content in multiple languages.

[0034] Specifically, a plurality of sample image-text pair data include sample image-text pair data in a general field and sample image-text pair data in a preset business field in a target scenario, and a plurality of sample text data include text data in a preset business field. The target scenario can be a specific business scenario, such as a cross-border business scenario, and the preset business field can be a specific business field under the target scenario, such as a multilingual business field, etc. Accordingly, a multilingual image-text multimodal training data set including a cross-border field is obtained by constructing a training set, including multilingual sample image-text pair data and multilingual sample text data. Exemplarily, the sample image-text pair data in a general field can be an image-text pair of an existing public training set, or an image-text pair obtained from the Internet, and no source and content restrictions are made here. The sample image-text pair data in the preset field can include a multilingual cross-border image-text data set, which can specifically include multilingual image-text data in the fields of multilingual payment, e-commerce platform, multilingual trade and logistics, etc., covering multilingual product images and texts, trade material contract documents, product images, help manuals, logistics information attachments, store audit background screenshots and photo information, product labels, currencies of different countries, and other contents.

[0035] Specifically, when constructing a multilingual image-text pre-training dataset for a preset business field, it includes sample image-text pair data and sample text data for the preset business field, and the two are mixed in a certain training ratio. The training ratio can be exemplified as 3:7, etc. The training ratio refers to the data ratio input in each iteration during the training process; at the same time, multiple open source image-text data are obtained and sampled according to the data source to obtain an open source image-text dataset, which may include only image-text pair data, or may include image-text pair data and plain text data at the same time, and is mixed with the pre-training dataset for the preset business field in a certain data ratio according to task requirements to obtain a multimodal training dataset.

[0036] Specifically, image enhancement processing can be performed on the images in the sample image-text pairs, such as using StyleGAN3 for image enhancement.

[0037] In a possible implementation manner, the type of sample image-text data in a preset business field is determined based on the amount of text in the image, including the first type, the second type, the third type, and the fourth type with increasing text amounts. The image corresponding to the first type is a text-free image, and the image corresponding to the fourth type is a dense document image, such as a text document image. Exemplarily, the text-free image can be a multilingual product image of a multilingual e-commerce platform, etc., and the corresponding product description text with the product image. The second type of image can be a sparse text image, such as a screenshot and photo information of the store audit background. The third type of image can be a multi-text image, such as a multilingual trade material contract document, a logistics information attachment, etc. The fourth type of image can be a dense long text image, such as a multilingual payment field help manual, etc., which can be a multi-page PDF document. The original data type of the image source in the sample image-text data can be PDF, picture, Word, PowerPoint, Excel, TXT, etc., and the corresponding image data is obtained by image conversion.

[0038] It can be understood that after converting different types of data into initial images of preset fields through image conversion, the initial images are preprocessed, including but not limited to image data filtering and deduplication, etc., to eliminate low-quality, non-standard, and unreasonable images, such as file formats zip, rar, 7z compressed files, etc.; it can also include image cleaning and enhancement, denoising the image, enhancing contrast and clarity to improve the recognizability of the digital area. Cleaning and enhancement methods include but are not limited to adjusting color difference, random rotation, random horizontal or vertical flipping, scaling, cropping, translation, etc. For example, deblurring and enhancing low-quality photo scenes in logistics scenarios to correct the tilt and distortion in the image to ensure that the numbers are presented at the correct angle.

[0039] Accordingly, after obtaining the image of the preset business field, image data annotation and text generation are performed on it. The method for obtaining the sample image and text data of the preset business field includes S301-S307: S301: For an image of the first type, obtain description texts of multiple language versions of the image, and combine the description texts of each language version with the image to obtain sample image-text pair data.

[0040] Specifically, for the first type of text-free images, the description text attached to the image can be directly extracted from the database to generate a picture-text pair, such as obtaining the description text of multilingual product images on a multilingual e-commerce platform to generate a product image-product description pair, and translating the description text into texts in multiple languages ​​to obtain a multilingual picture-text pair. It can be understood that the description text can be used as the first sample annotation of the first type of image.

[0041] S303: For the second type of image, perform content understanding and content extraction on the image based on the multimodal model to obtain indication information and a first sample annotation corresponding to the indication information; combine the image, the indication information and the first sample annotation to obtain sample image-text pair data.

[0042] Specifically, for the second type of sparse text images, a multimodal model, such as a multimodal large model, is used to set instruction information (prompt) and corresponding first sample annotations in the corresponding language based on the language corresponding to the image. The instruction information can be, for example, a content question, and the first sample annotation can be a corresponding answer, such as "question: What is the name of the store in this picture?", "answer: stars"; or, "question: What is the name of the magasin sur cettephoto?", "answer: étoile".

[0043] S305: For the third type of image, perform layout area analysis on the image to obtain graphic elements and document elements in the image; obtain indication information of at least one of the graphic elements and document elements and a first sample annotation corresponding to the indication information; combine the image, the indication information and the first sample annotation to obtain sample image-text pair data.

[0044] Specifically, the third type of multi-text images, such as multi-language trade material contract documents, logistics information attachments, etc., have various image formats, which may be electronic versions, scanned versions, etc., and may contain irregular tables, pictures, texts and other elements. By performing layout area analysis on the image, such as layout analysis, marking the image elements and document elements with annotation boxes, and outputting the annotation box coordinates and categories, the text, table and other document elements in the image, as well as elements such as pictures, can also be parsed and saved in a lightweight markup language format as the first sample annotation. In this way, the data of the preset business field is classified based on the amount of text, and the corresponding types of images, indication information and first sample annotations are generated. In combination with the characteristics of the business field, the model can not only learn multi-language domain knowledge, but also learn the recognition and understanding capabilities of different text images, thereby better mastering the multimodal data understanding capabilities of the preset business field, so as to facilitate the model fine-tuning of subsequent specific tasks.

[0045] Specifically, the indication information of the image elements and document elements can be used to instruct the model to mark the position of the corresponding elements, identify the text content or understand the text content, etc. Correspondingly, the types of indication information of the third type of image include content question indications for document elements, position detection indications for image elements or document elements, and content question indications and position detection indications for at least one of the image elements and document elements. The content question indication includes document content recognition or content understanding questions and answers, and the position detection indication is used to instruct the model to identify the position of the corresponding element in the image. The first annotation information is the reply information corresponding to the indication information, such as the text content in the document corresponding to the content question indication or the answer to the content understanding question, or the annotation box coordinates of the document element or image element corresponding to the position detection indication. In this way, fine-grained, multi-type indication information and annotation information are generated for the third type of multi-text images, further improving the model's ability to understand multi-text images in the field, so as to achieve accurate understanding of multi-modal fine-grained content.

[0046] Exemplarily, the indication information of the third type of image and the first sample annotation may only include text content recognition in the content question indication, such as annotation text recognition data, such as question: " \nPlease give the content of the text in the picture", answer:"***********"; or, the indication information and the first sample annotation may only include a position detection indication, such as an indication annotation detection data, such as question:" struct with bbox", answer: " <h2>< / h2> <box> 190,841,315,859< / box> <lb> <lb> <h2>< / h2> <box> 199,154,478,174< / box> <lb> <lb> <box> 148,719,853,823< / box> <lb> <lb> <box> 149,89,850,136< / box> <lb> <lb> <h2>< / h2> <box> 198,525,498,545< / box> <lb> <lb> <h2>< / h2> <box> 198,682,437,701< / box> <lb> <lb> <footer>< / footer> <box> 491,928,509,939< / box> <lb> <lb> <footer>< / footer> <box> 900,957,997,989< / box> <lb> <lb> <box> 190,878,514,895< / box> <lb> <lb>"; or, the indication information and the first sample annotation may include both a content question indication and a position detection indication, such as annotation detection + text recognition data, such as question: " \nI want to know the content of the text in the picture and their respective positions", answer:" <ref>test <box> 190,841,315,859< / box> < / ref> <ref>result <box> 199,154,478,174< / box> < / ref> "; Alternatively, the indication information and the first sample annotation may include content understanding question answering, such as multi-round question answering data, such as question:' \n what is the "seller" ? Answer the question using a single word or phrase.'}, answer: "*****", question: "What is the account number ?", answer: "*****".

[0047] S307: For the fourth type of image, the target document corresponding to the image is input into the large language model for content understanding and content extraction, and the indication information corresponding to the image and the first sample annotation corresponding to the indication information are obtained; the image, the indication information and the first sample annotation are combined to obtain sample image-text pair data; the target document is obtained by segmenting the original text document, and the fourth type of image is obtained by converting the target document into a picture.

[0048] Specifically, for the fourth type of dense document images, such as images of multi-page PDF documents such as help manuals in the cross-border payment field, the original long document is parsed and segmented into a size that the model can handle to obtain multiple target documents, which are then converted into image formats to construct sample image-text pair data of multiple image-text types. At the same time, the target document, the indication information corresponding to the document content, and the first sample annotation can be combined into sample text data. Among them, a large language model (LLM) can be used to understand and extract the content of the segmented target document to set indication information for the fourth type of image, and obtain image annotations as the first sample annotations, such as "question: Which sites does the XX e-commerce platform support for payment collection?", "answer: "*****""; "question: How much is the cross-border payment collection fee?", "answer: "*****"", etc.

[0049] Specifically, in the process of acquiring sample text and image data in general fields or preset business fields, image enhancement methods can also be used to synthesize diversified data, such as rendering text in question-answer data sets into images, rendering table data into composite images of tables and charts, or generating multilingual field-specific composite images, such as multilingual product labels, images of currencies of different countries, etc. Exemplarily, the image enhancement method can be StyleGAN3, etc.

[0050] By building a multi-scenario, multi-language, and multi-modal professional dataset covering general fields and preset business fields, the model is deeply adapted to multilingual business fields. The versatility of the model in multilingual scenarios is enhanced through diversified training data, and its ability to handle different types of inputs is improved.

[0051] Specifically, the initial model includes a visual encoder, a projection module and a decoding module connected in sequence. The decoding module is built based on a large language model, such as internlm2-chat, Qwen2 and other multi-language large models, to build a multimodal model architecture. The vision encoder is used to process images to generate visual features, the projection module is used to map visual features to text feature space and process text and generate text features, and the decoding module is used to receive the output of the projection module and understand the content to obtain the output result corresponding to the indication information. Exemplarily, the projection module can adopt a multi-layer perceptron model (MLP).

[0052] S203: Based on multiple sample image-text pair data, the initial model is trained for comparative learning of the alignment of visual features and text features. During the training process, the model parameters of the decoding module are frozen and the model parameters of the visual encoder and the projection module are adjusted until the first end condition is met.

[0053] S205: Based on multiple sample image-text pair data and multiple sample text data, constrained training of content understanding is performed on the initial model that meets the first end condition, and the model parameters of the visual encoder, projection module and decoding module are adjusted during the training process until the second end condition is met.

[0054] Specifically, the pre-training method of the present application is divided into two stages, a contrastive learning training stage and a joint training stage of mixed data. The first stage is a contrastive learning training based on image-text pairs, which is used to align the visual features (visual embedding) of the visual encoder and the projection module with the input feature space of the large language model of the decoding module. By freezing the large language model of the decoding module, the visual encoder and the projection module are trained on large-scale image-text pair data to perform visual-text feature alignment. During this period, a contrastive learning strategy is used to optimize the visual encoder and the projection module so that they can learn the mapping relationship between visual and multilingual text representations. The second stage is the joint training of the visual encoder, projection module and decoding module based on sample image-text pair data and sample text data. In the second stage, all modules are unfrozen, and the large language models of the visual encoder, projection module and decoding module are jointly trained using large-scale sample image-text pair data and sample text data of pure text. During the training, a progressive unfreezing strategy is adopted to gradually unfreeze and fine-tune the parameters of each layer in the large language model of the decoding module to improve the training effect and reduce the training resource usage.

[0055] Combining cross-modal pre-training of images and texts enables the model to better understand cross-language and cross-domain data, improving the migration effect of pre-trained models in specific fields. By using image-text pairs and pure text data to train multimodal fusion, the problem of language model capability degradation during alignment training of different modalities is greatly reduced, further improving cross-domain and cross-language multimodal understanding capabilities.

[0056] S207: Determine the initial model that meets the second end condition as the target model.

[0057] Specifically, the text in the sample image-text pair data and the sample text data need to be segmented. During the training process, the text in the sample image-text pair data and the sample text data in the preset business field are segmented in combination with the target vocabulary corresponding to the preset business field as the input of the projection module. Exemplarily, the target vocabulary can contain multilingual versions of professional terms in the cross-border field, and the BPE (Byte Pair Encoding) method can be used for text segmentation. Based on the customized target vocabulary as the knowledge base for segmentation, the model is deeply adapted to the preset business field and the model's ability to understand specific languages ​​in the preset business field is enhanced.

[0058] In some embodiments, S203 may include S401-S415: S401: Sampling a plurality of sample image-text pairs to obtain positive sample image-text pairs and negative sample image-text pairs; S403: performing word segmentation and feature embedding on the text of the positive sample image-text pair and the text of the negative sample image-text pair to obtain a first initial text feature; S405: Inputting the image of the positive sample image-text pair and the image of the negative sample image-text pair into a visual encoder for feature encoding to obtain a first visual feature; S407: Inputting the first visual feature and the first initial text feature into a projection module to perform text space mapping of the visual feature and text feature extraction, to obtain a first mapping feature corresponding to the first visual feature and a first text feature corresponding to the first initial text feature; S409: concatenating the first mapping feature and the first text feature to obtain a first joint feature; S411: Inputting the first joint feature into the decoding module to understand the content of the combined indication information to obtain a first output result; S413: Determine a first model loss based on a first output result and a first sample annotation of the positive sample image-text pair, and a first output result and a first sample annotation of the negative sample image-text pair; S415: Train the initial model based on the first model loss to adjust the model parameters of the visual encoder and the projection module until the first end condition is met.

[0059] Specifically, a positive sample image-text pair refers to an image-text pair in which the image and text match, and a negative sample image-text pair refers to an image-text pair in which the image and text do not match. By sampling and obtaining part of the original sample image-text pair data as a positive sample image-text pair, and combining images with non-matching texts, multiple negative sample image-text pairs are obtained.

[0060] Specifically, the multi-language texts in the positive sample image-text pairs and the negative sample image-text pairs are segmented to obtain text segmentation (tokens), and feature embedding is implemented through word embedding conversion to generate vector representations of text segmentation, and the vector representations of text segmentation are concatenated to obtain the first initial text features. In the word segmentation process, the target word library is used as the word segmentation knowledge base to segment the text corresponding to the preset business field.

[0061] Specifically, the images in each positive sample image-text pair and each negative sample image-text pair obtained by sampling are respectively input into the visual encoder, feature encoding and feature extraction are performed on the images, and the first visual features corresponding to each image are output. Then, the first visual features and the first initial text features of each positive sample text pair are input into the projection module, and the first visual features and the first initial text features of each negative sample text pair are input into the projection module, so that the input first visual features are aligned in the text space through the projection module, mapped into the first mapping features, and the input first initial text features are feature extracted to obtain the first text features mapped to the text space. It can be understood that the text space here refers to the text feature space of the large language model of the decoding module.

[0062] Furthermore, the first mapping feature and the first text feature of the positive sample image-text pair, as well as the first mapping feature and the first text feature of the negative sample image-text pair, are concatenated to obtain their respective first joint features as input to the decoding module, and combined with the content understanding guidance of the indication information to obtain the first output result. The first output result is the reply information output by the decoding module for the indication information, and the first sample annotation is the annotation true value corresponding to the indication information.

[0063] Combined with the training method of contrastive learning, when the parameters of the decoding module are frozen, the distance between the first output result of the positive sample image-text pair and the first sample annotation is shortened, that is, the distance between the first mapping feature and the first text feature is shortened accordingly, and the distance between the first output result of the negative sample image-text pair and the first sample annotation is increased, that is, the distance between the first mapping feature and the first text feature is increased, thereby determining the first model loss, which at least includes the contrastive loss. Based on the first model loss, the model parameters of the visual encoder and the projection module are adjusted to obtain an updated initial model, and the aforementioned S401-S411 is repeatedly executed with the updated initial model, and so on, until the first end condition is met. Here, satisfying the first end condition means that the number of training iterations meets the preset number, or the first model loss is less than the preset loss, etc. It can also be other state conditions that meet the end of training, which are not limited here.

[0064] In this way, by constructing professional domain datasets and target lexicons that cover multiple scenarios, multiple languages, and multiple modalities, such as multilingual payment, e-commerce, and trade, combined with contrastive learning, the visual encoder's feature extraction capabilities for general domain images and preset business domain images are improved, as well as the projection module's ability to align domain visual features and text features, thereby providing high-quality input for the subsequent decoding module's comprehension training, enabling it to acquire excellent domain visual text comprehension capabilities.

[0065] In some embodiments, S205 may include S501-S513: S501: Perform word segmentation and feature embedding on the text of the sample image-text pair data and the sample text data to obtain a second initial text feature corresponding to the sample image-text pair data and a third initial text feature corresponding to the sample text data; S503: Inputting the image of the sample image-text pair data into a visual encoder for feature encoding to obtain a second visual feature; S505: Inputting the second visual feature and the second initial text feature of the sample image-text pair data, and the third initial text feature into the projection module respectively, performing text space mapping of the visual feature and text feature extraction, and obtaining the second mapping feature and the second text feature corresponding to the sample image-text pair data, and the third text feature corresponding to the sample text data; S507: concatenating the second mapping feature corresponding to the sample image-text pair data with the second text feature to obtain a second joint feature; S509: inputting the second joint feature and the third text feature into the decoding module respectively to understand the content in combination with the indication information, and obtaining a second output result corresponding to the second joint feature and a third output result corresponding to the third text feature, wherein the indication information is used to provide the decoding module with guidance information required for understanding the content, and the indication information includes prompt texts in multiple languages; S511: Determine a second model loss based on the second output result and the first sample annotation of the sample image and text data, and the third output result and the second sample annotation of the sample text data; S513: Based on the second model loss, train the initial model that meets the first end condition to adjust the model parameters of the visual encoder, the projection module and the decoding module until the second end condition is met.

[0066] Specifically, in the word segmentation process, the target word library is used as the word segmentation knowledge base to segment the text corresponding to the preset business field. The word segmentation process in S501 is consistent with the word segmentation process involved in the previous text, and will not be repeated here.

[0067] In the second stage, sample image-text pair data and sample text data of general fields and preset business fields in the multimodal training data set are used as training inputs, and image features are extracted through the data encoder to obtain the corresponding second visual features, and the second visual features are used as the input of the projection module to perform text space alignment to obtain the corresponding second mapping features; at the same time, the second initial text features corresponding to the sample image-text pair data and the third initial text features corresponding to the sample text data are respectively input into the projection module, and feature extraction is performed respectively to obtain the second text features corresponding to the second initial text features, and the third text features corresponding to the third initial text features. Then, the second mapping features and the second text features corresponding to the sample image-text pair data are spliced, and the obtained second joint features are used as the input of the decoding module to perform feature-based content understanding, and the third text features are used as the input of the decoding module to perform content understanding, so as to output the second output results and the third output results respectively. It is understandable that the input of the decoding module also includes indication information, such as the indication information carried in the aforementioned sample image-text pair data. Accordingly, the annotated answer to the indication information in the text of the image-text pair is the corresponding first sample annotation, such as the position coordinates of the image element or the answer to the question about the content of the image element, etc. The indication information can also be a question about the text content carried in the sample text data, etc. The annotated answer to the indication information carried in the sample text data is the corresponding second sample annotation, such as the indication information can be "how much is the liquidated damages proposed in the contract document", and the second sample annotation is "the liquidated damages are XXX US dollars". It is understandable that the indication information is used to provide the guiding information required by the decoding module for content understanding. The indication information includes prompt texts in multiple languages. Accordingly, the first sample annotation or the second sample annotation also includes the corresponding annotated answers in multiple languages.

[0068] Specifically, based at least on the difference between the second output result and the corresponding first sample annotation, and the difference between the third output result and the corresponding second sample annotation, the second model loss is calculated to adjust the model parameters of each module in the initial model that satisfies the first end condition until the second end condition is satisfied. The second end condition is similar to the limitation of the aforementioned first end condition and will not be elaborated here.

[0069] In this way, in the second training stage, sample image-text pair data and sample text data can be alternately input into the initial model that meets the first end condition, allowing the model to effectively integrate and utilize visual information while maintaining language comprehension capabilities, avoiding the problems of language model capability degradation, knowledge updating and forgetting caused by alignment training, improving the migration effect and generalization ability of the pre-trained model in specific fields, and improving the performance of the model on multimodal tasks.

[0070] In a specific embodiment, the second training stage uses a large number of image-text interaction pairs and plain text corpora to train the visual encoder, projection module and decoding module at the same time. In this stage, the visual encoder processes the input image and generates visual features, which are aligned with the text tags through the projection module. The projection module maps the text and visual embeddings into the same semantic space based on the self-attention mechanism, and then adds the processed mapping features to the front of the text features to form a joint representation, which is passed to the large language model of the decoding module for processing, so that each layer of the large language model (including the decoder layer) is trained separately in its attention mechanism of query (Q), key (K), value (V) matrix and feedforward neural network (FFN) to further ensure the model's multi-language understanding ability and visual information understanding ability, as well as the ability to integrate the two.

[0071] Based on some or all of the above implementations, in some embodiments, the visual encoder includes a feature extraction module, a feature fusion module and a feature extraction module. Among them, the feature extraction module is used to extract features from the sub-image corresponding to the input image to obtain a feature map, and the feature fusion module uses the feature maps corresponding to the image as input to perform feature fusion to obtain fused features, which are then used as input to the feature extraction module to obtain corresponding visual features. Accordingly, the feature encoding process of the visual encoder for the input image includes S601-S605: S601: Input the image into a feature extraction module, extract local features of the image based on a self-attention mechanism of a sliding window, and obtain a multi-scale feature map; S603: Input the multi-scale feature map into the feature fusion module for feature fusion to obtain fusion features; S605: Input the fused features into a feature extraction module for feature extraction to obtain a first visual feature or a second visual feature corresponding to the image.

[0072] Specifically, in the first training stage and the second training stage, the feature extraction module extracts regional features of the image based on a sliding window, and based on the self-attention mechanism, uses the features of other regions corresponding to the current region as contexts for feature crossover to obtain a multi-scale feature map. The multi-scale feature map is fused by the feature fusion module to obtain a fusion feature, which is then input into the feature extraction module for further feature extraction to obtain the first visual feature of the first training stage or the second visual feature of the second training stage.

[0073] In this way, the granularity of image feature extraction is improved through multi-feature map extraction and self-attention mechanism, and fine-grained visual features are obtained. The full expression of visual information is achieved through fusion and re-extraction, thereby improving the comprehensiveness and accuracy of the input information of the decoding module.

[0074] In some embodiments, S601 may include S6011-S6012: S6011: performing adaptive image segmentation on the image to obtain a plurality of sub-images corresponding to the image; S6012: Extract local features of each sub-image based on the self-attention mechanism of the sliding window to obtain a multi-scale feature map corresponding to the sub-image.

[0075] Through adaptive image segmentation and multi-scale feature map extraction, we can avoid the problem of too large image segmentation features caused by too large image input, adapt to image input of different sizes and contents, and improve the accuracy and comprehensiveness of image feature extraction. Specifically, an adaptive image segmentation algorithm is used to adaptively segment the image, and the slice size is dynamically adjusted according to the aspect ratio and resolution of each image, and it is cropped into a certain number of sub-images. The resolution of each sub-image can be fixed to a preset resolution, such as 448×448 pixels. The number of sub-images for image segmentation can be exemplified as 1 to 12.

[0076] Specifically, the feature extraction module, as the backbone network of the video encoder, performs image segmentation on the input sub-image based on the sliding window to obtain multiple visual areas of preset sizes, so as to divide the image into multiple tiles of preset sizes according to the image resolution as image slices (tokens), for example, into 1024 small tiles of 14*14, and then uses a convolution connection structure such as a convolution kernel to process each small tile to obtain an image token, such as using a convolution kernel with a size and a step size of 1x4 to process 4 adjacent tiles in the horizontal direction at a time and merge them into a new feature, so that the number of image tokens for each sub-image is 256, reducing the length of the sequence to meet the subsequent model processing requirements. In the sub-image processing process of the feature extraction module, a multi-scale feature map is output for each sub-image, each attention layer in the feature extraction module obtains multi-scale information, and each attention layer can selectively merge image slices based on different receptive fields to achieve the merging and information extraction of related image areas, and retain some image slices to obtain fine-grained features and better process targets of different scales. Exemplarily, the feature extraction module of the visual encoder can be built based on SwinTransformer, and a Shunted Self-Attention module is used in each attention layer inside Swin Transformer to focus on different parts of the feature map.

[0077] In some embodiments, the feature fusion module uses the multi-scale feature map output by the feature extraction module as the input of the feature fusion module, constructs a feature pyramid through upsampling and lateral connection, performs object detection or semantic segmentation on different layers of the feature pyramid, uses multi-scale features to improve the performance of the model, and uses the output fusion features as the input of the feature extraction module. Exemplarily, the feature fusion module can be, for example, FPN (Feature Pyramid Networks, feature pyramid model).

[0078] In some embodiments, the sample image-text pair data of each preset business field includes multiple types of image-text pairs, the feature extraction module includes a gating network, and multiple expert networks matching the multiple types, and S605 may include: estimating the type of the image based on the gating network, and inputting the fused features into the expert network matching the estimated type for feature extraction to obtain a first visual feature or a second visual feature.

[0079] Specifically, the gated network is used to determine the image type of the image-text pair corresponding to the input fusion feature, and route it to the expert network corresponding to the type. The expert network is used to process the fusion feature to obtain features for input into the decoding module. The image-text pair type here refers to the aforementioned type determined based on the amount of text, such as the first type, the second type, the third type, and the fourth type. It can be understood that the network structure of each expert network can be consistent or specifically set. Each expert network processes images of different text types to expand the scale of the visual encoder and improve training efficiency, while reducing computing resource usage and improving the processing and understanding capabilities of different images. Exemplarily, the feature extraction module can be built based on MoE (Mixed Expert Models).

[0080] Through the data processing scheme of the above-mentioned video encoder, the problems of poor representation effect and poor model performance of text-intensive images in the prior art can be solved through expert network settings, efficient and flexible visual processing can be achieved, the ability of the model to process complex visual information can be improved, and fine-grained feature extraction of images can be performed.

[0081] Based on some or all of the above implementations, in some embodiments, the method for obtaining the first model loss includes S701-S709: S701: Generate a contrast loss based on a difference between a first output result of a positive sample image-text pair and a first sample annotation, and a difference between a first output result of a negative sample image-text pair and a first sample annotation; S703: Based on the selection probability of each expert network and the number of expert networks for the images input in the training batch, the auxiliary loss of the feature extraction module is calculated to obtain the load balancing loss; S705: performing auxiliary loss calculation of the feature extraction module based on the probability index data obtained by type estimation of the fused feature by the gated network to obtain the router z loss, where the probability index data includes the probability that the image corresponding to the fused feature belongs to each expert network; S707: For the sample image-text pair data of the multilingual text type, based on the output results of the multilingual version in the first output result, perform cross-language consistency loss calculation to obtain the cross-language consistency loss; S709: Fusion contrast loss, load balancing loss, router z loss and cross-language consistency loss to obtain a first model loss.

[0082] Specifically, the contrast loss can be based on the existing contrast loss calculation method, with the difference between the text or position mark in the first output result and the text true value and position true value of the first sample annotation as the input of the contrast loss function, with the goal of bringing the first output result and the first sample annotation of the positive sample image-text pair closer, and moving the first output result and the first sample annotation of the negative sample image-text pair further apart, to obtain the contrast loss.

[0083] Specifically, the load balancing loss and the router z loss are auxiliary losses for the gated network and the expert network in the feature extraction module, and are used to maintain load balance between each expert network in the visual encoder and the projection module. Different types of images tend to be processed by different expert networks, and there may be data imbalance problems. By statistically analyzing the selection probability of the images input in the training batch in each expert network, that is, combining the probability of the images processed by each expert network to perform auxiliary loss calculation, the load balancing loss is obtained. The input of the router z loss calculation includes the probability index data obtained by the gated network for type estimation of the fused features. After the fused features are input into the gated network, the gated network estimates which expert network the image token corresponding to the fused features belongs to, so as to obtain the probability that each image token belongs to each expert network, and then generate probability index data. The router z loss is used to encourage the model to generate more moderate probability values, thereby preventing the model from generating overly extreme outputs. Exemplarily, the router z loss function and the load balancing loss function of the MoE model can be used.

[0084] Specifically, the cross-lingual consistency loss is a loss function introduced to ensure that the model's performance in different languages ​​remains consistent. The output of the multilingual version of the first output result of the same image-text pair data is used as the input of the loss function, and the loss is calculated with the goal of minimizing the semantic difference between the output results of the same image-text pair data in different languages, so as to improve the cross-lingual generalization ability of the model.

[0085] By integrating contrast loss, load balancing loss, router z loss and cross-language consistency loss, the alignment ability of each module for visual features and text features can be comprehensively optimized, thereby improving the training effect. Specifically, the second model loss L1=Σ(w i * L i ), where w i is the adaptive weight, L i Characterizes the loss to be fused.

[0086] Based on some or all of the above implementations, in some embodiments, the method for obtaining the second model loss includes S801-S809: S801: generating a cross entropy loss based on a difference between the second output result and the first sample annotation, and a difference between the third output result and the second sample annotation; S803: Based on the selection probability of each expert network for the input image in the training batch and the number of expert networks, the auxiliary loss of the feature extraction module is calculated to obtain the load balancing loss; S805: performing auxiliary loss calculation of the feature extraction module based on the probability index data obtained by performing type estimation on the fusion feature based on the gated network to obtain the router z loss, where the probability index data includes the probability that the image corresponding to the fusion feature belongs to each expert network respectively; S807: For the sample image-text pair data or text data of the multilingual text type, based on the output result of the multilingual version in the second output result and the output result of the multilingual version in the third output result, a cross-language consistency loss is calculated to obtain a cross-language consistency loss; S809: Fuse the cross entropy loss, load balancing loss, router z loss and cross-language consistency loss to obtain the second model loss.

[0087] Specifically, the cross entropy loss can be based on the existing cross entropy loss calculation method, with the difference between the text or position mark in the second output result and the text true value and position true value annotated by the first sample, and the difference between the text in the third output result and the text true value annotated by the second sample as the input of the cross entropy loss function, thereby obtaining the cross entropy loss.

[0088] Specifically, the load balancing loss and router z loss here are similar to the calculation methods mentioned above and will not be described here. The calculation input of the cross-language consistency loss here includes the output results of the multi-language version in the second output result and the output results of the multi-language version in the third output result, which is a loss function introduced to ensure that the performance of the model in different languages ​​remains consistent. By minimizing the semantic differences between the output results in different languages, the cross-language generalization ability of the model is improved.

[0089] By integrating cross entropy loss, load balancing loss, router z loss and cross-language consistency loss, it is possible to comprehensively optimize the semantic expression and understanding of each module for different languages ​​and different types of images, thereby improving the training effect. Specifically, the second model loss L2=Σ(w j * L j ), where w j is the adaptive weight, L j Characterizes the loss to be fused.

[0090] Specifically, during the training process, the model hyperparameters are set to minimize the model loss as the goal of training until the model converges, and the performance of the model that meets the second end condition is evaluated based on the test set of the preset business field (such as the cross-border business field). Comprehensive evaluation indicators (such as image and text matching, text generation, image understanding, etc.) can be designed to test the model performance. According to the evaluation results, the model architecture and training strategy are iteratively optimized. In addition, the pre-training process data can also be detected, including recording and visualizing training indicators, detecting GPU utilization and memory usage, and saving model checkpoints.

[0091] In addition, distributed training can be used to implement the above model pre-training process, such as using DeepSpeed ​​ZeRO-3 for distributed training, applying gradient accumulation and model parallel strategies, simulating large-scale training, realizing pipeline parallelism, and improving GPU utilization.

[0092] The present application embodiment also provides a model pre-training device 800 for multi-language tasks, such as Figure 3 As shown, Figure 3 A schematic structural diagram of a model pre-training device for multilingual tasks provided in an embodiment of the present application is shown, and the device may include the following modules.

[0093] Acquisition module 10: used to acquire a multimodal training data set and an initial model, the training data set includes a plurality of sample text data and a plurality of sample image-text pair data, the plurality of sample image-text pair data and the plurality of sample text data include content in multiple languages, and the plurality of sample image-text pair data include sample image-text pair data in a general field and sample image-text pair data in a preset business field in a target scene, the plurality of sample text data include text data in a preset business field, and the initial model includes a visual encoder, a projection module and a decoding module connected in sequence, and the decoding module is constructed based on a large language model; The first training module 20 is used to perform comparative learning training of the initial model for aligning visual features and text features based on multiple sample image-text pair data, freeze the model parameters of the decoding module and adjust the model parameters of the visual encoder and the projection module during the training process until the first end condition is met; The second training module 30 is used to perform content understanding constraint training on the initial model that meets the first end condition based on multiple sample image-text pair data and multiple sample text data, and adjust the model parameters of the visual encoder, the projection module and the decoding module during the training process until the second end condition is met; Model generation module 40: used to determine the initial model that meets the second end condition as the target model; during the training process, the text and sample text data in the sample image and text data of the preset business field are segmented in combination with the target vocabulary corresponding to the preset business field to serve as the input of the projection module.

[0094] In some embodiments, the first training module 20 is specifically used to: Sampling multiple sample image-text pairs to obtain positive sample image-text pairs and negative sample image-text pairs; Perform word segmentation and feature embedding on the text of the positive sample image-text pair and the negative sample image-text pair to obtain a first initial text feature. In the word segmentation process, use the target word library as a word segmentation knowledge base to segment the text corresponding to the preset business field. Inputting the image of the positive sample image-text pair and the image of the negative sample image-text pair into a visual encoder for feature encoding to obtain a first visual feature; Inputting the first visual feature and the first initial text feature into a projection module to perform text space mapping of the visual feature and text feature extraction, thereby obtaining a first mapping feature corresponding to the first visual feature and a first text feature corresponding to the first initial text feature; Concatenate the first mapping feature and the first text feature to obtain a first joint feature; Inputting the first joint feature into a decoding module to understand the content of the combined indication information to obtain a first output result; Determine a first model loss based on a first output result and a first sample annotation of the positive sample image-text pair, and a first output result and a first sample annotation of the negative sample image-text pair; The initial model is trained based on the first model loss to adjust the model parameters of the visual encoder and the projection module until the first end condition is met.

[0095] In some embodiments, the second training module 30 may be specifically used for: Perform word segmentation and feature embedding on the text of the sample image-text pair data and the sample text data to obtain a second initial text feature corresponding to the sample image-text pair data and a third initial text feature corresponding to the sample text data. In the word segmentation process, the target word library is used as a word segmentation knowledge base to perform word segmentation on the text corresponding to the preset business field. Inputting the image of the sample image-text pair data into a visual encoder for feature encoding to obtain a second visual feature; The second visual feature and the second initial text feature of the sample image-text pair data, as well as the third initial text feature are respectively input into the projection module to perform text space mapping of the visual feature and text feature extraction, thereby obtaining the second mapping feature and the second text feature corresponding to the sample image-text pair data, and the third text feature corresponding to the sample text data; Concatenate the second mapping feature corresponding to the sample image-text pair data and the second text feature to obtain a second joint feature; The second joint feature and the third text feature are respectively input into a decoding module to understand the content of the combined indication information, and a second output result corresponding to the second joint feature and a third output result corresponding to the third text feature are obtained; Determine a second model loss based on the second output result and the first sample annotation of the sample image and text data, and the third output result and the second sample annotation of the sample text data; The initial model that satisfies the first end condition is trained based on the second model loss to adjust the model parameters of the visual encoder, the projection module, and the decoding module until the second end condition is satisfied.

[0096] In some embodiments, the visual encoder includes a feature extraction module, a feature fusion module and a feature extraction module; the device also includes a visual encoding module for: The image is input into the feature extraction module, and local features of the image are extracted based on the self-attention mechanism of the sliding window to obtain a multi-scale feature map; Input the multi-scale feature maps into the feature fusion module for feature fusion to obtain fused features; The fused features are input into a feature extraction module for feature extraction to obtain a first visual feature or a second visual feature corresponding to the image.

[0097] In some embodiments, the sample image-text pair data of each preset business field includes multiple types of image-text pairs, the feature extraction module includes a gated network and multiple expert networks matching the multiple types, and the visual encoding module is further specifically used for: The type of the image is estimated based on the gating network, and the fused features are input into the expert network matching the estimated type for feature extraction to obtain the first visual feature or the second visual feature.

[0098] In some embodiments, the visual encoding module is further specifically used for: Performing adaptive image segmentation on the image to obtain multiple sub-images corresponding to the image; Based on the sliding window self-attention mechanism, local features of each sub-image are extracted to obtain the multi-scale feature map corresponding to the sub-image.

[0099] In some embodiments, the type of sample image-text pair data in a preset business field is determined based on the amount of text in the image, including a first type, a second type, a third type, and a fourth type with increasing amounts of text, the image corresponding to the first type is a text-free image, and the image corresponding to the fourth type is a dense document image; the acquisition module 10 is specifically used to: For the first type of image, obtain the description text of the multi-language version of the image, and combine the description text of each language version with the image to obtain sample image-text pair data; For the second type of image, perform content understanding and content extraction on the image based on the multimodal model to obtain indication information and a first sample annotation corresponding to the indication information; combine the image, the indication information and the first sample annotation to obtain sample image-text pair data; For the third type of image, performing layout area analysis on the image to obtain image elements and document elements in the image; obtaining indication information of at least one of the image elements and the document elements and a first sample annotation corresponding to the indication information; combining the image, the indication information and the first sample annotation to obtain sample image-text pair data; For the fourth type of image, the target document corresponding to the image is input into the large language model for content understanding and content extraction to obtain the indication information corresponding to the image and the first sample annotation corresponding to the indication information; the image, the indication information and the first sample annotation are combined to obtain sample image-text pair data; the target document is obtained by segmenting the original text document, and the fourth type of image is obtained by converting the target document into a picture.

[0100] In some embodiments, the types of indication information of the third type of image include content question indication for document elements, position detection indication for picture elements or document elements, and content question indication and position detection indication for at least one of picture elements and document elements.

[0101] In some embodiments, the first training module 20 is further specifically configured to: Generate a contrast loss based on a difference between a first output result of the positive sample image-text pair and a first sample annotation, and a difference between a first output result of the negative sample image-text pair and a first sample annotation; Based on the selection probability of each expert network and the number of expert networks for the input images in the training batch, the auxiliary loss of the feature extraction module is calculated to obtain the load balancing loss; Based on the probability index data obtained by type estimation of the fused features by the gating network, the auxiliary loss calculation of the feature extraction module is performed to obtain the router z loss. The probability index data includes the probability that the image corresponding to the fused features belongs to each expert network respectively; For the sample image-text pair data of the multilingual text type, based on the output results of the multilingual version in the first output result, a cross-language consistency loss is calculated to obtain the cross-language consistency loss; The first model loss is obtained by fusing contrast loss, load balancing loss, router z loss and cross-lingual consistency loss.

[0102] In some embodiments, the second training module 30 is further specifically configured to: generating a cross entropy loss based on a difference between the second output result and the first sample annotation, and a difference between the third output result and the second sample annotation; Based on the selection probability of each expert network and the number of expert networks for the input images in the training batch, the auxiliary loss of the feature extraction module is calculated to obtain the load balancing loss; Based on the probability index data obtained by type estimation of the fused features by the gating network, the auxiliary loss calculation of the feature extraction module is performed to obtain the router z loss. The probability index data includes the probability that the image corresponding to the fused features belongs to each expert network respectively; For the sample image-text pair data or text data of the multilingual text type, based on the output result of the multilingual version in the second output result and the output result of the multilingual version in the third output result, a cross-language consistency loss is calculated to obtain the cross-language consistency loss; The second model loss is obtained by fusing cross entropy loss, load balancing loss, router z loss and cross-language consistency loss.

[0103] It should be noted that the above device embodiment and method embodiment are based on the same implementation method.

[0104] An embodiment of the present application provides a device, which may be a terminal or a server, including a processor and a memory, wherein at least one instruction or at least one program is stored in the memory, and the at least one instruction or the at least one program is loaded and executed by the processor to implement a model pre-training method or a neural network training method for multi-language tasks as provided in the above method embodiment.

[0105] The memory can be used to store software programs and modules. The processor executes various functional applications and abnormality detection by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required for functions, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0106] The method embodiments provided in the embodiments of the present application can be executed in electronic devices such as mobile terminals, computer terminals, servers or similar computing devices. Figure 4 1 is a hardware structure block diagram of an electronic device for a model pre-training method for multi-language tasks provided in an embodiment of the present application. Figure 4 As shown, the electronic device 900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 910 (the processor 910 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 930 for storing data, and one or more storage media 920 (such as one or more mass storage devices) for storing application programs 923 or data 922. Among them, the memory 930 and the storage medium 920 can be short-term storage or permanent storage. The program stored in the storage medium 920 may include one or more modules, each of which may include a series of instruction operations in the electronic device. Furthermore, the central processing unit 910 can be configured to communicate with the storage medium 920 to execute a series of instruction operations in the storage medium 920 on the electronic device 900. The electronic device 900 may also include one or more power supplies 960, one or more wired or wireless network interfaces 950, one or more input and output interfaces 940, and / or one or more operating systems 921, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0107] The input / output interface 940 may be used to receive or send data via a network. A specific example of the network may include a wireless network provided by a communication provider of the electronic device 900. In one example, the input / output interface 940 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In one example, the input / output interface 940 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0108] It can be understood by those skilled in the art that Figure 4 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 4 More or fewer components as shown, or with Figure 4 Different configurations are shown.

[0109] An embodiment of the present application also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to an anomaly detection method in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the anomaly detection method provided by the above method embodiment.

[0110] Optionally, in this embodiment, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0111] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various optional implementations.

[0112] The model pre-training method, device, equipment, storage medium, server, terminal and program product for multilingual tasks provided by the above-mentioned present application, the multimodal training data set adopted by the technical solution of the present application includes multiple sample text data and multiple sample image-text pair data, the multiple sample image-text pair data and the multiple sample text data include multiple language contents, and the multiple sample image-text pair data include sample image-text pair data of general fields and sample image-text pair data of preset business fields in target scenarios, and the multiple sample text data include text data of preset business fields, so as to provide multilingual image-text data and pure text data covering preset business fields, improve the adaptability of multimodal models to multilingual tasks in specific business fields, and have the ability to understand both multimodal data and text data. In addition, in the pre-training process, a two-stage training method is adopted, and visual feature and text feature alignment training and content understanding constraint training are respectively performed, so as to realize the multimodal capability migration from high-resource language to low-resource language through multilingual and multimodal pre-training, reduce data requirements, shorten training time, and improve model performance and versatility at the same time.

[0113] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of the present application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0114] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0115] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0116] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.< / lb> < / lb> < / lb> < / lb> < / lb> < / lb> < / lb> < / lb> < / lb> < / lb> < / lb> < / lb> < / lb> < / lb> < / lb> < / lb> < / lb> < / lb>

Claims

1. A model pre-training method for multilingual tasks, characterized in that: The method comprises: Acquire a multimodal training data set and an initial model, wherein the training data set includes a plurality of sample text data and a plurality of sample image-text pair data, wherein the plurality of sample image-text pair data and the plurality of sample text data include content in multiple languages, and the plurality of sample image-text pair data include sample image-text pair data in a general field and sample image-text pair data in a preset business field in a target scene, wherein the plurality of sample text data include text data in the preset business field, and the initial model includes a visual encoder, a projection module, and a decoding module connected in sequence, wherein the decoding module is constructed based on a large language model; Based on the plurality of sample image-text pair data, performing comparative learning training of aligning visual features and text features on the initial model, freezing the model parameters of the decoding module and adjusting the model parameters of the visual encoder and the projection module during the training process until a first end condition is met; Based on the multiple sample image-text pair data and the multiple sample text data, the initial model that meets the first end condition is subjected to constrained training for content understanding, and during the training process, the model parameters of the visual encoder, the projection module, and the decoding module are adjusted until the second end condition is met; The initial model that meets the second end condition is determined as the target model; during the training process, the text in the sample image-text data and the sample text data of the preset business field are segmented in combination with the target vocabulary corresponding to the preset business field to serve as the input of the projection module.

2. The method according to claim 1, characterized in that: The step of performing comparative learning training on the initial model to align visual features and text features based on the plurality of sample image-text pair data includes: Sampling the plurality of sample image-text pairs to obtain positive sample image-text pairs and negative sample image-text pairs; Performing word segmentation processing and feature embedding on the text of the positive sample image-text pair and the text of the negative sample image-text pair to obtain a first initial text feature, and in the word segmentation processing, performing word segmentation on the text corresponding to the preset business field using the target word library as a word segmentation knowledge base; Inputting the image of the positive sample image-text pair and the image of the negative sample image-text pair into the visual encoder for feature encoding to obtain a first visual feature; Inputting the first visual feature and the first initial text feature into the projection module to perform text space mapping of the visual feature and text feature extraction, thereby obtaining a first mapping feature corresponding to the first visual feature and a first text feature corresponding to the first initial text feature; Concatenate the first mapping feature and the first text feature to obtain a first joint feature; Inputting the first joint feature into the decoding module to understand the content of the combined indication information to obtain a first output result; Determine a first model loss based on the first output result and the first sample annotation of the positive sample image-text pair, and the first output result and the first sample annotation of the negative sample image-text pair; The initial model is trained based on the first model loss to adjust model parameters of the visual encoder and the projection module until the first end condition is met.

3. The method according to claim 1, characterized in that The step of performing content understanding constraint training on the initial model that meets the first end condition based on the plurality of sample image-text pair data and the plurality of sample text data includes: Performing word segmentation processing and feature embedding on the text of the sample image-text pair data and the sample text data to obtain a second initial text feature corresponding to the sample image-text pair data and a third initial text feature corresponding to the sample text data, and performing word segmentation on the text corresponding to the preset business field using the target word library as a word segmentation knowledge base during the word segmentation processing; Inputting the image of the sample image-text pair data into the visual encoder for feature encoding to obtain a second visual feature; Inputting the second visual feature and the second initial text feature of the sample image-text pair data, and the third initial text feature into the projection module respectively, performing text space mapping of the visual feature and text feature extraction, and obtaining the second mapping feature and the second text feature corresponding to the sample image-text pair data, and the third text feature corresponding to the sample text data; Concatenate the second mapping feature and the second text feature corresponding to the sample image-text pair data to obtain a second joint feature; Inputting the second joint feature and the third text feature into the decoding module respectively to understand the content of the combined indication information, and obtaining a second output result corresponding to the second joint feature and a third output result corresponding to the third text feature; Determine a second model loss based on the second output result and the first sample annotation of the sample image-text data, and the third output result and the second sample annotation of the sample text data; An initial model that satisfies the first end condition is trained based on the second model loss to adjust model parameters of the visual encoder, the projection module, and the decoding module until the second end condition is satisfied.

4. The method according to claim 2 or 3, characterized in that: The visual encoder includes a feature extraction module, a feature fusion module and a feature extraction module; the feature encoding process of the visual encoder for the input image includes: Inputting the image into the feature extraction module, extracting local features of the image based on a self-attention mechanism of a sliding window, and obtaining a multi-scale feature map; Inputting the multi-scale feature map into the feature fusion module to perform feature fusion to obtain fused features; The fused feature is input into the feature extraction module for feature extraction to obtain the first visual feature or the second visual feature corresponding to the image.

5. The method according to claim 4, characterized in that The sample image-text pair data of each preset business field includes multiple types of image-text pairs, the feature extraction module includes a gated network, and multiple expert networks matching the multiple types, and the fusion feature is input into the feature extraction module for feature extraction to obtain the first visual feature or the second visual feature corresponding to the image, including: The type of the image is estimated based on the gating network, and the fused features are input into an expert network matching the estimated type for feature extraction to obtain the first visual feature or the second visual feature.

6. The method according to claim 4, characterized in that The step of inputting the image into the feature extraction module and extracting local features of the image based on a self-attention mechanism of a sliding window to obtain a multi-scale feature map includes: Performing adaptive image segmentation on the image to obtain a plurality of sub-images corresponding to the image; Based on the self-attention mechanism of the sliding window, local features of each sub-image are extracted to obtain a multi-scale feature map corresponding to the sub-image.

7. The method according to claim 4, characterized in that The type of the sample image-text pair data of the preset business field is determined based on the amount of text in the image, including a first type, a second type, a third type and a fourth type with increasing amounts of text, the image corresponding to the first type is a text-free image, and the image corresponding to the fourth type is a dense document image; The method for obtaining the sample image and text data of the preset business field includes: For the first type of image, obtaining description texts of the multi-language versions of the image, and combining the description texts of the respective language versions with the image to obtain the sample image-text pair data; For the second type of image, perform content understanding and content extraction on the image based on the multimodal model to obtain indication information and a first sample annotation corresponding to the indication information; combine the image, the indication information and the first sample annotation to obtain the sample image-text pair data; For the third type of image, performing layout area analysis on the image to obtain graphic elements and document elements in the image; obtaining indication information of at least one of the graphic element and the document element and a first sample annotation corresponding to the indication information; combining the image, the indication information and the first sample annotation to obtain the sample graphic-text pair data; For the fourth type of image, the target document corresponding to the image is input into a large language model for content understanding and content extraction to obtain indication information corresponding to the image and a first sample annotation corresponding to the indication information; the image, the indication information and the first sample annotation are combined to obtain the sample image-text pair data; the target document is obtained by segmenting the original text document, and the fourth type of image is obtained by converting the target document into a picture.

8. The method according to claim 7, characterized in that The types of indication information of the third type of image include content question indication for the document element, position detection indication for the picture element or the document element, and content question indication and position detection indication for at least one of the picture element and the document element.

9. The method according to claim 5, characterized in that The method for obtaining the first model loss includes: Generate a contrast loss based on a difference between a first output result of the positive sample image-text pair and the first sample annotation, and a difference between a first output result of the negative sample image-text pair and the first sample annotation; Based on the selection probability of each of the expert networks and the number of the expert networks for the images input in the training batch, the auxiliary loss calculation of the feature extraction module is performed to obtain the load balancing loss; Based on the probability index data obtained by the gating network performing type estimation on the fused feature, the auxiliary loss calculation of the feature extraction module is performed to obtain the router z loss, wherein the probability index data includes the probability that the image corresponding to the fused feature belongs to each of the expert networks; For the sample image-text pair data of the multilingual text type, based on the output results of the multilingual version in the first output result, a cross-language consistency loss is calculated to obtain the cross-language consistency loss; The contrast loss, the load balancing loss, the router z loss and the cross-language consistency loss are integrated to obtain the first model loss.

10. The method according to claim 5, characterized in that The method for obtaining the second model loss includes: generating a cross entropy loss based on a difference between the second output result and the first sample annotation, and a difference between the third output result and the second sample annotation; Based on the selection probability of each of the expert networks and the number of the expert networks for the images input in the training batch, the auxiliary loss calculation of the feature extraction module is performed to obtain the load balancing loss; Based on the probability index data obtained by the gating network performing type estimation on the fused feature, the auxiliary loss calculation of the feature extraction module is performed to obtain the router z loss, wherein the probability index data includes the probability that the image corresponding to the fused feature belongs to each of the expert networks; For the sample image-text pair data or text data of the multilingual text type, based on the output result of the multilingual version in the second output result and the output result of the multilingual version in the third output result, a cross-language consistency loss is calculated to obtain the cross-language consistency loss; The cross entropy loss, the load balancing loss, the router z loss and the cross-language consistency loss are integrated to obtain the second model loss.

11. A model pre-training device for multilingual tasks, characterized in that: The device comprises: Acquisition module: used to acquire a multimodal training data set and an initial model, wherein the training data set includes a plurality of sample text data and a plurality of sample image-text pair data, wherein the plurality of sample image-text pair data and the plurality of sample text data include content in multiple languages, and the plurality of sample image-text pair data include sample image-text pair data in a general field and sample image-text pair data in a preset business field in a target scene, wherein the plurality of sample text data include text data in the preset business field, and the initial model includes a visual encoder, a projection module and a decoding module connected in sequence, wherein the decoding module is constructed based on a large language model; A first training module: used for performing comparative learning training of aligning visual features and text features on the initial model based on the plurality of sample image-text pair data, freezing the model parameters of the decoding module and adjusting the model parameters of the visual encoder and the projection module during the training process until a first end condition is met; A second training module: used for performing content understanding constraint training on the initial model that meets the first end condition based on the multiple sample image-text pair data and the multiple sample text data, and adjusting the model parameters of the visual encoder, the projection module and the decoding module during the training process until the second end condition is met; Model generation module: used to determine the initial model that meets the second end condition as the target model; during the training process, the text and sample text data in the sample image and text data of the preset business field are segmented in combination with the target vocabulary corresponding to the preset business field to serve as the input of the projection module.

Images