Document language image-text data set construction method and device for internet data and medium

Abstract

The invention relates to a method and equipment for constructing a minority language image-text data set for internet data and a medium, and the method comprises the steps: obtaining an HTML webpage file, extracting a multi-modal document containing minority language image-text data from the HTML webpage file, and screening out image-text related pairs according to replaceable texts corresponding to images in the multi-modal document; inputting a pure picture which does not contain a replaceable text in the multi-modal document, the text information after the picture is removed and a self-defined cue word into a first multi-modal large model, generating corresponding picture description information and image-text question and answer information, forming an image-text description pair by the pure picture and the picture description information, and displaying the image-text description pair in the first multi-modal large model; forming a visual question-answer pair by the pure picture and the image-text question-answer information; inputting the multi-modal document, the image-text related pair, the image-text description pair and the visual question-answer pair into a second multi-modal large model, and outputting an image scene label; and combining the information to obtain a small language image-text data set. Compared with the prior art, the method has the advantages that high-quality, diversified and standardized small language image-text data set construction can be realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method, device and medium for constructing a minority language graphic and text data set for Internet data. Background Art

[0002] By training the large model on multimodal image and text datasets and fusing visual and text information, it not only provides the large model with rich information and context, but also enables it to master and perform complex visual and language tasks. This is crucial for cultivating a large model that can understand and process multimodal inputs, and can achieve significant performance improvements in key areas such as image recognition, image description generation, and visual question answering. It is currently widely used in fields including autonomous driving, medical image analysis, and smart education.

[0003] However, most current image and text datasets focus on mainstream languages, such as English and Chinese, while coverage of minority languages ​​is relatively insufficient, and cannot meet the growing demand for multilingual processing. The lack of image and text datasets in minority languages ​​limits model training and application development in these languages, causing minority language countries to lag behind in the development and innovation of big model technology, and limiting the benefits of minority language countries and communities from advanced artificial intelligence technologies. In addition, minority languages ​​are often closely linked to specific cultures and communities, and the lack of datasets in these languages ​​means that the important value of big models in cultural diversity is ignored. In addition, the annotation quality of existing image and text datasets varies and there is a lack of standardized processes, which affects the practicality and reliability of the datasets.

[0004] Therefore, continuing to design a method for constructing a minority language image and text dataset can overcome the shortcomings of existing minority language image and text datasets, especially the limitations in data diversity, annotation quality and availability. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a method, device and medium for constructing a minority language graphic and text dataset for Internet data, which can realize the construction of high-quality, diversified and standardized minority language graphic and text datasets.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to a first aspect of the present invention, a method for constructing a minority language graphic and text dataset for Internet data is provided, the method comprising:

[0008] Obtaining an HTML webpage file from the Internet, extracting a multimodal document containing image and text data in a minority language from the HTML webpage file, and obtaining image-text related pairs according to replaceable texts corresponding to images in the multimodal document;

[0009] Inputting a pure image without replaceable text in a multimodal document, text information after removing the image, and a custom prompt word into a first multimodal large model, generating corresponding image description information and image-text question-answer information, forming an image-text description pair with the pure image and the corresponding image description information, and forming a visual question-answer pair with the pure image and the corresponding image-text question-answer information;

[0010] Input the multimodal documents, image-text pairs, image-text description pairs, and visual question-answer pairs into the second multimodal large model, and output the image scene label;

[0011] The minority language image and text dataset is obtained by combining multimodal documents, image-text pairs, image-text description pairs, visual question-answer pairs, and image scene labels.

[0012] Preferably, after obtaining the minority language HTML web page file, the method further includes: using a document object model tree cleaning strategy to simplify the minority language HTML web page file by removing irrelevant text and images; wherein the simplified minority language HTML web page file retains the structure of the original minority language HTML web page file.

[0013] Preferably, the text description information associated with the image is extracted from the small language HTML web page file through the Alt-text attribute of the HTML web page, and the text context information corresponding to the image is extracted from the small language HTML web page file using the paragraph, title, and list information of the text surrounding the image.

[0014] Preferably, the method further comprises filtering the multimodal document, wherein the filtering comprises image screening, text paragraph screening, document level screening, and image and text deduplication processing.

[0015] Preferably, after extracting the multimodal document containing graphic and text data in a minority language, the method further includes extracting background information from the images in the multimodal document, which is used to subsequently input the background information into the first multimodal large model together with the pure image, the text information after removing the image, and the custom prompt words to obtain annotation information, wherein the annotation information includes graphic and text description pairs and visual question and answer pairs; the background information includes human body detection information, minority language OCR recognition information, object detection information, object relationship detection information, and image tone detection information.

[0016] Preferably, before the text information after removing the picture is sent to the first multimodal large model, it also includes translating the minority language text information into Chinese text information; after being processed by the first multimodal large model, the Chinese picture description information and Chinese picture and text question and answer information output by the first multimodal large model are translated into the corresponding minority language and then output.

[0017] Preferably, the visual question-answer pair covers key image information, including object, action, scene and context information.

[0018] Preferably, the method further comprises: performing grammar checking of minority languages ​​on multimodal documents, image-text related pairs, image-text description pairs, and visual question-answer pairs.

[0019] According to a second aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, any one of the methods described above is implemented.

[0020] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, any one of the methods described above is implemented.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1) The present invention uses a multimodal large model to clean the image and text data in the minority language HTML web page files, and at the same time annotates the image and text data, thereby constructing a minority language image and text data set with higher quality and data diversity.

[0023] 2) The present invention adopts a document object model tree cleaning strategy to simplify the processing of HTML web page files in small languages ​​by removing irrelevant texts and images, thereby improving the availability of data in the document object model tree.

[0024] 3) The text description associated with the image is extracted through the Alt-text attribute in the HTML web page. By analyzing the paragraphs, titles, lists and images around the image, more comprehensive contextual information can be obtained, thereby improving the data quality of the minority language image and text dataset.

[0025] 4) When the present invention performs multimodal large model annotation, it additionally introduces background information extracted from the image, human body detection information, minority language OCR recognition information, object detection information, object relationship detection information, and image tone detection information, which can generate more accurate image-text description pairs and visual question-answer pairs, thereby improving the quality of minority language image-text datasets.

[0026] 5) By performing minority language grammar checking on multimodal documents, image-text pairs, image-text description pairs, and visual question-answer pairs, the available accuracy of data in minority language datasets is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of the method of the present invention;

[0028] Figure 2 The processing process of the minority language image and text data set of the present invention;

[0029] Figure 3 It is a schematic diagram of the process of generating graphic description information and visual question and answer information in the embodiment;

[0030] Figure 4 Schematic diagram of the process of obtaining description prompt words and visual question and answer prompt words. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] Example

[0033] like Figure 1 and Figure 2 As shown, this embodiment provides a method for constructing a minority language graphic and text dataset for Internet data, the method comprising:

[0034] S1. Obtain an HTML web page file from the Internet, pre-process the HTML web page file, extract a multimodal document containing image and text data in a minority language from the HTML web page file, and obtain image-text related pairs based on replaceable texts corresponding to the images in the multimodal document.

[0035] The preprocessing process includes text deduplication, quality assessment and language identification to ensure the quality of the selected minority language content. Specifically, after obtaining the minority language HTML web page file, a refined document object model tree cleaning strategy is used to simplify the minority language HTML web page file by removing irrelevant text and images (such as advertisements, navigation bars, etc.), retaining only the core information, extracting multimodal documents containing text and images from the simplified HTML, and maintaining the structural integrity of the minority language HTML web page file to better understand the context of the content.

[0036] Through the Alt-text attribute of HTML web pages, text description information associated with images can be extracted from small language HTML web page files. By using the paragraphs, titles, and list information of the text around the images, the text context information corresponding to the images can be extracted from small language HTML web page files. By deeply mining the text content around the images, richer multimodal documents can be constructed.

[0037] After extracting multimodal documents containing minority language graphic and text data, we filter the multimodal documents, including image screening, text paragraph screening, document level screening, and image and text deduplication to ensure the uniqueness and diversity of the data set. In the implementation of the filtering strategy, we strictly abide by data privacy and copyright regulations to eliminate inappropriate content.

[0038] In addition, this step also includes extracting background information from the image in the multimodal document, which is then input into the first multimodal large model together with the pure image, the text information after removing the image, and the custom prompt words to obtain annotation information. The annotation information includes image-text description pairs and visual question-answer pairs, where Figure 3 As shown, the background information includes human body detection information, minority language OCR recognition information, object detection information, object relationship detection information, and image tone detection information.

[0039] S2. Input pure images without replaceable text in multimodal documents, text information after removing the images, and custom prompt words tailored for specific data sources into the first multimodal large model (visual language model VLM), generate corresponding image description information and image-text question and answer information, form a picture-text description pair with the pure image and the corresponding image description information, and form a visual question and answer pair with the pure image and the corresponding image-text question and answer information.

[0040] Visual Question and Answer (VQA) generation: Using the same large multimodal model and visual question and answer prompts, we can generate question and answer pairs related to the image. These question and answer pairs should cover the key information of the image, including objects, actions, scenes, and contextual information.

[0041] When processing images and text, multimodal large language models may produce so-called "hallucinations", that is, they generate outputs that are inconsistent with the actual visual content. This phenomenon is particularly common when the model generates longer answers. In order to solve the problem, background knowledge is added. The images are first pre-processed through proprietary models, such as human body detection, minority language OCR, object detection, object relationship detection, image hue detection, etc., and the scene labels of the image scenes during the cleaning process together constitute the background knowledge. By inputting the background knowledge and images into the large model and adding customized prompt words, accurate image-text description pairs and visual question-answer pairs are finally generated.

[0042] In addition, before the text information after removing the image is sent to the multimodal large model, it also includes translating the text information in the minority language into Chinese text information; after being processed by the first multimodal large model, the Chinese image description information and Chinese image and text question and answer information output by the first multimodal large model are translated into the corresponding minority language and then output.

[0043] S3: Input the multimodal documents, image-text pairs, image-text description pairs, and visual question-answer pairs into the second multimodal large model, and output the image scene labels.

[0044] S4. Multimodal documents, image-text pairs, image-text description pairs, visual question-answer pairs, and image scene labels are combined to obtain a minority language image-text dataset.

[0045] In addition, this embodiment also includes minority language grammar checking for multimodal documents, image-text related pairs, image-text description pairs, and visual question-answer pairs.

[0046] like Figure 4 The process of obtaining the description prompt words used in the process of generating the graphic description information shown in the figure specifically includes:

[0047] Collect high-quality pictures and annotate them using the VLM model;

[0048] Generate corresponding description words based on the image's country of origin and image scene annotation information, including:

[0049] Please carefully observe this picture from [specific minority language country name] and generate a detailed, accurate and rich Chinese description based on the main elements and scenes in the picture. The description should cover the following aspects: Please state the subject of the picture, such as people, specific objects or a certain place.

[0050] Please describe the subject's activities, characteristics and status, and the emotional atmosphere or story conveyed by the subject. Please describe the elements in the background of the picture and how these elements interact with the subject.

[0051] Please describe the cultural characteristics of [minority language country] shown in the image, such as traditional clothing, architecture, festivals or customs. Please describe the emotion or atmosphere you feel after seeing the image, and explain the special meaning of this emotion or atmosphere in the cultural context. If there are special details, such as light and shadow effects, color textures, artistic styles, handicrafts, etc., please include them in your description and explain their cultural significance. Please make sure that your description is both accurate and in-depth, can reflect the cultural characteristics of [specific minority language country name], and provide enough details so that readers can form a clear impression of the image.

[0052] The visual question answering prompt words in the process of obtaining the visual question answering prompt words include:

[0053] General:

[0054] What details in the image stand out? What do these details symbolize? What emotion or atmosphere does the image convey in the context of [minority language country]?

[0055] If the action or event in the picture continued, what would happen next?

[0056] Comprehensive category:

[0057] What is the main object or scene in this picture? What is the function or role of the elements in the picture? What is their quantity and position relationship?

[0058] Characters:

[0059] Who are the people in the picture? What relationships might they have? What activities are they performing?

[0060] Considering the [minority language country] context, what emotional state or story do the characters’ expressions, postures, and clothing convey?

[0061] What kind of scene does the scene picture describe? What are the overall color tone, lighting conditions and composition perspective of the picture?

[0062] What historical background and cultural significance does the picture involve in [minority language country]? What folk traditions or social phenomena does this picture reflect in [minority language country]? Are the elements in the picture related to a specific historical period or historical event?

[0063] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0064] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0065] The processing unit performs the various methods and processes described above. For example, in some embodiments, the method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the method described above can be performed. Alternatively, in other embodiments, the CPU can be configured to perform the method in any other appropriate manner (e.g., by means of firmware).

[0066] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip systems (SOCs), load programmable logic devices (CPLDs), and the like.

[0067] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0068] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0069] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for constructing a minority language graphic and text dataset for Internet data, characterized in that: The method includes: Obtaining an HTML webpage file from the Internet, extracting a multimodal document containing image and text data in a minority language from the HTML webpage file, and obtaining image-text related pairs according to replaceable texts corresponding to images in the multimodal document; Inputting a pure image without replaceable text in a multimodal document, text information after removing the image, and a custom prompt word into a first multimodal large model, generating corresponding image description information and image-text question-answer information, forming an image-text description pair with the pure image and the corresponding image description information, and forming a visual question-answer pair with the pure image and the corresponding image-text question-answer information; Input the multimodal documents, image-text pairs, image-text description pairs, and visual question-answer pairs into the second multimodal large model, and output the image scene label; The minority language image and text dataset is obtained by combining multimodal documents, image-text pairs, image-text description pairs, visual question-answer pairs, and image scene labels.

2. The method for constructing a minority language graphic and text dataset for Internet data according to claim 1, characterized in that: After obtaining the minority language HTML web page file, the method further includes: using a document object model tree cleaning strategy to simplify the minority language HTML web page file by removing irrelevant text and images; wherein the simplified minority language HTML web page file retains the structure of the original minority language HTML web page file.

3. The method for constructing a minority language graphic and text dataset for Internet data according to claim 1, characterized in that: The text description information associated with the image is extracted from the small language HTML web page file through the Alt-text attribute of the HTML web page, and the text context information corresponding to the image is extracted from the small language HTML web page file using the paragraph, title, and list information of the text surrounding the image.

4. The method for constructing a minority language graphic and text dataset for Internet data according to claim 1, characterized in that: The method also includes filtering the multimodal document, wherein the filtering includes image screening, text paragraph screening, document level screening, and image and text deduplication processing.

5. The method for constructing a minority language graphic and text dataset for Internet data according to claim 1, characterized in that: After extracting the multimodal document containing the image and text data in a minority language, the method further includes extracting background information from the image in the multimodal document, which is used to be subsequently input into the first multimodal large model together with the pure image, the text information after removing the image, and the custom prompt words to obtain annotation information, wherein the annotation information includes an image-text description pair and a visual question-answer pair; The background information includes human body detection information, minority language OCR recognition information, object detection information, object relationship detection information, and image tone detection information.

6. The method for constructing a minority language graphic and text dataset for Internet data according to claim 5, characterized in that: Before the text information after removing the image is sent to the first multimodal large model, it also includes translating the minority language text information into Chinese text information; after being processed by the first multimodal large model, the Chinese image description information and Chinese image and text question and answer information output by the first multimodal large model are translated into the corresponding minority language and then output.

7. The method for constructing a minority language graphic and text dataset for Internet data according to claim 1, characterized in that: The visual question-answer pairs cover key image information, including objects, actions, scenes, and context information.

8. The method for constructing a minority language graphic and text dataset for Internet data according to claim 1, characterized in that: The method also includes: performing minority language grammar checking on multimodal documents, image-text related pairs, image-text description pairs, and visual question-answer pairs.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images