Method, apparatus and computer program product for data enhancement

Abstract

The embodiment of the invention relates to a method, equipment and a computer program product for data enhancement. The method includes generating an image embedding based on an image in the unstructured document, and generating a text embedding based on text associated with the image in the unstructured document. The method further includes obtaining descriptive information from the repository based on the generated image embedding and text embedding. The method also includes adding the obtained descriptive information to the unstructured document. In this way, the non-structured document can be understood and analyzed in a cross-modal mode, the representation of the multi-modal data in the non-structured document can be utilized to enrich the non-structured document, and therefore the data quantity and diversity are improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate generally to the field of computers, and more particularly to methods, devices, and computer program products for data enhancement. Background Art

[0002] Unstructured data refers to data that has an irregular or incomplete data structure, no predefined data model, or cannot or is difficult to represent using a two-dimensional logical table in a database. With the rapid growth of data, unstructured data is growing explosively. These unstructured data include information in various forms, such as text, images, audio, video, etc.

[0003] Compared with formatted data, unstructured data is diverse and flexible because it contains multiple forms and is not restricted by fixed data models. In addition, unstructured data contains a large amount of information. Through data mining technology, valuable information can be extracted from it for decision-making and judgment. In order to better process and analyze unstructured data, it is necessary to continuously explore new technologies and methods. Summary of the invention

[0004] The embodiments of the present disclosure provide a solution for data enhancement, through which unstructured documents can be understood and analyzed cross-modally, and representations of multimodal data in unstructured documents can be used to enrich unstructured documents, thereby increasing the quantity and diversity of data.

[0005] In a first aspect of the present disclosure, a method for data enhancement is provided, the method comprising generating an image embedding based on an image in an unstructured document, and generating a text embedding based on text associated with the image in the unstructured document. The method further comprises obtaining descriptive information from a repository based on the generated image embedding and text embedding. The method further comprises adding the obtained descriptive information to the unstructured document.

[0006] In another aspect of the present disclosure, an electronic device is provided, the device including a processor and a memory, the memory being coupled to the processor and having instructions stored thereon, the instructions, when executed by the processor, causing the device to perform actions, the actions including generating an image embedding based on an image in an unstructured document, and generating a text embedding based on text associated with the image in the unstructured document. The actions also include obtaining descriptive information from a repository based on the generated image embedding and text embedding. The actions also include adding the obtained descriptive information to the unstructured document.

[0007] In another aspect of the present disclosure, a computer program product is provided, which is tangibly stored on a non-transitory computer-readable storage medium and includes computer-executable instructions, which, when executed, cause a computer to perform a method or process according to an embodiment of the present disclosure.

[0008] Please note that the invention summary is provided to introduce a series of concepts in a simplified form, which will be further described in the detailed description below. The invention summary is not intended to identify key features or essential features of the present disclosure, nor is it intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other purposes, features and advantages of the present disclosure will become more clearly understood by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings, in which:

[0010] Figure 1 A schematic diagram illustrating an example environment in which methods and / or processes according to embodiments of the present disclosure may be implemented;

[0011] Figure 2 A flow chart of a method for data enhancement according to an embodiment of the present disclosure is illustrated;

[0012] Figure 3 is a schematic diagram illustrating a data enhancement process for unstructured documents according to an embodiment of the present disclosure;

[0013] Figure 4 is a diagram illustrating an embedding generation process for multimodal data according to an embodiment of the present disclosure;

[0014] Figure 5 is a diagram illustrating an index building process according to the present disclosure;

[0015] Figure 6 is a diagram illustrating an example of a storage repository according to an embodiment of the present disclosure;

[0016] Figure 7 illustrates the interaction between a plurality of unified block-like storage container groups according to an embodiment of the present disclosure; and

[0017] Figure 8 is a schematic block diagram of an example device that may be used to implement embodiments according to the present disclosure.

[0018] Throughout the drawings, same or similar reference numbers generally refer to same or similar elements. Specific embodiments

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] In the description of the embodiments of the present disclosure, the term "including" and its variations should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects, unless explicitly indicated to be different.

[0021] As mentioned above, with the rapid growth of data, unstructured data has shown explosive growth. Unstructured data has various forms and has important applications in many fields, such as natural language processing, image recognition, speech recognition, etc. Unstructured data is diverse and flexible, and by processing and analyzing these data, such as text mining, sentiment analysis, image recognition, etc., a large amount of usable information can be obtained, such as for decision support. In order to more effectively process and analyze unstructured data, it is necessary to continuously study and adopt improved technical means.

[0022] Unstructured documents are a representative type of unstructured data. Examples of unstructured documents include presentation documents in office software. Data enhancement for unstructured documents is an important application that can enrich the content of documents, thereby improving the quality and usability of documents. However, there are still some challenges and difficulties during data enhancement for unstructured documents.

[0023] Unstructured documents are diverse and flexible because they do not have a fixed and consistent format and pattern. In addition, unstructured documents generally have more than one type of unstructured data, such as text, images, audio, video, etc. In view of the characteristics of unstructured documents, traditional technologies cannot well understand such multimodal unstructured data in unstructured documents, and thus cannot well characterize such multimodal unstructured data. In addition, in traditional solutions, it is also difficult to establish a knowledge graph or knowledge base for such multimodal unstructured data in unstructured documents, resulting in the inability to search or retrieve corresponding knowledge for data enhancement of unstructured documents.

[0024] In order to solve at least some of the above and other potential problems, an embodiment of the present disclosure proposes a scheme for data enhancement. The scheme includes generating image embeddings based on images in unstructured documents, and generating text embeddings based on text associated with the images in unstructured documents. The scheme also includes obtaining descriptive information from a repository based on the generated image embeddings and text embeddings. The scheme also includes adding the obtained descriptive information to the unstructured document. In this way, not only can unstructured documents be understood and analyzed cross-modally, but also the representation of multimodal data in unstructured documents can be used to enrich unstructured documents, thereby increasing the amount and diversity of data.

[0025] Reference below Figures 1 to 8 It should be understood that these exemplary embodiments are provided only to enable those skilled in the art to better understand and implement the embodiments of the present disclosure, and are not intended to limit the scope of the present disclosure in any way.

[0026] Figure 1 1 is a schematic diagram of an example environment 100 in which methods and / or processes according to embodiments of the present disclosure may be implemented. Figure 1 As shown in , the example environment 100 may include an input 110, a computing device 120, a storage device 130, and a display device 140. It should be understood that limited components are shown in the example environment 100 for implementing the solution according to the embodiment of the present disclosure only for the purpose of facilitating understanding and ease of illustration, and the embodiment of the present disclosure is not limited thereto. For example, the example environment 100 may also include other different components that are beneficial to or promote the implementation of the solution according to the embodiment of the present disclosure, and the present disclosure is not limited thereto.

[0027] According to an embodiment of the present disclosure, input 110 may be transmitted to a computing device 120 for processing. In addition, input 110 may also be stored in a storage device 130 for prior knowledge of subsequent processing. Input 110 may be or include an unstructured document, such as a presentation document or a slide document in office software. It should be noted that the examples of unstructured documents according to embodiments of the present disclosure are not limited thereto, and may also include other different unstructured documents such as text documents, social media, and emails. The scheme for data enhancement according to embodiments of the present disclosure may also be applied to these unstructured documents.

[0028] According to an embodiment of the present disclosure, the computing device 120 may be configured to perform data enhancement for the input 110. During the data enhancement for the input 110, the computing device 120 may perform preprocessing on the unstructured document included in the input 110, and then may perform various detections and recognitions on the preprocessed unstructured document. In addition, the computing device 120 may obtain corresponding knowledge data from the storage device 130 based on the results of these operations to perform data enhancement on the unstructured document. It should be noted that the computing device 120 Figure 1 In the figure, one computing device is shown, but this is only illustrative and not restrictive, and a greater number of computing devices may exist.

[0029] By way of example and not limitation, computing device 120 may include, but is not limited to, a personal computer, a server computer, a handheld or laptop device, a mobile device (such as a mobile phone, a personal digital assistant (PDA), a multimedia player, etc.), a multi-processor system, consumer electronics, wearable electronic devices, smart home devices, small computing devices, large computing devices, edge computing devices, a distributed computing environment including any one or more of the above devices or systems, and the like.

[0030] According to an embodiment of the present disclosure, the storage device 130 may be configured to store the input 110 and the results of various operations performed by the computing device 120 on the input 110, such as preprocessing, detection, recognition, etc. The storage device 130 may also be configured to store knowledge data and the relationship between the unstructured document and the corresponding knowledge data. These stored data and information form a repository for the unstructured document. It should be noted that the storage device 130 Figure 1 In the figure, one storage device is shown, but this is only illustrative and not restrictive, and a greater number of storage devices may exist.

[0031] By way of example and not limitation, the storage device 130 may include but is not limited to a local storage device, a remote storage device, etc. In some embodiments, multiple storage devices in the storage device 130 may include but are not limited to a mechanical hard disk HDD, a solid state disk SSD, etc., and some of the multiple storage devices may be arranged locally, while others may be arranged remotely, for example, coupled together via a line or a network, etc.

[0032] The display device 140 can be configured to display the unstructured document itself without data enhancement, and can also display the unstructured document with data enhancement. In some embodiments, the display device 140 can be integrated into the computing device 120 to reduce the volume and occupied space of the overall device. By way of example and not limitation, the display device 140 can include but is not limited to a liquid crystal display (LCD), an organic light emitting diode display (OLED), a projector, etc.

[0033] Combined with the above Figure 1 A schematic diagram of an example environment 100 is depicted in which methods and / or processes according to embodiments of the present disclosure may be implemented. Figure 2 A flowchart of a method 200 for data enhancement according to an embodiment of the present disclosure is described. In order to better understand and analyze unstructured documents and enrich the content of unstructured documents, a method 200 for data enhancement according to an embodiment of the present disclosure is proposed.

[0034] At 210, an image embedding is generated based on the image in the unstructured document included in the input 110, and a text embedding is generated based on the text associated with the image in the unstructured document. Through the generated image embedding and text embedding, the image included in the unstructured document and the text associated therewith can be effectively represented.

[0035] Here, the embedding generation for the image and the text associated therewith at 210 can be implemented at the computing device 120 by a pre-trained multimodal deep learning neural network, wherein the training data may include paired images and texts, and the training process is collaborative. That is, a collaborative supervised or semi-supervised learning. In the following, the embedding generation process and the model training process according to the embodiment of the present disclosure will be further described in detail.

[0036] At 220, descriptive information is obtained from the repository based on the generated image embedding and text embedding. As described above, the generated image embedding and text embedding can effectively characterize the image and the text associated therewith in the unstructured document. Here, based on the generated image embedding and text embedding, corresponding descriptive information can be obtained from the repository.

[0037] The storage device 130 may include a repository that stores knowledge data, and these prior knowledge can help enrich unstructured documents. The knowledge data stored in the repository may include descriptive information, which corresponds to the corresponding image or text or a combination thereof, and indicates the corresponding image and text. In some embodiments, the descriptive information may include structured information. Hereinafter, the descriptive information acquisition process according to an embodiment of the present disclosure will be further described in detail.

[0038] At 230, the obtained descriptive information is added to the unstructured document. Through the generated image embedding and text embedding, the corresponding descriptive information can be found from the repository storing the knowledge data. The unstructured document can be enriched with the found descriptive information. In some embodiments, the unstructured document with the added corresponding descriptive information can be displayed at the reality device 140.

[0039] Through the method 200 for data enhancement according to an embodiment of the present disclosure, it is not only possible to understand and analyze unstructured documents cross-modally, but also possible to enrich the unstructured documents by utilizing the representation of multimodal data in the unstructured documents, thereby increasing the amount and diversity of data.

[0040] Figure 3 FIG. 3 is a schematic diagram illustrating a data enhancement process 300 for unstructured documents according to an embodiment of the present disclosure. Figure 3 Only limited operations and components are shown. However, it should be understood that the embodiments of the present disclosure may also include other different operations and components that are beneficial to or promote the implementation of the data enhancement process 300 for unstructured documents, and the present disclosure is not limited thereto.

[0041] According to an embodiment of the present disclosure, images and text may be extracted 320 from an unstructured document 310. Figure 3 In the example shown in , the unstructured document 310 is shown as a presentation document, which is exemplary and non-limiting. Examples of the unstructured document 310 may also include text documents, videos, audios, etc., and the data enhancement process 300 for unstructured documents according to an embodiment of the present disclosure is also applicable to them. At 320, for example, the 310 image and text may be extracted by optical character recognition (OCR) technology, image recognition technology, natural language processing technology, and speech recognition technology.

[0042] According to an embodiment of the present disclosure, the extracted image and text may be processed at 330. In some embodiments, object detection 331 may be performed to detect object entities from the extracted image, and text recognition 332 may be performed to identify label entities from the extracted text. Object entities are relatively representative objects in an image, such as faces in photos including people. Label entities are relatively representative texts in a text, such as titles or important words in an article, etc. Object detection 331 and text recognition 332 may be based on target detection algorithms (such as YOLOv3) and named entity recognition (NER), for example. For example, a NER machine learning model may detect "December 19" as a date.

[0043] According to an embodiment of the present disclosure, the object entity detected at 331 can be encoded as an image embedding, and the label entity identified at 332 can be encoded as a text embedding. In some embodiments, the encoding for the object entity is performed by an image encoder corresponding to the category type of the object entity among multiple image encoders, and the encoding for the label entity is performed by a text encoder corresponding to the category type of the label entity among multiple text encoders, wherein the multiple image encoders and the multiple text encoders are pre-coordinated based on training data including image-text pairs, wherein the image-text pairs include images and texts associated with each other. Here, multiple image encoders, multiple text encoders, and image-text pairs for training can be stored in a repository. Hereinafter, this will be described in further detail.

[0044] Figure 4 4 is a diagram illustrating an embedding generation process 400 for multimodal data according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, using contrastive language-image pre-training (CLIP), visual concepts can be effectively learned from natural language supervision. Based on CLIP, an image encoder and a text encoder are pre-trained to predict which images are paired with which texts.

[0045] According to an embodiment of the present disclosure, the object entity detected from the extracted image, i.e. Figure 4 The object entity banana 410 exemplarily shown in FIG. 4 is encoded into an image embedding (also known as an object density vector) via a pre-trained image encoder 430 for the “fruit” type, and the label entity identified from the extracted text, i.e. Figure 4 The label entity "banana" 420 exemplarily shown in the figure is encoded as a text embedding (also referred to as a label density vector) via a pre-trained text encoder 440 for the "fruit" type. Here, the object entity banana 410 and the label entity "banana" 420 are associated with each other. The model training process according to an embodiment of the present disclosure can be based on collaborative supervision or semi-supervised learning, and an accurate and robust encoder can be obtained through a large number of available supervision sources. It should be noted that the object entity banana 410 and the label entity "banana" 420 are only simple examples to help understand the present disclosure.

[0046] According to an embodiment of the present disclosure, descriptive information or structured information can be accessed through image embedding and text embedding. Image embedding and text embedding can be stored in a repository as an index of knowledge entities. Figure 5 An index building process 500 according to an embodiment of the present disclosure is described.

[0047] Figure 55 is a diagram illustrating an index building process 500 according to the present disclosure. An image 510 and a text 520 are encoded into an image embedding 550 and a text embedding 560 via an image encoder 530 corresponding to the type of the image 510 and a text encoder 540 corresponding to the type of the text 520, respectively. Figure 5 In the example shown in , image 510 and text 520 can be object entities detected from the image and label entities recognized from the text, respectively. In some embodiments, image embedding 550 and text embedding 560, and descriptive information corresponding to image embedding 550 or text embedding 560 or a combination thereof are stored as JSON files in the form of triples as knowledge data and knowledge entities for subsequent data enhancement for unstructured documents.

[0048] return Figure 3 After obtaining the image embedding and text embedding representing the image and the text associated therewith in the unstructured document 310, the repository 350 can be searched or retrieved based on the obtained image embedding and text embedding to obtain descriptive information that can be used to enrich the unstructured document 310. Figure 3 As shown in , the repository 350 may include a knowledge graph 351 and a knowledge base 352 to store knowledge entities and indicate the relationships between them. Figure 6 The storage repository 350 according to an embodiment of the present disclosure is described in further detail.

[0049] Figure 6 6 is a diagram illustrating an example 600 of a repository according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the repository may include a data repository 610 configured to store image-text pairs including images and texts associated with each other, and descriptive information corresponding to one of the images and texts associated with each other included in the image-text pairs, wherein the descriptive information indicates the images and texts associated with each other in the image-text pairs.

[0050] According to an embodiment of the present disclosure, a repository may include a model repository 620, which is configured to store a plurality of image encoders and a plurality of image encoders, each of the plurality of image encoders corresponding to a corresponding type of an object entity of an image, and each of the plurality of text encoders corresponding to a corresponding category type of a label text entity of a text.

[0051] According to an embodiment of the present disclosure, the repository may include a feature repository 630, which is configured to store multiple multimodal embeddings, each of which corresponds to a corresponding image-text pair. Various embedding representations for characterizing an image and a text corresponding thereto as described above may be stored in the feature repository 630 in the repository.

[0052] Return again Figure 3 At the embedding-based multimodal search 360, in order to roughly screen out relatively relevant knowledge data, according to an embodiment of the present disclosure, based on text embedding, a first predetermined number of multimodal embeddings are filtered out from a plurality of multimodal embeddings stored in the feature repository 630. Text embedding has a strong identification, and by utilizing the characteristics of text embedding, obviously unrelated knowledge entities can be excluded.

[0053] According to an embodiment of the present disclosure, based on image embedding and text embedding, a multimodal embedding indicating images and texts in an unstructured document can be generated, so that multimodal data such as images and texts can be characterized cross-modally. By comparing the generated multimodal embedding with a first predetermined number of multimodal embeddings, the similarity of the multimodal embedding to each multimodal embedding in the first predetermined number of multimodal embeddings can be determined. Next, a second predetermined number of multimodal embeddings in the first predetermined number of multimodal embeddings whose similarity to the multimodal embedding is higher than a predetermined similarity threshold can be determined as another multimodal embedding associated with the multimodal embedding, wherein the second predetermined number is less than the first predetermined number. In this way, by, for example, using a nearest neighbor search, it is possible to search or retrieve more similar or most similar knowledge entities. If the similarity is within a specified threshold, we enrich the identified object entities and label entities with, for example, the structured information of the most similar knowledge entity.

[0054] According to an embodiment of the present disclosure, an image-text pair corresponding to the determined additional multimodal embedding may be determined, and descriptive information corresponding to the determined image-text pair may be determined. After determining the associated multimodal embeddings, corresponding descriptive information may be determined from the repository 350 based on these multimodal embeddings. These corresponding descriptive information may then be added to the unstructured document 310 to complete data enhancement for the unstructured document 310.

[0055] According to an embodiment of the present disclosure, the images and texts in the unstructured document 310 may be stored in the data repository 610 in association with the descriptive information determined at 360, and the multimodal embeddings for the images and texts in the unstructured document 310 may be stored in the feature repository 630 as metadata for subsequent data enhancement processes.

[0056] Figure 7 The diagram of an example 700 of an unstructured document after adding descriptive information according to an embodiment of the present disclosure is illustrated. Take a page of a presentation document as an example, which includes an object entity banana 410 and a label entity "banana" 420 corresponding thereto. According to the data enhancement for unstructured documents of an embodiment of the present disclosure, descriptive information 710 is added to the page of the presentation document. In some embodiments, the added descriptive information 710 can be editable. In addition, in addition to the examples shown, the descriptive information 710 can also include structured information, such as table data, etc.

[0057] Figure 8 8 is a schematic block diagram of an example device 800 that can be used to implement some embodiments of the present disclosure. Figure 8 As shown in FIG. 8 , the device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 802 or loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0058] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0059] The various processes and processing described above, such as method 200, may be performed by processing unit 801. For example, in some embodiments, method 200 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU 801, one or more actions of method 200 described above may be performed.

[0060] The present disclosure may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0061] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive lists) of computer readable storage medium include: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disk read-only memories (CD-ROM), digital versatile disks (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or convex structures in grooves on which instructions are stored, and any suitable combination thereof. Computer readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (e.g., light pulses by optical fiber cables), or electrical signals transmitted by wires.

[0062] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0063] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0064] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0065] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0066] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0067] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0068] The embodiments of the present disclosure are described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for data enhancement, comprising: Generate image embeddings based on images in unstructured documents; generating a text embedding based on text in the unstructured document associated with the image; Retrieving descriptive information from a repository based on the generated image embedding and the generated text embedding; as well as The acquired descriptive information is added to the unstructured document.

2. The method of claim 1, wherein generating the image embedding comprises: extracting the image from the unstructured document; Detecting an object entity from the extracted image; as well as The detected object entity is encoded as the image embedding.

3. The method of claim 2, wherein generating the text embedding comprises: extracting the text from the unstructured document; Identifying tag entities from the extracted text; as well as The identified tag entities are encoded as the text embeddings.

4. The method according to claim 3, wherein: The encoding for the object entity is performed by an image encoder corresponding to the type of the object entity among a plurality of image encoders, and the encoding for the tag entity is performed by a text encoder corresponding to the type of the tag entity among a plurality of text encoders, wherein the plurality of image encoders and the plurality of text encoders are collaboratively trained in advance based on training data including image-text pairs, wherein the image-text pairs include images and texts associated with each other.

5. The method according to claim 1, wherein the storage repository comprises: a data repository configured to store an image-text pair including an image and a text associated with each other, and descriptive information corresponding to one of the image and the text associated with each other included in the image-text pair, wherein the descriptive information indicates the image and the text associated with each other in the image-text pair; a model repository configured to store a plurality of image encoders and a plurality of image encoders, each of the plurality of image encoders corresponding to a corresponding type of an object entity of an image, and each of the plurality of text encoders corresponding to a corresponding type of a tag entity of a text; as well as A feature repository is configured to store a plurality of multimodal embeddings, each of the plurality of multimodal embeddings corresponding to a respective image-text pair.

6. The method according to claim 5, further comprising: Based on the text embeddings, a first predetermined number of multimodal embeddings are filtered out from the plurality of multimodal embeddings stored in the feature repository.

7. The method according to claim 6, further comprising: generating, based on the image embedding and the text embedding, a multimodal embedding indicating the image and the text in the unstructured document; determining a similarity of the multimodal embedding to each of the first predetermined number of multimodal embeddings by comparing the generated multimodal embedding to the first predetermined number of multimodal embeddings; as well as A second predetermined number of multimodal embeddings among the first predetermined number of multimodal embeddings having a similarity with the multimodal embedding above a predetermined similarity threshold is determined as additional multimodal embeddings associated with the multimodal embedding, wherein the second predetermined number is smaller than the first predetermined number.

8. The method of claim 7, wherein obtaining the descriptive information from the repository comprises: determining an image-text pair corresponding to the additional multimodal embedding; as well as The descriptive information corresponding to the determined image-text pair is determined.

9. The method according to claim 7, further comprising: storing the image and the text in the unstructured document in association with the descriptive information in the data repository; The multimodal embedding is stored in the feature repository.

10. The method of claim 1, wherein: The added descriptive information is editable, and the added descriptive information includes structured information.

11. An electronic device, comprising: processor; as well as a memory coupled to the processor and storing instructions that, when executed by the processor, cause the device to perform actions comprising: Generate image embeddings based on images in unstructured documents; generating a text embedding based on text in the unstructured document associated with the image; Retrieving descriptive information from a repository based on the generated image embedding and the generated text embedding; and The acquired descriptive information is added to the unstructured document.

12. The electronic device of claim 11, wherein generating the image embedding comprises: extracting the image from the unstructured document; Detecting an object entity from the extracted image; as well as The detected object entity is encoded as the image embedding.

13. The electronic device of claim 12, wherein generating the text embedding comprises: extracting the text from the unstructured document; Identifying tag entities from the extracted text; as well as The identified tag entities are encoded as the text embeddings.

14. The electronic device according to claim 13, wherein: The encoding for the object entity is performed by an image encoder corresponding to the type of the object entity among a plurality of image encoders, and the encoding for the tag entity is performed by a text encoder corresponding to the type of the tag entity among a plurality of text encoders, wherein the plurality of image encoders and the plurality of text encoders are collaboratively trained in advance based on training data including image-text pairs, wherein the image-text pairs include images and texts associated with each other.

15. The electronic device according to claim 11, wherein the storage repository comprises: a data repository configured to store an image-text pair including an image and a text associated with each other, and descriptive information corresponding to one of the image and the text associated with each other included in the image-text pair, wherein the descriptive information indicates the image and the text associated with each other in the image-text pair; a model repository configured to store a plurality of image encoders and a plurality of image encoders, each of the plurality of image encoders corresponding to a corresponding type of an object entity of an image, and each of the plurality of text encoders corresponding to a corresponding type of a tag entity of a text; as well as A feature repository is configured to store a plurality of multimodal embeddings, each of the plurality of multimodal embeddings corresponding to a respective image-text pair.

16. The electronic device according to claim 15, wherein the actions further comprise: Based on the text embeddings, a first predetermined number of multimodal embeddings are filtered out from the plurality of multimodal embeddings stored in the feature repository.

17. The electronic device according to claim 16, wherein the actions further comprise: generating, based on the image embedding and the text embedding, a multimodal embedding indicating the image and the text in the unstructured document; determining a similarity of the multimodal embedding to each of the first predetermined number of multimodal embeddings by comparing the generated multimodal embedding to the first predetermined number of multimodal embeddings; as well as A second predetermined number of multimodal embeddings among the first predetermined number of multimodal embeddings having a similarity with the multimodal embedding above a predetermined similarity threshold is determined as additional multimodal embeddings associated with the multimodal embedding, wherein the second predetermined number is smaller than the first predetermined number.

18. The electronic device of claim 17, wherein obtaining the descriptive information from the repository comprises: determining an image-text pair corresponding to the additional multimodal embedding; as well as The descriptive information corresponding to the determined image-text pair is determined.

19. The electronic device according to claim 17, wherein the actions further comprise: storing the image and the text in the unstructured document in association with the descriptive information in the data repository; The multimodal embedding is stored in the feature repository.

20. A computer program product tangibly stored on a non-transitory computer readable medium and comprising computer executable instructions which, when executed, cause a computer to: Generate image embeddings based on images in unstructured documents; generating a text embedding based on text in the unstructured document associated with the image; Retrieving descriptive information from a repository based on the generated image embedding and the generated text embedding; as well as The acquired descriptive information is added to the unstructured document.

Images