Image-based entity relationship annotation model processing method and related equipment

Abstract

The embodiments of the present application belong to the field of artificial intelligence and relate to a method, apparatus, computer equipment, and storage medium for processing an entity relationship annotation model based on an image. The method comprises: obtaining a training image, whose entity annotation information includes text information and relationship information of each entity in the image; generating a text composite vector for each entity based on the text information, and generating an initial composite vector for the training image; performing separate convolution on the training image to obtain a convolution feature vector; merging the initial composite vector and the convolution feature vector to obtain an image composite vector; inputting the text composite vector and the image composite vector into an entity relationship annotation network to obtain entity relationship prediction information; calculating the model loss based on the relationship information and the entity relationship prediction information to train the model; and inputting the image to be annotated into the model to obtain entity relationship information. The present application also relates to blockchain technology, and the training images can be stored in the blockchain. The present application realizes the automatic recognition of entities in images and the automatic annotation of entity relationships.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, computer equipment and storage medium for processing an image-based entity relationship annotation model. Background Art

[0002] As a carrier of information, images contain richer information and are convenient to use. With the advancement of computer technology, we can now use computer vision algorithms to extract content information from images. Beyond extracting content information from images, we also want to explore the relationships within image content. For example, we can identify entities in images and obtain their information, attributes, and relationships. However, current entity relationship annotation is primarily implemented in text scenarios and cannot be directly implemented in image scenarios. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to propose an image-based entity relationship annotation model processing method, device, computer equipment and storage medium to realize entity relationship annotation in images.

[0004] In order to solve the above technical problems, the present application provides an image-based entity relationship annotation model processing method, which adopts the following technical solutions:

[0005] Acquire a training image with entity annotation information, wherein the entity annotation information includes text information and relationship information of each entity in the training image, the text information includes a text note value and coordinate information of each entity, and the relationship information includes entity relationships and association relationships between the entities;

[0006] Generating a text composite vector of each entity according to the text information, and generating an initial composite vector of the training image;

[0007] Inputting the text composite vector, the training image, and the initial composite vector into an initial entity relationship annotation model, so as to perform separation convolution processing on the training image through a convolutional network in the initial entity relationship annotation model to obtain a convolution feature vector;

[0008] Combining the initial composite vector and the convolution feature vector to obtain an image composite vector;

[0009] Inputting the text composite vector and the image composite vector into the entity relationship annotation network in the initial entity relationship annotation model to obtain entity relationship prediction information;

[0010] Calculating a model loss based on the relationship information and the entity relationship prediction information, so as to adjust parameters of the initial entity relationship labeling model according to the model loss until the model loss meets a training stop condition, thereby obtaining an entity relationship labeling model;

[0011] An image to be annotated is acquired, and the image to be annotated is input into the entity relationship annotation model to obtain entity relationship information.

[0012] In order to solve the above technical problems, the present application also provides an image-based entity relationship annotation model processing device, which adopts the following technical solutions:

[0013] An image acquisition module is configured to acquire training images with entity annotation information, wherein the entity annotation information includes text information and relationship information of each entity in the training image, the text information includes a text note value and coordinate information of each entity, and the relationship information includes entity relationships and association relationships between the entities;

[0014] a vector generation module, configured to generate a text composite vector of each entity according to the text information, and to generate an initial composite vector of the training image;

[0015] a convolution processing module, configured to input the text composite vector, the training image, and the initial composite vector into an initial entity relationship annotation model, so as to perform separation convolution processing on the training image through a convolutional network in the initial entity relationship annotation model to obtain a convolution feature vector;

[0016] A vector merging module, configured to merge the initial composite vector and the convolution feature vector to obtain an image composite vector;

[0017] A vector input module, configured to input the text composite vector and the image composite vector into an entity relationship annotation network in the initial entity relationship annotation model to obtain entity relationship prediction information;

[0018] A model adjustment module, configured to calculate a model loss based on the relationship information and the entity relationship prediction information, so as to adjust parameters of the initial entity relationship labeling model according to the model loss until the model loss satisfies a training stop condition, thereby obtaining an entity relationship labeling model;

[0019] The image annotation module is used to obtain the image to be annotated and input the image to be annotated into the entity relationship annotation model to obtain entity relationship information.

[0020] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0021] Acquire a training image with entity annotation information, wherein the entity annotation information includes text information and relationship information of each entity in the training image, the text information includes a text note value and coordinate information of each entity, and the relationship information includes entity relationships and association relationships between the entities;

[0022] Generating a text composite vector of each entity according to the text information, and generating an initial composite vector of the training image;

[0023] Inputting the text composite vector, the training image, and the initial composite vector into an initial entity relationship annotation model, so as to perform separation convolution processing on the training image through a convolutional network in the initial entity relationship annotation model to obtain a convolution feature vector;

[0024] Combining the initial composite vector and the convolution feature vector to obtain an image composite vector;

[0025] Inputting the text composite vector and the image composite vector into the entity relationship annotation network in the initial entity relationship annotation model to obtain entity relationship prediction information;

[0026] Calculating a model loss based on the relationship information and the entity relationship prediction information, so as to adjust parameters of the initial entity relationship labeling model according to the model loss until the model loss meets a training stop condition, thereby obtaining an entity relationship labeling model;

[0027] An image to be annotated is acquired, and the image to be annotated is input into the entity relationship annotation model to obtain entity relationship information.

[0028] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0029] Acquire a training image with entity annotation information, wherein the entity annotation information includes text information and relationship information of each entity in the training image, the text information includes a text note value and coordinate information of each entity, and the relationship information includes entity relationships and association relationships between the entities;

[0030] Generating a text composite vector of each entity according to the text information, and generating an initial composite vector of the training image;

[0031] Inputting the text composite vector, the training image, and the initial composite vector into an initial entity relationship annotation model, so as to perform separation convolution processing on the training image through a convolutional network in the initial entity relationship annotation model to obtain a convolution feature vector;

[0032] Combining the initial composite vector and the convolution feature vector to obtain an image composite vector;

[0033] Inputting the text composite vector and the image composite vector into the entity relationship annotation network in the initial entity relationship annotation model to obtain entity relationship prediction information;

[0034] Calculating a model loss based on the relationship information and the entity relationship prediction information, so as to adjust parameters of the initial entity relationship labeling model according to the model loss until the model loss meets a training stop condition, thereby obtaining an entity relationship labeling model;

[0035] An image to be annotated is acquired, and the image to be annotated is input into the entity relationship annotation model to obtain entity relationship information.

[0036] Compared with the prior art, the embodiments of the present application have the following advantages: obtaining a training image with entity annotation information, the entity annotation information includes text information and relationship information of each entity in the training image, the text information includes the text remark value and coordinate information of each entity, and the relationship information includes the entity relationship and association relationship between each entity; generating a text composite vector of each entity according to the text information, and generating an initial composite vector of the training image; inputting the text composite vector, the training image and the initial composite vector into the initial entity relationship annotation model, the initial entity relationship annotation model includes a convolutional network and an entity relationship annotation network; the convolutional network performs separation convolution processing on the training image to obtain a convolution as a supplementary feature Product feature vector, separating convolution can reduce the amount of calculation and reduce parameters, the inductive bias of the convolutional network can reduce the required samples and improve training efficiency; merge the initial composite vector and the convolution feature vector to obtain the image composite vector, input the text composite vector and the image composite vector into the entity relationship annotation network to obtain entity relationship prediction information; calculate the model loss according to the relationship information and the entity relationship prediction information to adjust the model parameters until the model loss meets the training stop condition, and obtain the entity relationship annotation model; obtain the image to be annotated, input the image to be annotated into the entity relationship annotation model to obtain the entity relationship information, thereby realizing automatic recognition of entities in the image and automatic annotation of entity relationships. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0039] Figure 2 is a flowchart of an embodiment of an image-based entity relationship annotation model processing method according to the present application;

[0040] Figure 3 1 is a structural diagram of an embodiment of an image-based entity relationship annotation model processing device according to the present application;

[0041] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0043] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0044] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0045] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0046] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0047] Terminal devices 101, 102, and 103 can be various electronic devices with display screens and support web browsing, including but not limited to smartphones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV), laptop computers, desktop computers, etc.

[0048] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0049] It should be noted that the image-based entity relationship annotation model processing method provided in the embodiment of the present application is generally executed by a server. Accordingly, the image-based entity relationship annotation model processing device is generally set in the server.

[0050] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0051] Continue to refer Figure 2 , shows a flow chart of an embodiment of an image-based entity relationship annotation model processing method according to the present application. The image-based entity relationship annotation model processing method includes the following steps:

[0052] Step S201, obtain a training image with entity annotation information, the entity annotation information includes text information and relationship information of each entity in the training image, the text information includes the text comment value and coordinate information of each entity, and the relationship information includes the entity relationship and association relationship between each entity.

[0053] In this embodiment, the image-based entity relationship annotation model processing method is run on the electronic device (eg Figure 1The server shown in the figure) can communicate with the terminal through a wired connection or a wireless connection. It should be noted that the above-mentioned wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0054] Specifically, we first obtain training images with entity annotation information. Entity annotation information is equivalent to the label of the training image. This application uses an entity relationship annotation model to identify entities in the image and annotate the relationships between entities. Therefore, the entity annotation information contains the text information and relationship information of each entity in the training image.

[0055] The text information includes the text annotation value and coordinate information of each entity. The text annotation value is text, which is the character representation of the entity. The entity is located in a certain area of ​​the training image, and this area has coordinate information. For example, in an image of a medical bill, there are two entities, "Personal Payment Amount" and "4180". You need to add two sets of text annotation values, "Personal Payment Amount" and "4180", to the image, select the image area where these two entities are located, and then obtain the coordinate information of "Personal Payment Amount" and "4180" based on the selected image area.

[0056] An image can contain multiple entities, each with its own associated relationship. Two entities with an associated relationship are related, and related entities have a mutual entity relationship. These associations and entity relationships constitute the relationship information between entities. For example, in an image of a medical bill, there are two entities, "Nine Thousand Six Hundred and Eighty-Five Yuan" and "9685." These entities have an associated relationship. In this relationship, "Nine Thousand Six Hundred and Eighty-Five Yuan" is capitalized, while "9685" is lowercase. Therefore, you need to add an associated relationship to these two entities and note that the entity relationship for "Nine Thousand Six Hundred and Eighty-Five Yuan" is capitalized, while the entity relationship for "9685" is lowercase.

[0057] It should be emphasized that in order to further ensure the privacy and security of the above training images, the above training images can also be stored in a blockchain node.

[0058] The blockchain referred to in this application is a new application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0059] Step S202 : generating a text composite vector for each entity based on the text information, and generating an initial composite vector for the training image.

[0060] Specifically, the entity's relevant information and training images need to be converted into vectors respectively. Since the entity's text information includes the entity's text note value and coordinate information, both the text note value and the coordinate information must be converted into vectors. After the entity's relevant information is converted into a vector, it is compounded with the text note value and coordinate information, thus obtaining a text composite vector.

[0061] When converting a training image into a vector, the training image needs to be cut into multiple image blocks first. The image blocks have position information compared to the original training image. Therefore, the vector obtained by converting the training image includes not only the image feature information of the training image but also the position information, thus obtaining an initial composite vector.

[0062] In step S203 , the text composite vector, the training image, and the initial composite vector are input into the initial entity relationship labeling model, so as to perform separation convolution processing on the training image through the convolutional network in the initial entity relationship labeling model to obtain a convolution feature vector.

[0063] Specifically, the present application implements entity relationship annotation through an entity relationship annotation model. The initial entity relationship annotation model can be an entity relationship annotation model that has not yet completed training. The initial entity relationship annotation model / entity relationship annotation model can include two parts: a convolutional network and an entity relationship annotation network.

[0064] After the text composite vector, training image, and initial composite vector are input into the initial entity relationship annotation model, the training image is first input into a convolutional neural network (CNN). The CNN performs separable convolution on the training image to obtain a convolutional feature vector. Compared to ordinary convolution, separable convolution can reduce the amount of computation and parameters, which helps maintain the lightweight nature of the model. Furthermore, the CNN has the characteristic of inductive bias, which is prior knowledge and pre-made assumptions. For example, adjacent regions in an image have similar features and are translationally invariant. With this prior information, a relatively good model can be learned based on relatively few samples. Some relationships between entities in the image are translationally invariant, while their positions and shapes may change to some extent. The convolutional network can compensate for this shortcoming. Furthermore, the convolutional feature vector output by the CNN can be used as a supplementary feature of the training image, which is equivalent to extracting more information from the training image. In step S204, the initial composite vector and the convolutional feature vector are combined to obtain the image composite vector.

[0065] Specifically, the convolution feature vector can be used as a supplementary feature of the training image, and the convolution feature vector is merged (concat) with the initial composite vector to obtain the image composite vector of the training image.

[0066] Step S205 : inputting the text composite vector and the image composite vector into the entity relationship annotation network in the initial entity relationship annotation model to obtain entity relationship prediction information.

[0067] Specifically, the text composite vector and the image composite vector are input into the entity relationship annotation network in the initial entity relationship annotation model. The entity relationship annotation network processes the text composite vector and the image composite vector and outputs entity relationship prediction information, namely, the entity relationships and association relationships between entities contained in the training image predicted by the entity relationship annotation network.

[0068] In one embodiment, the entity relationship prediction information may further include text annotation values ​​and coordinate information of each entity in the training image predicted by the entity relationship annotation network.

[0069] Step S206 , calculating the model loss based on the relationship information and the entity relationship prediction information, and adjusting the parameters of the initial entity relationship labeling model according to the model loss until the model loss meets the training stop condition, thereby obtaining the entity relationship labeling model.

[0070] Specifically, the model loss is calculated based on the relationship information in the entity annotation information and the entity relationship prediction information; when the entity relationship prediction information contains text information, that is, the text annotation value and coordinate information of each entity, the model loss needs to be calculated based on the text information in the entity annotation information and the text information in the entity relationship prediction information, the relationship information in the entity annotation information and the relationship information in the entity relationship prediction information.

[0071] After obtaining the model loss, the model parameters of the initial entity relationship labeling model are adjusted with the goal of minimizing the model loss, and the initial entity relationship labeling model is iteratively trained after the parameter adjustment until the obtained model loss meets the training stop condition (for example, the model loss converges, or the model loss is less than the preset loss threshold), then the training is stopped to obtain the entity relationship labeling model.

[0072] Step S207 , obtaining an image to be annotated, and inputting the image to be annotated into an entity relationship annotation model to obtain entity relationship information.

[0073] Specifically, during application, an image to be annotated is obtained and fed into a trained entity relationship annotation model to obtain entity relationship information output by the entity relationship annotation model. This entity relationship information can include relationship information between entities in the image to be annotated (i.e., entity relationships and association relationships between entities), as well as textual information about each entity in the image to be annotated (e.g., textual annotation values ​​and coordinate information of each entity), thereby enabling automatic recognition of entities in the image and automatic annotation of entity relationships.

[0074] In this embodiment, a training image with entity annotation information is obtained, and the entity annotation information includes text information and relationship information of each entity in the training image, the text information includes the text remark value and coordinate information of each entity, and the relationship information includes the entity relationship and association relationship between each entity; a text composite vector of each entity is generated according to the text information, and an initial composite vector of the training image is generated; the text composite vector, the training image and the initial composite vector are input into an initial entity relationship annotation model, and the initial entity relationship annotation model includes a convolutional network and an entity relationship annotation network; the convolutional network performs separation convolution processing on the training image to obtain a convolution feature vector as a supplementary feature, and the separation convolution feature vector is used as the convolution feature vector. The product can reduce the amount of calculation and reduce parameters. The inductive bias of the convolutional network can reduce the required samples and improve training efficiency. The initial composite vector and the convolution feature vector are merged to obtain the image composite vector, and the text composite vector and the image composite vector are input into the entity relationship annotation network to obtain entity relationship prediction information. The model loss is calculated according to the relationship information and the entity relationship prediction information to adjust the model parameters until the model loss meets the training stop condition to obtain the entity relationship annotation model. The image to be annotated is obtained, and the image to be annotated is input into the entity relationship annotation model to obtain the entity relationship information, thereby realizing the automatic recognition of entities in the image and the automatic annotation of entity relationships.

[0075] Furthermore, before the above step S201, it may also include: obtaining an initial training image with entity annotation information; performing image enhancement processing on the initial training image to obtain an enhanced image; and obtaining entity annotation information of the enhanced image based on the image enhancement processing to obtain a training image.

[0076] Specifically, an initial training image with entity annotation information is obtained. The entity annotation information of the initial training image is the same as the entity annotation information mentioned above. In order to reduce the number of samples required for training and the labor cost consumed by annotation, the initial training image is subjected to image enhancement processing according to a preset image enhancement method, including rotation processing, blurring processing, brightness adjustment, etc., to obtain multiple enhanced images of the initial training image. It is understandable that the initial training image can be enhanced according to only one image enhancement method, or multiple image enhancement processes can be superimposed. The initial training image itself can also be used as an enhanced image.

[0077] After performing image enhancement processing on the initial training image, the entity annotation information of the enhanced image may be different from the entity annotation information of the initial training image. For example, after rotation processing, the position information of the entity may change. It is necessary to adjust the entity annotation information of the initial training image according to the image enhancement processing performed to obtain the entity annotation information of each enhanced image, thereby obtaining an accurate training image.

[0078] In this embodiment, image enhancement processing is performed on the initial training images to expand the samples, and the entity annotation information of the initial training images is adjusted according to the image enhancement processing to obtain the entity annotation information of the enhanced images, thereby obtaining a large number of training images, increasing the multi-sample of the samples, and improving the adaptability and robustness of the model.

[0079] Furthermore, the above-mentioned step of generating a text composite vector for each entity based on the text information may include: converting the text remark value of each entity in the text information into a word vector respectively; generating a one-dimensional position vector for each entity according to the text remark value of each entity, and generating a two-dimensional position vector for each entity according to the coordinate information of each entity; generating a text composite vector for each entity according to the word vector, one-dimensional position vector and two-dimensional position vector corresponding to each entity.

[0080] Specifically, the text annotation value of each entity is obtained from the text information and converted into a word vector. The text annotation value of each entity forms the text corresponding to the training image, and based on this text, a one-dimensional position vector of each entity is obtained. The entity's coordinate information reflects the entity's position in the training image and can reflect the layout of the training image. Based on the entity's coordinate information, a two-dimensional position vector of the entity is generated. The word vector, one-dimensional position vector, and two-dimensional position vector of each entity are added together to obtain a composite text vector for each entity.

[0081] In one embodiment, word vectors can be generated by a pre-trained Roberta model, and one-dimensional position vectors and two-dimensional position vectors can also be generated by a pre-trained Roberta model. The Roberta model that generates word vectors, one-dimensional position vectors, and two-dimensional position vectors can be the same model or different models.

[0082] In this embodiment, the text composite vector of an entity is obtained by adding the entity's word vector, one-dimensional position vector, and two-dimensional position vector. It takes into account the text semantics of the entity itself, the position of the entity in the text, and the position of the entity in the training image, so that the text composite vector can accurately and comprehensively represent the entity.

[0083] Furthermore, the above-mentioned step of generating the initial composite vector of the training image may include: adjusting the training image to a preset size, and cutting the resized training image according to a preset cutting method to obtain a number of image blocks; generating a one-dimensional position vector of the training image, and generating image features of each image block respectively; generating the initial composite vector of the training image based on the one-dimensional position vector and the image features of each image block.

[0084] Specifically, the size of the training image is adjusted and scaled to a preset size, and then the resized training image is cut according to a preset cutting method to obtain several image blocks. The size of each image block can be the same. For example, the training image is cut into 3*3 16*16 image blocks.

[0085] Based on a preset order, such as their position in the training image, the image blocks are arranged in a row to generate a learnable one-dimensional position vector for the training image. A linear mapping is then performed on each image block to obtain image features. The image features are added to the one-dimensional position vector to obtain the initial composite vector for the training image.

[0086] In this embodiment, the one-dimensional position vector of the training image is obtained by arranging the image blocks, and the image features of each image block are added to obtain the initial composite vector. This takes into account both the position information and the image features of the training image, ensuring the accuracy of the initial composite vector.

[0087] Furthermore, before the above-mentioned step of inputting the text composite vector, the training image and the initial composite vector into the initial entity relationship annotation model, it may also include: obtaining an initial entity relationship annotation network, wherein the initial entity relationship annotation network is constructed based on the layoutLMv3 network; pre-training the initial entity relationship annotation network according to a preset pre-training task to obtain an entity relationship annotation network, wherein the preset pre-training task includes a masked language modeling task, a masked image modeling task and a word block alignment task.

[0088] Specifically, the initial entity relationship annotation model includes an entity relationship annotation network, which requires pre-training. This network can be built on the layoutLMv3 network, a multimodal transformer architecture that combines text and images in a unified manner. It segments images into patches, represents them as linear projections, and aligns the linear projections with the textual labels, reducing the required parameters and overall computational effort.

[0089] The layoutLMv3 network has been redesigned for both training and image processing. It no longer uses a visual model, but instead adopts a visual information processing (VIT) to reduce model parameters. LayoutLMv3 directly utilizes image patches, significantly saving parameters and avoiding complex text preprocessing (such as manual object bounding box annotation and text object detection). Its simple, unified architecture and training objectives make LayoutLMv3 a versatile pre-trained model suitable for both text-centric and image-centric document tasks.

[0090] The initial entity relationship annotation network is pre-trained according to the preset pre-training tasks, and the multimodal feature representation is learned in a self-supervised manner to obtain the entity relationship annotation network. The preset pre-training tasks include masked language modeling task, masked image modeling task and word block alignment task.

[0091] Masked Language Modeling (MLM) randomly masks a certain percentage of text word vectors to help the network learn the correspondence between layout information and text and images, while retaining the corresponding two-dimensional position (layout) information. Similar to BERT and LayoutLM, the model's goal is to restore the masked words in the text based on the unmasked image and layout information.

[0092] Masked Image Modeling (MIM): To encourage the model to infer image information from the context of text and images, this task randomly masks a certain percentage of image patches. Similar to BEiT, the model's goal is to recover the discretized IDs of the masked image patches based on the information of the unmasked text and image.

[0093] Word-Patch Alignment (WPA). For text, each word corresponds to an image patch. Because the previous two tasks randomly masked some words and image patches, the model cannot explicitly learn the fine-grained alignment between words and image patches. This objective learns the fine-grained alignment between the linguistic and visual modalities by explicitly predicting whether the corresponding image patch of a word is masked.

[0094] In this application, the convolutional network CNN has the characteristics of inductive bias, and the Transformer in the layoutLMv3 network has strong global inductive modeling capabilities; after adding the convolutional network CNN, the limitation of the transformer's lack of inductive bias is broken through, and better migration effects can be achieved in downstream tasks, greatly increasing the model's ability in small sample learning tasks. In the small sample scenario, the Transformer is freed from its dependence on large sample big data, reducing the sample size required for training and improving training efficiency.

[0095] In this embodiment, the initial entity relationship annotation network is pre-trained according to the masked language modeling task, the masked image modeling task, and the word chunk alignment task, ensuring that the obtained entity relationship annotation network can perform entity relationship annotation.

[0096] Furthermore, after the above step S206 , the following may be further included: performing image detection on the image to be annotated according to the entity relationship information to obtain an image detection result; and performing business processing on the image to be annotated according to the image detection result.

[0097] Specifically, after obtaining the entity relationship information, image detection can be performed on the image to be annotated based on the entity relationship information. The image detection here can be to perform business review of the image to be annotated based on the image type of the image to be annotated, such as detecting whether a specific type of entity is missing in the image, and whether there is an error in the entity. In the image of the medical bill, if there is no text note value of the personal payment fee, that is, the specific personal payment fee value is not recorded, then the medical bill may have an error; if two entities representing the amount are associated, one of them is uppercase and the other is lowercase, but the amount values ​​corresponding to their text note values ​​are not equal, then the medical bill may have an error.

[0098] After image detection, the system obtains the image detection result, which indicates whether the image to be annotated has passed the test. If it has not passed, it records the entities that have errors. Business processing can be performed on the image to be annotated based on the image detection result. For example, if the image detection result indicates that the image to be annotated has passed the test, business processing will proceed to the next process node. For example, if the image of a medical bill passes the test, the process will enter the reimbursement process. If the image to be annotated has not passed the test, an error message will be generated and the image to be annotated will be returned to the previous process node.

[0099] In this embodiment, image detection is performed on the annotated image based on the entity relationship information to obtain the image detection result, and business processing is performed on the annotated image based on the image detection result, entering the next process node or returning to the previous process node, thereby realizing automatic business processing and improving business processing efficiency.

[0100] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0101] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0102] This application can be applied in the field of smart medical care, thereby promoting the construction of smart cities.

[0103] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware via computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0104] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0105] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of an image-based entity relationship annotation model processing device, which is similar to the embodiment of the present invention. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0106] like Figure 3 As shown, the image-based entity relationship annotation model processing device 300 described in this embodiment includes: an image acquisition module 301, a vector generation module 302, a convolution processing module 303, a vector merging module 304, a vector input module 305, a model adjustment module 306 and an image annotation module 307, wherein:

[0107] The image acquisition module 301 is used to acquire training images with entity annotation information. The entity annotation information includes the text information and relationship information of each entity in the training image. The text information includes the text annotation value and coordinate information of each entity. The relationship information includes the entity relationship and association relationship between each entity.

[0108] The vector generation module 302 is used to generate a text composite vector of each entity according to the text information, and to generate an initial composite vector of the training image.

[0109] The convolution processing module 303 is used to input the text composite vector, the training image and the initial composite vector into the initial entity relationship labeling model, so as to perform separate convolution processing on the training image through the convolution network in the initial entity relationship labeling model to obtain a convolution feature vector.

[0110] The vector merging module 304 is configured to merge the initial composite vector and the convolution feature vector to obtain an image composite vector.

[0111] The vector input module 305 is used to input the text composite vector and the image composite vector into the entity relationship annotation network in the initial entity relationship annotation model to obtain entity relationship prediction information.

[0112] The model adjustment module 306 is used to calculate the model loss based on the relationship information and the entity relationship prediction information, so as to adjust the parameters of the initial entity relationship labeling model according to the model loss until the model loss meets the training stop condition, thereby obtaining the entity relationship labeling model.

[0113] The image annotation module 307 is used to obtain the image to be annotated and input the image to be annotated into the entity relationship annotation model to obtain entity relationship information.

[0114] In this embodiment, a training image with entity annotation information is obtained, and the entity annotation information includes text information and relationship information of each entity in the training image, the text information includes the text remark value and coordinate information of each entity, and the relationship information includes the entity relationship and association relationship between each entity; a text composite vector of each entity is generated according to the text information, and an initial composite vector of the training image is generated; the text composite vector, the training image and the initial composite vector are input into an initial entity relationship annotation model, and the initial entity relationship annotation model includes a convolutional network and an entity relationship annotation network; the convolutional network performs separation convolution processing on the training image to obtain a convolution feature vector as a supplementary feature, and the separation convolution feature vector is used as the convolution feature vector. The product can reduce the amount of calculation and reduce parameters. The inductive bias of the convolutional network can reduce the required samples and improve training efficiency. The initial composite vector and the convolution feature vector are merged to obtain the image composite vector, and the text composite vector and the image composite vector are input into the entity relationship annotation network to obtain entity relationship prediction information. The model loss is calculated according to the relationship information and the entity relationship prediction information to adjust the model parameters until the model loss meets the training stop condition to obtain the entity relationship annotation model. The image to be annotated is obtained, and the image to be annotated is input into the entity relationship annotation model to obtain the entity relationship information, thereby realizing the automatic recognition of entities in the image and the automatic annotation of entity relationships.

[0115] In some optional implementations of this embodiment, the image-based entity relationship annotation model processing device 300 may further include: an initial acquisition module, an image enhancement module, and an image generation module, wherein:

[0116] The initial acquisition module is used to obtain initial training images with entity annotation information.

[0117] The image enhancement module is used to perform image enhancement processing on the initial training image to obtain an enhanced image.

[0118] The image generation module is used to obtain entity annotation information of the enhanced image according to the image enhancement processing to obtain a training image.

[0119] In this embodiment, image enhancement processing is performed on the initial training images to expand the samples, and the entity annotation information of the initial training images is adjusted according to the image enhancement processing to obtain the entity annotation information of the enhanced images, thereby obtaining a large number of training images, increasing the multi-sample of the samples, and improving the adaptability and robustness of the model.

[0120] In some optional implementations of this embodiment, the vector generation module 302 may include: a text conversion submodule, a vector generation submodule, and a composite generation submodule, wherein:

[0121] The text conversion submodule is used to convert the text annotation value of each entity in the text information into a word vector.

[0122] The vector generation submodule is used to generate a one-dimensional position vector of each entity according to the text remark value of each entity, and to generate a two-dimensional position vector of each entity according to the coordinate information of each entity.

[0123] The composite generation submodule is used to generate a text composite vector for each entity based on the word vector, one-dimensional position vector and two-dimensional position vector corresponding to each entity.

[0124] In this embodiment, the text composite vector of an entity is obtained by adding the entity's word vector, one-dimensional position vector, and two-dimensional position vector. It takes into account the text semantics of the entity itself, the position of the entity in the text, and the position of the entity in the training image, so that the text composite vector can accurately and comprehensively represent the entity.

[0125] In some optional implementations of this embodiment, the vector generation module 302 may further include: an image adjustment submodule, a generation submodule, and an initial generation submodule, wherein:

[0126] The image adjustment submodule is used to adjust the training image to a preset size and cut the resized training image according to a preset cutting method to obtain a plurality of image blocks.

[0127] The generation submodule is used to generate a one-dimensional position vector of the training image and generate image features for each image block respectively.

[0128] The initial generation submodule is used to generate an initial composite vector of the training image according to the one-dimensional position vector and the image features of each image block.

[0129] In this embodiment, the one-dimensional position vector of the training image is obtained by arranging the image blocks, and the image features of each image block are added to obtain the initial composite vector. This takes into account both the position information and the image features of the training image, ensuring the accuracy of the initial composite vector.

[0130] In some optional implementations of this embodiment, the image-based entity relationship annotation model processing apparatus 300 may further include: a network acquisition module and a pre-training module, wherein:

[0131] The network acquisition module is used to obtain an initial entity relationship annotation network, wherein the initial entity relationship annotation network is constructed based on the layoutLMv3 network.

[0132] The pre-training module is used to pre-train the initial entity relationship annotation network according to the preset pre-training tasks to obtain the entity relationship annotation network, wherein the preset pre-training tasks include the masked language modeling task, the masked image modeling task and the word block alignment task.

[0133] In this embodiment, the initial entity relationship annotation network is pre-trained according to the masked language modeling task, the masked image modeling task, and the word chunk alignment task, ensuring that the obtained entity relationship annotation network can perform entity relationship annotation.

[0134] In some optional implementations of this embodiment, the image-based entity relationship annotation model processing device 300 may further include: an image detection module and a business processing module, wherein:

[0135] The image detection module is used to perform image detection on the annotated image according to the entity relationship information to obtain the image detection result.

[0136] The business processing module is used to perform business processing on the image to be annotated according to the image detection results.

[0137] In this embodiment, image detection is performed on the annotated image based on the entity relationship information to obtain the image detection result, and business processing is performed on the annotated image based on the image detection result, entering the next process node or returning to the previous process node, thereby realizing automatic business processing and improving business processing efficiency.

[0138] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0139] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0140] The computer device may be a desktop computer, notebook computer, PDA, cloud server, etc. The computer device may interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.

[0141] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for the image-based entity relationship labeling model processing method. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0142] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or process data, such as executing computer-readable instructions for the image-based entity relationship labeling model processing method.

[0143] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0144] The computer device provided in this embodiment can execute the above-mentioned image-based entity relationship annotation model processing method. Here, the image-based entity relationship annotation model processing method can be the image-based entity relationship annotation model processing method of each of the above-mentioned embodiments.

[0145] In this embodiment, a training image with entity annotation information is obtained, and the entity annotation information includes text information and relationship information of each entity in the training image, the text information includes the text remark value and coordinate information of each entity, and the relationship information includes the entity relationship and association relationship between each entity; a text composite vector of each entity is generated according to the text information, and an initial composite vector of the training image is generated; the text composite vector, the training image and the initial composite vector are input into an initial entity relationship annotation model, and the initial entity relationship annotation model includes a convolutional network and an entity relationship annotation network; the convolutional network performs separation convolution processing on the training image to obtain a convolution feature vector as a supplementary feature, and the separation convolution feature vector is used as the convolution feature vector. The product can reduce the amount of calculation and reduce parameters. The inductive bias of the convolutional network can reduce the required samples and improve training efficiency. The initial composite vector and the convolution feature vector are merged to obtain the image composite vector, and the text composite vector and the image composite vector are input into the entity relationship annotation network to obtain entity relationship prediction information. The model loss is calculated according to the relationship information and the entity relationship prediction information to adjust the model parameters until the model loss meets the training stop condition to obtain the entity relationship annotation model. The image to be annotated is obtained, and the image to be annotated is input into the entity relationship annotation model to obtain the entity relationship information, thereby realizing the automatic recognition of entities in the image and the automatic annotation of entity relationships.

[0146] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned image-based entity relationship labeling model processing method.

[0147] In this embodiment, a training image with entity annotation information is obtained, and the entity annotation information includes text information and relationship information of each entity in the training image, the text information includes the text remark value and coordinate information of each entity, and the relationship information includes the entity relationship and association relationship between each entity; a text composite vector of each entity is generated according to the text information, and an initial composite vector of the training image is generated; the text composite vector, the training image and the initial composite vector are input into an initial entity relationship annotation model, and the initial entity relationship annotation model includes a convolutional network and an entity relationship annotation network; the convolutional network performs separation convolution processing on the training image to obtain a convolution feature vector as a supplementary feature, and the separation convolution feature vector is used as the convolution feature vector. The product can reduce the amount of calculation and reduce parameters. The inductive bias of the convolutional network can reduce the required samples and improve training efficiency. The initial composite vector and the convolution feature vector are merged to obtain the image composite vector, and the text composite vector and the image composite vector are input into the entity relationship annotation network to obtain entity relationship prediction information. The model loss is calculated according to the relationship information and the entity relationship prediction information to adjust the model parameters until the model loss meets the training stop condition to obtain the entity relationship annotation model. The image to be annotated is obtained, and the image to be annotated is input into the entity relationship annotation model to obtain the entity relationship information, thereby realizing the automatic recognition of entities in the image and the automatic annotation of entity relationships.

[0148] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0149] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A method for processing an entity relationship annotation model based on an image, characterized in that: The steps include: Acquire a training image with entity annotation information, wherein the entity annotation information includes text information and relationship information of each entity in the training image, the text information includes a text note value and coordinate information of each entity, and the relationship information includes entity relationships and association relationships between the entities; Generating a text composite vector of each entity according to the text information, and generating an initial composite vector of the training image; Inputting the text composite vector, the training image, and the initial composite vector into an initial entity relationship annotation model, so as to perform separation convolution processing on the training image through a convolutional network in the initial entity relationship annotation model to obtain a convolution feature vector; Combining the initial composite vector and the convolution feature vector to obtain an image composite vector; Inputting the text composite vector and the image composite vector into the entity relationship annotation network in the initial entity relationship annotation model to obtain entity relationship prediction information; Calculating a model loss based on the relationship information and the entity relationship prediction information, so as to adjust parameters of the initial entity relationship labeling model according to the model loss until the model loss meets a training stop condition, thereby obtaining an entity relationship labeling model; An image to be annotated is acquired, and the image to be annotated is input into the entity relationship annotation model to obtain entity relationship information.

2. The image-based entity relationship annotation model processing method according to claim 1, characterized in that: Before the step of obtaining the training image with entity annotation information, the method further includes: Obtain initial training images with entity annotation information; Performing image enhancement processing on the initial training image to obtain an enhanced image; According to the image enhancement processing, entity annotation information of the enhanced image is obtained to obtain a training image.

3. The image-based entity relationship annotation model processing method according to claim 1, characterized in that: The step of generating a text composite vector of each entity according to the text information includes: Convert the text remark value of each entity in the text information into a word vector respectively; Generating a one-dimensional position vector of each entity according to the text remark value of each entity, and generating a two-dimensional position vector of each entity according to the coordinate information of each entity; A text composite vector of each entity is generated according to the word vector, one-dimensional position vector and two-dimensional position vector corresponding to each entity.

4. The image-based entity relationship annotation model processing method according to claim 1, characterized in that: The step of generating the initial composite vector of the training image comprises: Adjusting the training image to a preset size, and cutting the resized training image according to a preset cutting method to obtain a plurality of image blocks; generating a one-dimensional position vector of the training image and respectively generating image features of each image block; An initial composite vector of the training image is generated according to the one-dimensional position vector and the image features of each image block.

5. The image-based entity relationship annotation model processing method according to claim 1, characterized in that: Before the step of inputting the text composite vector, the training image and the initial composite vector into the initial entity relationship annotation model, the method further includes: Obtaining an initial entity relationship annotation network, wherein the initial entity relationship annotation network is constructed based on the layoutLMv3 network; The initial entity relationship annotation network is pre-trained according to a preset pre-training task to obtain an entity relationship annotation network, wherein the preset pre-training task includes a masked language modeling task, a masked image modeling task and a word chunk alignment task.

6. The image-based entity relationship annotation model processing method according to claim 1, characterized in that: After the step of inputting the image to be annotated into the entity relationship annotation model to obtain entity relationship information, the method further includes: Performing image detection on the image to be annotated according to the entity relationship information to obtain an image detection result; Perform business processing on the image to be labeled according to the image detection result.

7. An image-based entity relationship annotation model processing device, characterized in that: include: An image acquisition module is configured to acquire training images with entity annotation information, wherein the entity annotation information includes text information and relationship information of each entity in the training image, the text information includes a text note value and coordinate information of each entity, and the relationship information includes entity relationships and association relationships between the entities; a vector generation module, configured to generate a text composite vector of each entity according to the text information, and to generate an initial composite vector of the training image; a convolution processing module, configured to input the text composite vector, the training image, and the initial composite vector into an initial entity relationship annotation model, so as to perform separation convolution processing on the training image through a convolutional network in the initial entity relationship annotation model to obtain a convolution feature vector; A vector merging module, configured to merge the initial composite vector and the convolution feature vector to obtain an image composite vector; A vector input module, configured to input the text composite vector and the image composite vector into an entity relationship annotation network in the initial entity relationship annotation model to obtain entity relationship prediction information; A model adjustment module, configured to calculate a model loss based on the relationship information and the entity relationship prediction information, so as to adjust parameters of the initial entity relationship labeling model according to the model loss until the model loss satisfies a training stop condition, thereby obtaining an entity relationship labeling model; The image annotation module is used to obtain the image to be annotated and input the image to be annotated into the entity relationship annotation model to obtain entity relationship information.

8. The image-based entity relationship annotation model processing device according to claim 7, characterized in that: The device further comprises: The initial acquisition module is used to obtain the initial training images with entity annotation information; An image enhancement module, configured to perform image enhancement processing on the initial training image to obtain an enhanced image; An image generation module is used to obtain entity annotation information of the enhanced image according to the image enhancement processing to obtain a training image.

9. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of the image-based entity relationship labeling model processing method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the image-based entity relationship labeling model processing method according to any one of claims 1 to 6.

Images