Image information extraction method and device, computer equipment and storage medium
By segmenting the original image into sub-images and inputting it into the decoding model with the description text features, the problem of inaccurate image information extraction in the prior art is solved, and the accuracy and expression ability of image information extraction are improved.
Patent Information
- Application Number
- CN202411064653.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art ignores the semantic information of the image in image processing and only focuses on pixel-level features, which leads to the model being unable to understand the higher-level semantic features in the image, which is susceptible to noise and interference factors, resulting in the accuracy of image information extraction. Not high.
By segmenting the original image into multiple sub-images, the target image features and description text features of each sub-image are determined, combined with the trained decoding model, and the target sub-image related to the description text is output, thereby extracting the entity corresponding to the description text. .
It improves the accuracy of extraction of objects that need to be extracted in the image, enhances the feature dimensions and expression capabilities of image information, and can extract the entities corresponding to the description text in the original image more accurately.
Smart Images

Figure CN118918441A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image information extraction method, device, computer equipment and storage medium. Background Art
[0002] With the rise of image processing technology, image processing technology has been applied in many scenarios. For example, in the medical field, it can support disease auxiliary diagnosis, health management, remote consultation and other functions.
[0003] In the related art, the model is trained by using image information, and the trained model is used to extract images. However, this method ignores the semantic information of the image and only focuses on the features at the pixel level. This causes the model to be unable to understand the higher-level semantic features in the image and is easily affected by noise and other interference factors, resulting in low accuracy in image information extraction.
[0004] Therefore, the related art suffers from inaccurate extraction of entities in images. Summary of the invention
[0005] The main purpose of the embodiments of the present application is to provide an image information extraction method, apparatus, computer equipment and storage medium, which can improve the accuracy of extracting objects that need to be extracted from images.
[0006] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes an image information extraction method, the method comprising:
[0007] Segmenting the original image into multiple sub-images, and determining the target image features corresponding to each of the sub-images;
[0008] Determining a description text corresponding to the original image, and determining a target text feature corresponding to the description text;
[0009] The target text feature is respectively concatenated with each target image feature to obtain a target image-text feature corresponding to each sub-image;
[0010] Inputting the target graphic features and the target image features into a trained decoding model, and outputting a target sub-image related to the description text, wherein the trained decoding model is trained based on the corresponding loss between the prediction sub-sample image and the label sub-image;
[0011] An entity corresponding to the description text is extracted according to the target sub-image.
[0012] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an image information extraction device, the device comprising:
[0013] A segmentation module, used to segment the original image into multiple sub-images and determine the target image features corresponding to each of the sub-images;
[0014] A determination module, used to determine the description text corresponding to the original image, and determine the target text features corresponding to the description text;
[0015] A splicing module, used for splicing the target text feature with each target image feature to obtain the target image-text feature corresponding to each sub-image;
[0016] An input module, used for inputting the target graphic features and the target image features into a trained decoding model, and outputting a target sub-image related to the description text, wherein the trained decoding model is trained based on the corresponding loss between the prediction sub-sample image and the label sub-image;
[0017] The extraction module is used to extract the entity corresponding to the description text according to the target sub-image.
[0018] In some embodiments, the segmentation module includes a first input submodule and a second input submodule; the first input submodule is used to input each of the sub-images into a pre-trained convolutional neural network and output image features corresponding to each of the sub-images;
[0019] The second input submodule is used to input the image features corresponding to each of the sub-images into the trained first encoding model, and output the target image features corresponding to each of the sub-images.
[0020] In some embodiments, the trained first encoding model includes a plurality of sequentially connected sub-first attention models, and the second input submodule is used to input the image feature corresponding to each of the sub-images into the first sub-first attention model to obtain the sub-first attention feature output by the first sub-first attention model;
[0021] Inputting the sub-first attention feature output by the first sub-first attention model into the next sub-first attention model to obtain the sub-first attention feature output by the next sub-first attention model;
[0022] Repeatedly input the sub-first attention features output by the previous sub-first attention model into the next sub-first attention model until the last sub-first attention model outputs the target image features corresponding to each sub-image.
[0023] In some implementations, the determination module is used to divide the description text into multiple sub-texts;
[0024] Determine the text features corresponding to each of the sub-texts, input the text features into the trained second encoding model, and output the sub-target text features corresponding to each of the sub-texts;
[0025] The target text feature corresponding to the description text is determined according to the sub-target text feature corresponding to each of the sub-texts.
[0026] In some embodiments, an input module is used to input the target graphic feature and the target image feature into a trained decoding model to obtain a correlation score between the target graphic feature and the target image feature;
[0027] The sub-image corresponding to the target image feature whose correlation score is higher than a preset score value is determined, and the sub-image corresponding to the target image feature whose correlation score is higher than the preset score value is determined as a target sub-image related to the description text.
[0028] In some embodiments, an extraction module is used to determine the outline of the entity corresponding to the description text in the target sub-image;
[0029] The entities in the target sub-image are divided based on the contour to obtain the entities.
[0030] In some embodiments, the image information extraction model further includes a training module, which is used to divide the sample image into a plurality of sub-sample images, input each of the sub-sample images into a pre-trained convolutional neural network, and output sample image features corresponding to each of the sub-sample images;
[0031] Inputting sample image features corresponding to each of the sub-sample images into a first coding model, and outputting target sample image features corresponding to each of the sub-sample images;
[0032] Determining a sample description text corresponding to the sample image and a sample text feature of the sample description text;
[0033] Inputting the sample text features into a second encoding model, and outputting target sample text features of the sample description text;
[0034] The target sample image feature and the target sample text feature are concatenated to obtain the target sample image and text feature;
[0035] The sample image and text features and the target sample image features are input into the decoding model, the predicted sub-sample image related to the sample description text is output, and according to the loss between the predicted sub-sample image and the label sub-image, the model parameters of the first encoding model, the second encoding model and the decoding model are adjusted according to the loss, and the sample image features corresponding to each of the sub-sample images are input into the first encoding model, and the target sample image features corresponding to each of the sub-sample images are output until the loss meets the preset loss adjustment, thereby obtaining the trained first encoding model, the trained second encoding model and the trained decoding model.
[0036] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores multiple instructions, and the instructions are suitable for a processor to load to execute the image information extraction method provided in the embodiment of the present application.
[0037] To achieve the above-mentioned objectives, the fourth aspect of an embodiment of the present application proposes a computer device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the image information extraction method provided in the embodiment of the present application when executing the computer program.
[0038] In an embodiment of the present application, an original image is divided into multiple sub-images, and the target image features corresponding to each sub-image are determined; the description text corresponding to the original image is determined, and the target text features corresponding to the description text are determined; the target text features are respectively spliced with each target image feature to obtain the target graphic features corresponding to each sub-image; the target graphic features and the target image features are input into the trained decoding model, and the target sub-image related to the description text is output; the entity corresponding to the description text is extracted according to the target sub-image, wherein the trained decoding model is trained based on the corresponding loss between the predicted sub-sample image and the label sub-image. In this way, the target graphic features are obtained by combining the target image features of the original image with the target text features of the description text of the original image, thereby enhancing the feature dimension of the image information of the original image and enhancing the expression of the image information of the original image, so that the trained decoding model can accurately determine the target sub-image in multiple sub-images according to the target graphic features, and finally accurately extract the entity corresponding to the description text in the target sub-image. Compared with the related art that only extracts the entity to be extracted by the pixel information of the image, the entity corresponding to the description text can be more accurately extracted in the original image in the present application.
[0039] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0041] Figure 1 It is a schematic diagram of an application scenario of the image information extraction method provided in an embodiment of the present application;
[0042] Figure 2 It is a flowchart of the image information extraction method provided in the embodiment of the present application;
[0043] Figure 3 It is a structural schematic diagram of the image information extraction model provided in the embodiment of the present application;
[0044] Figure 4 is a schematic diagram of the structure of the first coding model after training provided in an embodiment of the present application;
[0045] Figure 5 is a schematic diagram of the structure of the trained second encoding model provided in an embodiment of the present application;
[0046] Figure 6 It is a schematic diagram of the training process of the trained image information extraction model provided in an embodiment of the present application;
[0047] Figure 7 is another flow chart of the image information extraction method provided in an embodiment of the present application;
[0048] Figure 8 is a schematic diagram of the structure of an image information extraction device provided in an embodiment of the present application;
[0049] Fig. 9 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0050] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.
[0051] It should be noted that in each specific implementation of the present application, when it comes to image data and other related data, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or jump to a confirmation page. After the user's separate permission or consent is clearly obtained, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.
[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0054] First, some terms used in this application are explained:
[0055] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0056] Attention Mechanism: It is a commonly used technique in computer science and machine learning that can make the model more accurate and effective when processing sequence data. In traditional neural networks, the output of each neuron depends only on the output of all neurons in the previous layer, while in the attention mechanism, the output of each neuron not only depends on the output of all neurons in the previous layer, but can also be weighted according to different parts of the input data, that is, different weights are given to different parts. This allows the model to pay more attention to the key information in the input sequence, thereby improving the accuracy and efficiency of the model. In natural language processing, the attention mechanism is often used in tasks such as machine translation, speech recognition, and text summarization. Taking machine translation as an example, the attention mechanism can help the model better focus on the important parts of the source language and the target language during the translation process.
[0057] Image description generates a natural language description for an image and uses the generated description to help applications understand the semantics expressed in the image visual scene. For example, image description can convert image retrieval into text retrieval, which is used to classify images and improve image retrieval results. People usually only need to take a quick glance to describe the details of the image visual scene, while automatically adding descriptions to images is a comprehensive and arduous computer vision task that requires converting the complex information contained in the image into a natural language description. Compared with ordinary computer vision tasks, image captioning requires not only identifying objects from images, but also associating the identified objects with natural semantics and describing them in natural language. Therefore, image description requires people to extract deep features of images, associate them with semantic features, and convert them for generating descriptions.
[0058] Secondly, let's explain the technical problems existing in the relevant technologies:
[0059] With the rise of image processing technology, image processing technology has been applied in many scenarios. For example, in the medical field, it can support disease auxiliary diagnosis, health management, remote consultation and other functions.
[0060] In the related art, the model is trained by using image information, and the trained model is used to extract images. However, this method ignores the semantic information of the image and only focuses on the features at the pixel level. This causes the model to be unable to understand the higher-level semantic features in the image and is easily affected by noise and other interference factors, resulting in low accuracy in image information extraction.
[0061] Therefore, the related art has the problem of inaccurate extraction of entities in images. For example, in medical images, it is impossible to accurately extract the image of the part that needs to be viewed.
[0062] In order to solve the above technical problems, the embodiments of the present application provide an image information extraction method, apparatus, computer equipment and storage medium, which can improve the accuracy of extracting objects to be extracted from images.
[0063] The image information extraction method, apparatus, computer equipment and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the recommended method in the embodiments of the present application is described.
[0064] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0065] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0066] The image information extraction method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The image information extraction method, device, computer equipment and storage medium provided in the embodiment of the present application. The method can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the image information extraction method, etc., but is not limited to the above forms.
[0067] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0068] See also Figure 1 , Figure 1 It is a schematic diagram of an application scenario of the image information extraction method provided in an embodiment of the present application.
[0069] Among them, the embodiment of the present application provides a trained image information extraction model, which has the ability to extract entities in images, for example, the image information extraction model can extract entities in medical images. The image information extraction model includes a trained first encoding model, a trained second encoding model and a trained decoding model. The input of the trained first encoding model is the image data corresponding to the image, and the input of the trained second encoding model is the text data of the descriptive text corresponding to the image. The outputs of the trained first encoding model and the trained second encoding model are spliced and used as input and input into the trained decoding model. The trained decoding model can output the target sub-image in the selected image and extract the entity to be extracted based on the target sub-image.
[0070] for example, Figure 1 The original image shown is a medical image in a medical application scenario, and the type of object contained in the original image is a lesion, that is, a part of the body where a disease occurs. Medical images refer to images of internal tissues, such as the stomach, abdomen, heart, knees, and brain, obtained in a non-invasive manner for medical treatment or medical research, such as CT (Computed Tomography), MRI (Magnetic Resonance Imaging), US (ultrasound), X-ray images, electroencephalograms, and optical photographic lights generated by medical instruments.
[0071] The medical image is specifically a vascular image. The medical image corresponds to a description text, which is "extract the aortic image". The medical image can be divided into multiple sub-images, and the target image features corresponding to each sub-image can be determined. The target image features can be the output of the trained first encoding model. The target text features corresponding to the description text are determined, and the target text features can be the output of the trained second encoding model. Finally, the target text features are spliced with each target image feature to obtain the target graphic features corresponding to each sub-image; the target graphic features and the target image features are input into the trained decoding model, and the target sub-image related to the description text is output. The entity corresponding to the description text is extracted according to the target sub-image. The entity is the aortic image described in the description text, thereby accurately extracting the aortic image from the medical image.
[0072] For a more detailed understanding of the image information extraction method provided in the embodiment of the present application, please continue to refer to Figure 2 , Figure 2 : is a flow chart of the image information extraction method provided in the embodiment of the present application. The image information extraction method may include the following steps:
[0073] Step 110, dividing the original image into a plurality of sub-images, and determining the target image feature corresponding to each sub-image;
[0074] Step 120: determining a description text corresponding to the original image, and determining target text features corresponding to the description text;
[0075] Step 130: Concatenate the target text feature with each target image feature to obtain the target image-text feature corresponding to each sub-image;
[0076] Step 140: input the target text-image features and the target image features into a trained decoding model, and output a target sub-image related to the description text, wherein the trained decoding model is trained based on the corresponding loss between the prediction sub-sample image and the label sub-image;
[0077] Step 150: extract the entity corresponding to the description text according to the target sub-image.
[0078] Steps 110 to 150 will be described in detail below.
[0079] In step 110, the original image is segmented into a plurality of sub-images, and the target image feature corresponding to each sub-image is determined.
[0080] Among them, the original image can be divided into multiple sub-images, and the image ratio of each sub-image is the same as that of the original image, that is, the multiple sub-images are obtained by dividing in equal proportions, and then the target image feature corresponding to each sub-image is determined. The target image feature can specifically be a vector after encoding the sub-image.
[0081] In some implementations, determining a target image feature corresponding to each sub-image includes:
[0082] (1.1) Input each sub-image into the pre-trained convolutional neural network and output the image features corresponding to each sub-image;
[0083] (1.2) The image features corresponding to each sub-image are input into the trained first encoding model, and the target image features corresponding to each sub-image are output.
[0084] Specifically, each sub-image can be input into a pre-trained convolutional neural network, which can realize feature extraction for each sub-image, thereby outputting image features corresponding to each sub-image. The image features can be understood as preliminary feature extraction of each sub-image, which can represent the image information of each sub-image.
[0085] Then, the image features corresponding to each sub-image are input into the trained first coding model, and the trained first coding model outputs the target image features corresponding to each sub-image. In this way, the feature expression of the high-dimensional space of each sub-image can be realized, thereby converting the image information of each sub-image into the target image features that can be recognized by the machine, which can improve the data processing efficiency.
[0086] To get a clearer understanding of the first encoding model after training, please refer to Figure 3 , Figure 3 1 is a schematic diagram of the structure of an image information extraction model provided in an embodiment of the present application. The image information extraction model includes a trained first encoding model, a trained second encoding model and a decoding model.
[0087] The input of the trained first coding model is the image feature of each sub-image, and the output of the trained first coding model is the target image feature of each sub-image. The trained first coding model can specifically be a model with an attention mechanism, and the trained first coding model can obtain the dependency relationship between different sub-images through the attention mechanism.
[0088] In some implementations, inputting the image feature corresponding to each sub-image into the trained first encoding model, and outputting the image feature corresponding to each sub-image includes:
[0089] (1.2.1) Inputting the image features corresponding to each sub-image into the first sub-first attention model to obtain the sub-first attention features output by the first sub-first attention model;
[0090] (1.2.2) inputting the sub-first attention feature output by the first sub-first attention model into the next sub-first attention model to obtain the sub-first attention feature output by the next sub-first attention model;
[0091] (1.2.3) Repeatedly input the sub-first attention features output by the previous sub-first attention model into the next sub-first attention model until the last sub-first attention model outputs the target image features corresponding to each sub-image.
[0092] Please combine Figure 4 , Figure 4 : is a schematic diagram of the structure of the trained first coding model provided in an embodiment of the present application. The first coding model includes a plurality of sequentially connected sub-first attention models, each of which can be a self-attention model.
[0093] Among them, the input of the first sub-first attention model is the image features corresponding to each sub-image, and then the first sub-first attention model can perform attention calculation between each sub-image according to the image features, so as to obtain the sub-first attention features corresponding to each sub-image output by the first sub-first attention model.
[0094] Then the sub-first attention features corresponding to each sub-image output by the first sub-first attention model are input into the second sub-first attention model, and the second sub-first attention model outputs the sub-first attention features corresponding to each sub-image.
[0095] Repeatedly input the sub-first attention features output by the previous sub-first attention model into the next sub-first attention model until the last sub-first attention model outputs the target image features corresponding to each sub-image.
[0096] Through multiple sub-first attention modules, the features of each sub-image are continuously extracted, so as to more accurately determine the attention relationship between different sub-images, obtain the dependency relationship between different sub-images through the attention relationship, and finally obtain the target image features corresponding to each sub-image. The target image features can be understood as vectors, which are the feature representations of the high-dimensional space of each sub-image.
[0097] In step 120, the description text corresponding to the original image is determined, and the target text features corresponding to the description text are determined.
[0098] The original image corresponds to a description text, which can be used to describe the entity that needs to be extracted in the original image. For example, the original image contains the aorta and veins, and the description text of the original image is "extract the aorta image in the image", then the aorta is the entity that needs to be extracted in the original image.
[0099] Then, a target text feature corresponding to the description text is determined. The target text feature can be understood as a high-dimensional space representation of the description text, which enables a computer device to recognize the description text.
[0100] In some implementations, determining the target text feature corresponding to the description text includes:
[0101] (1.1) dividing the description text into multiple sub-texts;
[0102] (1.2) determining the text features corresponding to each sub-text, and inputting the text features into the trained second encoding model, and outputting the sub-target text features corresponding to each sub-text;
[0103] (1.3) Determine the target text features corresponding to the description text based on the sub-target text features corresponding to each sub-text.
[0104] Specifically, the description text can be divided according to specific words, for example, multiple words are determined according to semantics, and then the description text is divided according to the multiple words to obtain multiple sub-texts. After obtaining multiple sub-texts, each sub-text can be encoded to obtain the text features corresponding to each sub-text.
[0105] Then, the text features are input into the trained second encoding model, and the sub-target text features corresponding to each sub-text are output. The trained second encoding model can specifically be a model with an attention mechanism, and the trained second encoding model can obtain the dependency relationship between different sub-texts through the attention mechanism.
[0106] Specifically, please combine Figure 5 , Figure 5It is a structural diagram of the trained second encoding model provided by an embodiment of the present application. Among them, the trained second encoding model includes a plurality of sub-second attention models connected in sequence, and the text features corresponding to each sub-text can be input into the first sub-second attention model to obtain the sub-second attention features output by the first sub-second attention model; the sub-second attention features output by the first sub-second attention model are input into the next sub-second attention model to obtain the sub-second attention features output by the next sub-second attention model; the sub-second attention features output by the previous sub-second attention model are repeatedly input into the next sub-second attention model until the last sub-second attention model outputs the sub-target text features corresponding to each sub-text.
[0107] Through multiple sub-second attention modules, the features of each sub-text are continuously extracted, so as to more accurately determine the attention relationship between different sub-texts, obtain the dependency relationship between different sub-texts through the attention relationship, and finally obtain the sub-target text features corresponding to each sub-text. Finally, the target text features corresponding to the description text are determined based on the sub-target text features corresponding to each sub-text. The sub-target text features can be understood as vectors, which are the feature representations of the high-dimensional space of each sub-text, while the target text features can be understood as a matrix composed of vectors.
[0108] In step 130, the target text feature is concatenated with each target image feature to obtain the target image-text feature corresponding to each sub-image.
[0109] After obtaining the target text features and the target image features, the target text features are concatenated with each target image feature to obtain the target image-text features corresponding to each sub-image.
[0110] The target graphic feature can be understood as the target image feature of the sub-image of the original image combined with the target text feature of the descriptive text. This allows each sub-image to have the semantic information corresponding to the descriptive text, thereby achieving enhanced expression of the features of each sub-image. In other words, the target graphic feature has the image information of each sub-image and the semantic information of the descriptive text. The trained decoding model can obtain more semantic information about the original image based on the target graphic feature, which helps the trained decoding model to determine the target sub-image among multiple sub-images, thereby improving the accuracy of determining the target sub-image.
[0111] In step 140, the target graphic features and the target image features are input into the trained decoding model, and the target sub-image related to the description text is output. The trained decoding model is trained based on the corresponding loss between the predicted sub-sample image and the label sub-image.
[0112] After obtaining the target image-text features, the target image features and the target image-text features can be input into the trained decoding model, and the trained decoding model outputs the target sub-image related to the description text. The trained decoding model is trained based on the corresponding loss between the predicted sub-sample image and the label sub-image, and the specific training process will be described later.
[0113] In some implementations, the target graphic features and the target image features are input into a trained decoding model, and a target sub-image associated with the description text is output, including:
[0114] (1.1) Inputting the target text and image features into the trained decoding model to obtain the correlation score between the target text and image features;
[0115] (1.2) Determine the sub-image corresponding to the target image feature whose relevance score is higher than the preset score value, and determine the sub-image corresponding to the target image feature whose relevance score is higher than the preset score value as the target sub-image related to the description text.
[0116] In the trained decoding model, the target graphic features can be used as query inputs, and the target image features can be used as queried inputs. In this way, the trained decoding model can query the target image features through the target graphic features, thereby determining the dependency relationship between the target graphic features and the target image features, which can be specifically expressed as a correlation score between the target graphic features and the target image features. The target image features with higher correlation scores are determined from the target image features through the correlation scores, and then the sub-image corresponding to the target image features with higher correlation scores is determined as the target sub-image.
[0117] The target sub-image can be understood as a target sub-image related to the entity described in the description text, and the entity described in the description text is included in the target sub-image.
[0118] It can be understood that in the above description, it can be determined that the target graphic features contain the semantic information corresponding to the original image, so that the trained decoding model can obtain more information about each sub-image, so that it can more accurately capture the dependency between the target graphic features and the target image features, that is, the correlation score, so as to accurately determine the target sub-image related to the description text.
[0119] The trained decoding model contains multiple sub-decoding models connected in sequence. The output of the previous sub-decoding model and the target image features are used as the input of the next sub-decoding model. The input of the first sub-decoding model is the target image features and the target graphic features. The sub-decoding model can be a cross-attention model that can obtain the dependency between the target graphic features and the target image features.
[0120] In step 150, entities corresponding to the description text are extracted according to the target sub-image.
[0121] After obtaining the target sub-image, the entity corresponding to the description text may be extracted from the target sub-image. Specifically, the target sub-image may be segmented to segment the image region corresponding to the entity.
[0122] In some implementations, extracting an entity corresponding to the description text according to the target sub-image includes:
[0123] (1.1) Determine the outline of the entity corresponding to the description text in the target sub-image;
[0124] (1.2) Divide the entities in the target sub-image based on the contour to obtain the entities.
[0125] Specifically, in the target sub-image, the outline of the entity corresponding to the description text can be determined, and then the entities in the target sub-image can be divided according to the outline, so as to accurately determine the corresponding entity in the target sub-image, thereby achieving accurate extraction of the entity described by the description text in the original image.
[0126] From the above, it can be seen that in the embodiments of the present application, by combining the target text features of the descriptive text and the target image features of the original image, the trained decoding model can obtain more semantic information about the original image, thereby achieving more accurate extraction of the entities corresponding to the descriptive text in the original image, thereby increasing the accuracy of entity extraction in the original image.
[0127] Please also read Figure 6 , Figure 6 It is a training flow diagram of the trained image information extraction model provided in the embodiment of the present application. It should be noted that the trained image information extraction model is obtained through image information extraction model training, and the image information extraction model includes a first encoding model, a second encoding model and a decoding model, the first encoding model and the trained first encoding model have the same model structure, the second encoding model and the trained second encoding model have the same model structure, and the decoding model and the trained decoding model have the same model structure.
[0128] The trained first encoding model, the trained second encoding model and the trained decoding model described above can be obtained in the following manner:
[0129] Step 201: Segment the sample image into multiple sub-sample images, input each sub-sample image into a pre-trained convolutional neural network, and output sample image features corresponding to each sub-sample image;
[0130] Step 202: input the sample image features corresponding to each sub-sample image into the first coding model, and output the target sample image features corresponding to each sub-sample image;
[0131] Step 203: determining a sample description text corresponding to the sample image and a sample text feature of the sample description text;
[0132] Step 204: input the sample text features into the second encoding model, and output the target sample text features of the sample description text;
[0133] Step 205: each target sample image feature is concatenated with the target sample text feature to obtain the target sample image and text feature;
[0134] Step 206: input the sample image and text features and the target sample image features into the decoding model, output the predicted sub-sample image related to the sample description text, and adjust the model parameters of the first encoding model, the second encoding model and the decoding model according to the loss between the predicted sub-sample image and the label sub-image when the loss does not meet the preset loss condition, and return to execute inputting the sample image features corresponding to each sub-sample image into the first encoding model, outputting the target sample image features corresponding to each sub-sample image, until the loss meets the preset loss adjustment, to obtain the trained first encoding model, the trained second encoding model and the trained decoding model.
[0135] Steps 201 to 206 will be described in detail below.
[0136] In step 201, the sample image is divided into a plurality of sub-sample images, each sub-sample image is input into a pre-trained convolutional neural network, and the sample image features corresponding to each sub-sample image are output.
[0137] The sample image can be divided into multiple sub-sample images, the ratio of each sub-sample image is consistent with the ratio of the sample image, and then each sub-sample image is input into a pre-trained convolutional neural network to output the sample image features corresponding to each sub-sample image.
[0138] In step 202, the sample image features corresponding to each sub-sample image are input into the first coding model, and the target sample image features corresponding to each sub-sample image are output.
[0139] Among them, the first encoding model can capture the attention relationship between sub-sample images through sample image features, and realize enhanced representation of each sub-sample image through this attention relationship, so as to obtain the target sample image features corresponding to each sub-sample image with higher spatial dimension.
[0140] In step 203, the sample description text corresponding to the sample image and the sample text features of the sample description text are determined.
[0141] Specifically, the sample description text is used to describe the target object that needs to be extracted in the sample image. The sample description text can be divided to obtain multiple sub-sample description texts, and then the sub-sample description texts are encoded to obtain the sample text features of each sub-sample description text, that is, the sample text features of the sample description text.
[0142] In step 204, the sample text features are input into the second encoding model, and the target sample text features of the sample description text are output.
[0143] The second encoding model can capture the attention relationship between the sub-sample description texts through the sample text features, and realize the enhanced representation of each sub-sample description text through the attention relationship, so as to obtain the sub-target sample text features corresponding to each sub-sample description text with higher spatial dimension. The target sample text features of the sample description text are composed according to the sub-target sample text features corresponding to each sub-sample description text.
[0144] In step 205, each target sample image feature is concatenated with the target sample text feature to obtain the target sample image and text feature.
[0145] Among them, each target sample image feature is concatenated with the target sample text feature to obtain the target sample image and text feature. The target sample image and text feature combines the semantic information of the sample image and can achieve a higher level of dimensional representation.
[0146] In step 206, the sample image and text features and the target sample image features are input into the decoding model, and the predicted sub-sample image related to the sample description text is output. Based on the loss between the predicted sub-sample image and the label sub-image, when the loss does not meet the preset loss condition, the model parameters of the first encoding model, the second encoding model and the decoding model are adjusted according to the loss, and the sample image features corresponding to each sub-sample image are input into the first encoding model, and the target sample image features corresponding to each sub-sample image are output until the loss meets the preset loss adjustment, thereby obtaining the trained first encoding model, the trained second encoding model and the trained decoding model.
[0147] For example, when the loss between the predicted sub-sample image and the label sub-image is greater than a preset loss value, the model parameters of the first encoding model, the second encoding model and the decoding model can be adjusted to achieve iterative update of the image information extraction model, and then the next round of training data is input into the iterated image information extraction model until the loss between the predicted sub-sample image and the label sub-image is less than or equal to the preset loss value, and after subsequent input of training data, the loss is still less than or equal to the preset loss value, then it means that the training of the image information extraction model is completed, and the trained image information extraction model is obtained, and the trained image information extraction model includes the trained first encoding model, the trained second encoding model and the trained decoding model.
[0148] From the above content, it can be seen that during the training process, the image information extraction model in this application inputs the sample text features corresponding to the sample image, so the image information extraction model can capture the semantic information of the input original image, thereby accurately determining the entity that needs to be extracted described by the descriptive text in the original image.
[0149] In an embodiment of the present application, an original image is divided into multiple sub-images, and the target image features corresponding to each sub-image are determined; the description text corresponding to the original image is determined, and the target text features corresponding to the description text are determined; the target text features are respectively spliced with each target image feature to obtain the target graphic features corresponding to each sub-image; the target graphic features and the target image features are input into the trained decoding model, and the target sub-image related to the description text is output; the entity corresponding to the description text is extracted according to the target sub-image, wherein the trained decoding model is trained based on the corresponding loss between the predicted sub-sample image and the label sub-image. In this way, the target graphic features are obtained by combining the target image features of the original image with the target text features of the description text of the original image, thereby enhancing the feature dimension of the image information of the original image and enhancing the expression of the image information of the original image, so that the trained decoding model can accurately determine the target sub-image in multiple sub-images according to the target graphic features, and finally accurately extract the entity corresponding to the description text in the target sub-image. Compared with the related art that only extracts the entity to be extracted by the pixel information of the image, the entity corresponding to the description text can be more accurately extracted in the original image in the present application.
[0150] See also Figure 7 , Figure 7 : is another flow chart of the image information extraction method provided in the embodiment of the present application. The image information extraction method may specifically include the following steps:
[0151] Step 301: Segment the original image into multiple sub-images, input each sub-image into a pre-trained convolutional neural network, and output image features corresponding to each sub-image;
[0152] Step 302: input the image features corresponding to each sub-image into the trained first encoding model, and output the target image features corresponding to each sub-image;
[0153] Step 303: divide the description text into multiple sub-texts;
[0154] Step 304: determine the text features corresponding to each sub-text, input the text features into the trained second encoding model, and output the target text features corresponding to each sub-text;
[0155] Step 305: Determine the target text feature corresponding to each sub-text as the target text feature corresponding to the description text;
[0156] Step 306: Concatenate the target text feature with each target image feature to obtain the target image-text feature corresponding to each sub-image;
[0157] Step 307: input the target graphic features and the target image features into the trained decoding model to obtain a correlation score between the target graphic features and the target image features;
[0158] Step 308: determine the sub-image corresponding to the target image feature whose correlation score is higher than the preset score value, and determine the sub-image corresponding to the target image feature whose correlation score is higher than the preset score value as the target sub-image related to the description text;
[0159] Step 309: determining the outline of the entity corresponding to the description text in the target sub-image;
[0160] Step 310: Divide the entities in the target sub-image based on the contour to obtain entities.
[0161] The specific implementation of the image information extraction method is basically the same as the specific embodiment of the above-mentioned image information extraction method, and will not be repeated here.
[0162] See also Figure 8 , Figure 8: It is a schematic diagram of the structure of the image information extraction device provided in the embodiment of the present application. The image information extraction device can implement the above-mentioned image information extraction method. In the embodiment of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that contains the function of the module or unit.
[0163] The image information extraction device 400 may include:
[0164] The segmentation module 410 is used to segment the original image into multiple sub-images and determine the target image features corresponding to each sub-image;
[0165] A determination module 420 is used to determine the description text corresponding to the original image and determine the target text features corresponding to the description text;
[0166] A splicing module 430 is used to splice the target text feature with each target image feature to obtain the target image-text feature corresponding to each sub-image;
[0167] An input module 440 is used to input the target text-image features and the target image features into a trained decoding model, and output a target sub-image related to the description text, wherein the trained decoding model is trained based on the corresponding loss between the prediction sub-sample image and the label sub-image;
[0168] The extraction module 450 is used to extract the entity corresponding to the description text according to the target sub-image.
[0169] In some embodiments, the segmentation module 410 includes a first input submodule 411 and a second input submodule 412; the first input submodule 411 is used to input each sub-image into a pre-trained convolutional neural network and output image features corresponding to each sub-image;
[0170] The second input submodule 412 is used to input the image features corresponding to each sub-image into the trained first encoding model, and output the target image features corresponding to each sub-image.
[0171] In some embodiments, the trained first encoding model includes a plurality of sequentially connected sub-first attention models, and the second input submodule 412 is used to input the image feature corresponding to each sub-image into the first sub-first attention model to obtain the sub-first attention feature output by the first sub-first attention model;
[0172] Input the sub-first attention feature output by the first sub-first attention model into the next sub-first attention model to obtain the sub-first attention feature output by the next sub-first attention model;
[0173] Repeatedly input the sub-first attention features output by the previous sub-first attention model into the next sub-first attention model until the last sub-first attention model outputs the target image features corresponding to each sub-image.
[0174] In some implementations, the determination module 420 is configured to divide the description text into multiple sub-texts;
[0175] Determine the text features corresponding to each of the sub-texts, input the text features into the trained second encoding model, and output the sub-target text features corresponding to each of the sub-texts;
[0176] The target text feature corresponding to the description text is determined according to the sub-target text feature corresponding to each of the sub-texts.
[0177] In some embodiments, the input module 440 is used to input the target graphic features and the target image features into the trained decoding model to obtain a correlation score between the target graphic features and the target image features;
[0178] A sub-image corresponding to a target image feature having a correlation score higher than a preset score value is determined, and the sub-image corresponding to the target image feature having a correlation score higher than the preset score value is determined as a target sub-image related to the description text.
[0179] In some embodiments, the extraction module 450 is used to determine the outline of the entity corresponding to the description text in the target sub-image;
[0180] The entities in the target sub-image are divided based on the contour to obtain the entities.
[0181] In some embodiments, the image information extraction model further includes a training module 460, which is used to divide the sample image into a plurality of sub-sample images, input each sub-sample image into a pre-trained convolutional neural network, and output sample image features corresponding to each sub-sample image;
[0182] Inputting sample image features corresponding to each sub-sample image into a first coding model, and outputting target sample image features corresponding to each sub-sample image;
[0183] Determine a sample description text corresponding to the sample image and a sample text feature of the sample description text;
[0184] Inputting the sample text features into the second encoding model, and outputting the target sample text features of the sample description text;
[0185] The target sample image features and the target sample text features are concatenated to obtain the target sample image and text features;
[0186] The sample image and text features and the target sample image features are input into the decoding model, and the predicted sub-sample image related to the sample description text is output. According to the loss between the predicted sub-sample image and the label sub-image, when the loss does not meet the preset loss condition, the model parameters of the first encoding model, the second encoding model and the decoding model are adjusted according to the loss, and the sample image features corresponding to each sub-sample image are input into the first encoding model, and the target sample image features corresponding to each sub-sample image are output until the loss meets the preset loss adjustment, thereby obtaining the trained first encoding model, the trained second encoding model and the trained decoding model.
[0187] In an embodiment of the present application, the segmentation module 410 segments the original image into multiple sub-images and determines the target image features corresponding to each sub-image; the determination module 420 determines the description text corresponding to the original image, and determines the target text features corresponding to the description text; the determination module 420 concatenates the target text features with each target image feature to obtain the target graphic features corresponding to each sub-image; the determination module 420 inputs the target graphic features and the target image features into the trained decoding model, and outputs the target sub-image related to the description text; the extraction module 450 extracts the entity corresponding to the description text according to the target sub-image, wherein the trained decoding model is trained based on the corresponding loss between the predicted sub-sample image and the label sub-image. In this way, by combining the target image features of the original image and the target text features of the descriptive text of the original image, the target graphic features are obtained, thereby enhancing the feature dimension of the image information of the original image and enhancing the expression of the image information of the original image. In this way, the trained decoding model can accurately determine the target sub-image among multiple sub-images based on the target graphic features, and finally accurately extract the entity corresponding to the descriptive text in the target sub-image. Compared with the related art that only extracts the entities to be extracted through the pixel information of the image, the present application can more accurately extract the entities corresponding to the descriptive text in the original image.
[0188] The specific implementation of the image information extraction device is basically the same as the specific implementation of the above-mentioned image information extraction method, and will not be repeated here.
[0189] The embodiment of the present application also provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image information extraction method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0190] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0191] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;
[0192] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the image information extraction method of the embodiment of this application;
[0193] Input / output interface 903, used to implement information input and output;
[0194] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);
[0195] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);
[0196] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .
[0197] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned image information extraction method is implemented.
[0198] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0199] The image information extraction method, image information extraction device, computer equipment and computer-readable storage medium provided in the embodiments of the present application combine the target image features of the original image with the target text features of the description text of the original image to obtain the target graphic features, thereby enhancing the feature dimension of the image information of the original image and enhancing the expression of the image information of the original image. In this way, the trained decoding model can accurately determine the target sub-image among multiple sub-images based on the target graphic features, and finally accurately extract the entity corresponding to the description text in the target sub-image. Compared with the related art that only extracts the entity to be extracted by using the pixel information of the image, the present application can more accurately extract the entity corresponding to the description text in the original image.
[0200] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0201] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0202] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0203] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0204] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0205] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0206] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0207] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0208] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0209] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0210] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.
Claims
1. A method for extracting image information, It is characterized in that include: Segmenting the original image into multiple sub-images, and determining the target image features corresponding to each of the sub-images; Determining a description text corresponding to the original image, and determining a target text feature corresponding to the description text; The target text feature is respectively concatenated with each target image feature to obtain a target image-text feature corresponding to each sub-image; Inputting the target graphic features and the target image features into a trained decoding model, and outputting a target sub-image related to the description text, wherein the trained decoding model is trained based on the corresponding loss between the prediction sub-sample image and the label sub-image; An entity corresponding to the description text is extracted according to the target sub-image.
2. The image information extraction method according to claim 1, It is characterized in that The determining of the target image feature corresponding to each of the sub-images comprises: Input each of the sub-images into a pre-trained convolutional neural network, and output image features corresponding to each of the sub-images; The image features corresponding to each of the sub-images are input into the trained first encoding model, and the target image features corresponding to each of the sub-images are output.
3. The image information extraction method according to claim 2, It is characterized in that The trained first encoding model includes a plurality of sequentially connected sub-first attention models, and the image features corresponding to each of the sub-images are input into the trained first encoding model, and the image features corresponding to each of the sub-images are output, including: Inputting the image features corresponding to each of the sub-images into the first sub-first attention model to obtain the sub-first attention features output by the first sub-first attention model; Inputting the sub-first attention feature output by the first sub-first attention model into the next sub-first attention model to obtain the sub-first attention feature output by the next sub-first attention model; Repeatedly input the sub-first attention features output by the previous sub-first attention model into the next sub-first attention model until the last sub-first attention model outputs the target image features corresponding to each sub-image.
4. The image information extraction method according to claim 1, It is characterized in that The determining of the target text feature corresponding to the description text includes: Performing text division on the description text to obtain multiple sub-texts; Determine the text features corresponding to each of the sub-texts, input the text features into the trained second encoding model, and output the sub-target text features corresponding to each of the sub-texts; The target text feature corresponding to the description text is determined according to the sub-target text feature corresponding to each of the sub-texts.
5. The image information extraction method according to claim 1, It is characterized in that The step of inputting the target graphic features and the target image features into a trained decoding model and outputting a target sub-image related to the description text comprises: Inputting the target graphic features and the target image features into a trained decoding model to obtain a correlation score between the target graphic features and the target image features; The sub-image corresponding to the target image feature whose correlation score is higher than a preset score value is determined, and the sub-image corresponding to the target image feature whose correlation score is higher than the preset score value is determined as a target sub-image related to the description text.
6. The image information extraction method according to claim 1, It is characterized in that Extracting an entity corresponding to the description text according to the target sub-image includes: Determining the outline of the entity corresponding to the description text in the target sub-image; The entities in the target sub-image are divided based on the contour to obtain the entities.
7. The image information extraction method according to claim 1, It is characterized in that The trained decoding model is trained as follows: Divide the sample image into multiple sub-sample images, input each of the sub-sample images into a pre-trained convolutional neural network, and output sample image features corresponding to each of the sub-sample images; Inputting sample image features corresponding to each of the sub-sample images into a first coding model, and outputting target sample image features corresponding to each of the sub-sample images; Determining a sample description text corresponding to the sample image and a sample text feature of the sample description text; Inputting the sample text features into a second encoding model, and outputting target sample text features of the sample description text; Each target sample image feature is concatenated with the target sample text feature to obtain a target sample image and text feature; The sample image and text features and the target sample image features are input into the decoding model, and the predicted sub-sample image related to the sample description text is output. Based on the loss between the predicted sub-sample image and the label sub-image, when the loss does not meet the preset loss condition, the model parameters of the first encoding model, the second encoding model and the decoding model are adjusted according to the loss, and the sample image features corresponding to each of the sub-sample images are input into the first encoding model, and the target sample image features corresponding to each of the sub-sample images are output until the loss meets the preset loss adjustment, thereby obtaining the trained first encoding model, the trained second encoding model and the trained decoding model.
8. An image information extraction device, It is characterized in that include: A segmentation module, used to segment the original image into multiple sub-images and determine the target image features corresponding to each of the sub-images; A determination module, used to determine the description text corresponding to the original image, and determine the target text features corresponding to the description text; A splicing module, used for splicing the target text feature with each target image feature to obtain the target image-text feature corresponding to each sub-image; An input module, used for inputting the target graphic features and the target image features into a trained decoding model, and outputting a target sub-image related to the description text, wherein the trained decoding model is trained based on the corresponding loss between the prediction sub-sample image and the label sub-image; The extraction module is used to extract the entity corresponding to the description text according to the target sub-image.
9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the image information extraction method according to any one of claims 1 to 7.
10. A computer device comprising a memory, a processor and a computer program stored in the memory and capable of being run on the processor, It is characterized in that When the processor executes the computer program, the image information extraction method according to any one of claims 1 to 7 is implemented.
Citation Information
Cited By
Image target counting method and device based on image text pairs
CN119763128A
