Image category recognition method, device, electronic device and storage device

Abstract

The present application discloses a method for identifying the category of an image, comprising: obtaining a target image; generating a graph structure according to the target image; obtaining vector feature information of the graph structure according to the graph structure; inputting the vector feature information into a vector classification model to obtain the category of the target image; the vector classification model is used to obtain the category of the image corresponding to the vector feature information according to the input vector feature information. The above method is used to solve the problem that a large amount of data training is required due to much interference when classifying document images according to pixel features in the prior art.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and specifically to a method, device, electronic device and storage device for identifying image categories; the present application also relates to an image search method, a method for identifying image identity cards, and a method for processing image tickets; the present application also relates to another method for identifying image categories and a method for constructing a knowledge graph. Background Art

[0002] Currently, optical character recognition technology (e.g., optical character recognition (OCR)) provides optical character recognition and information structuring services for specific document images, but this technology requires knowing in advance what type of document the document image belongs to. In real scenarios, many different types of document data are mixed together. Manually marking document types requires additional costs and also has an error rate. Users hope to have an automated data classification solution.

[0003] In the existing technology, when classifying document images, CNN (convolutional neural network) is used to extract the pixel features of the document image, and then the pixel features of the document image are input into the classifier to determine the image type of the document image. The disadvantage of this solution is that the pixel-level features are subject to a lot of interference, such as light and shadow, blur, color tone, shooting angle, etc., so a large amount of data is often required for training. Summary of the invention

[0004] The present application provides a method, device, electronic device and storage device for identifying image categories to solve the problem in the prior art that a large amount of data training is required due to much interference when classifying document images according to pixel features.

[0005] This application provides a method for identifying image categories, including:

[0006] Get the target image;

[0007] Obtaining a text block containing coordinate information and text content according to the target image;

[0008] generating a graph structure according to the text block;

[0009] Obtaining vector feature information of the graph structure according to the graph structure;

[0010] The vector feature information is input into a vector classification model to obtain the category of the target image; the vector classification model is used to obtain the category of the image corresponding to the vector feature information according to the input vector feature information.

[0011] Optionally, obtaining the vector feature information of the graph structure according to the graph structure includes:

[0012] Obtaining text block feature information of the graph structure according to the graph structure;

[0013] Converting text block feature information of the graph structure into vector feature information;

[0014] Optionally, generating a graph structure according to the text block includes:

[0015] Converting text content in the text block into vector feature information of a node as feature information of a node of a graph structure corresponding to the text block;

[0016] The relative position information and relative width and height information between the two text blocks are used as the feature information of the edge between the nodes corresponding to the two text blocks;

[0017] A graph structure is generated according to the feature information of the nodes and the feature information of the edges between the nodes.

[0018] Optionally, the using the relative position information and the relative width and height information between the two text blocks as the feature information of the edge between the nodes corresponding to the two text blocks includes:

[0019] According to the coordinate information of the text blocks corresponding to the two nodes, two rectangles corresponding to the text blocks are obtained;

[0020] Normalize the widths and heights of the two rectangles to obtain two normalized rectangles;

[0021] The relative position information between the corresponding vertices of the two rectangles and the normalized width information and height information of the rectangles are used as feature information of the edge between the nodes corresponding to the two text blocks.

[0022] Optionally, obtaining text block feature information of the graph structure according to the graph structure includes:

[0023] The feature information of each node in the graph structure and the feature information of all edges contained in the graph structure are combined and processed as the text block feature information of the graph structure.

[0024] Optionally, generating a graph structure according to the feature information of the nodes and the feature information of the edges between the nodes includes:

[0025] A fully connected directed graph is generated according to the feature information of the nodes and the feature information of the edges between the nodes.

[0026] Optionally, inputting the vector feature information into a vector classification model to obtain the category of the target image includes:

[0027] Inputting the vector feature information into a vector classification model, and the vector classification model outputs a probability value that the vector feature information belongs to each category;

[0028] Obtaining the category corresponding to the vector feature information according to the probability value that the vector feature information belongs to each category;

[0029] The category corresponding to the vector feature information is used as the category of the target image.

[0030] Optionally, obtaining the category corresponding to the vector feature information according to the probability value that the vector feature information belongs to each category includes:

[0031] Selecting the maximum probability value from the probability values ​​that the vector feature information belongs to each category;

[0032] Determining whether the maximum probability value is greater than or equal to a preset probability threshold;

[0033] If so, the picture category corresponding to the maximum probability value is used as the category corresponding to the vector feature information.

[0034] Optionally, the target image is a document image containing text information.

[0035] The present application also provides a device for identifying image categories, including:

[0036] A target image obtaining unit, used for obtaining a target image;

[0037] A text block obtaining unit, used to obtain a text block containing coordinate information and text content according to the target image;

[0038] A graph structure generating unit, used for generating a graph structure according to the text block;

[0039] A vector feature information obtaining unit, used to obtain vector feature information of a graph structure according to the graph structure;

[0040] The target image category obtaining unit is used to input the vector feature information into a vector classification model to obtain the category of the target image; the vector classification model is used to obtain the category of the image corresponding to the vector feature information based on the input vector feature information.

[0041] The present application also provides an electronic device, comprising:

[0042] Processor; and

[0043] The memory is used to store a program of a method for identifying a picture category. After the device is powered on and the program of the method for identifying a picture category is run by the processor, the following steps are performed:

[0044] Get the target image;

[0045] Obtaining a text block containing coordinate information and text content according to the target image;

[0046] generating a graph structure according to the text block;

[0047] Obtaining vector feature information of the graph structure according to the graph structure;

[0048] The vector feature information is input into a vector classification model to obtain the category of the target image; the vector classification model is used to obtain the category of the image corresponding to the vector feature information according to the input vector feature information.

[0049] The present application also provides a storage device storing a program of a method for identifying a picture category, wherein the program is executed by a processor to perform the following steps:

[0050] Get the target image;

[0051] Obtaining a text block containing coordinate information and text content according to the target image;

[0052] generating a graph structure according to the text block;

[0053] Obtaining vector feature information of the graph structure according to the graph structure;

[0054] The vector feature information is input into a vector classification model to obtain the category of the target image; the vector classification model is used to obtain the category of the image corresponding to the vector feature information according to the input vector feature information.

[0055] This application provides an image search method, including:

[0056] Get the image to be searched;

[0057] According to the picture to be searched, generating a graph structure of the picture to be searched;

[0058] Obtaining vector feature information of the graph structure;

[0059] Using a vector classification model, obtaining the category of the image to be searched;

[0060] According to the category of the picture to be searched, information data of the picture to be searched is output.

[0061] The present application provides a method for identifying a picture ID card, comprising:

[0062] Obtain a picture ID card to be identified;

[0063] Generating a graph structure of the picture ID card according to the picture ID card;

[0064] Using a vector classification model, obtaining the category of the picture identity card;

[0065] Outputting confirmation information to the user for confirming whether the category of the picture ID card is correct;

[0066] Get confirmation information entered by the user;

[0067] According to the confirmation information input by the user, the parameters of the vector classification model are adjusted.

[0068] The present application provides a method for processing a picture ticket, including:

[0069] Obtain the image ticket to be identified;

[0070] According to the picture ticket, generating a graph structure of the picture ticket;

[0071] Using a vector classification model, obtaining the category of the image ticket;

[0072] If the category of the picture ticket is an invoice, the amount of the picture ticket is counted to obtain the total amount of the invoice.

[0073] This application provides a method for identifying image categories, including:

[0074] Get the target image;

[0075] Obtaining a text block containing coordinate information and text content according to the target image;

[0076] According to the target image, generating a graph structure of the target image;

[0077] Obtaining vector feature information of the graph structure according to the graph structure;

[0078] Inputting the vector feature information into a vector classification model to obtain a candidate category of the target image; the vector classification model is used to obtain the category of the image corresponding to the vector feature information according to the input vector feature information;

[0079] It is determined whether the candidate category of the target image matches the text block; if so, the candidate category of the target image is determined as the category of the target image.

[0080] This application provides a method for constructing a knowledge graph, including:

[0081] In a training process of a vector classification model for image classification, obtaining training features of the vector classification model;

[0082] According to the training features, obtaining entity information of the image and the relationship between entities;

[0083] Construct an image knowledge graph based on the entity information and the relationships between the entities.

[0084] Compared with the prior art, this application has the following advantages:

[0085] The present application provides a method for identifying the category of an image, including: obtaining a target image; obtaining a text block containing coordinate information and text content according to the target image; generating a graph structure according to the text block; obtaining vector feature information of the graph structure according to the graph structure; inputting the vector feature information into a vector classification model to obtain the category of the target image; the vector classification model is used to obtain the category of the image corresponding to the vector feature information according to the input vector feature information. The method for identifying the category of an image provided by the present application first converts the target image into a text block; then generates a graph structure according to the text block; then obtains the vector feature information of the graph structure according to the graph structure, and classifies the document image according to the vector feature information. Since the vector feature information is obtained according to the text block, it will not be affected by light and shadow, blur, hue, shooting angle, etc., and does not require a large amount of data for training, which solves the problem of the prior art that a large amount of data training is required due to the large amount of interference when classifying document images according to pixel features. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1A This is an application scenario diagram of a method for identifying image categories provided in the first embodiment of the present application.

[0087] Figure 1 This is a flowchart of a method for identifying a picture category provided in the first embodiment of the present application.

[0088] Figure 2 It is a schematic diagram provided by the first embodiment of the present application of obtaining two rectangles corresponding to text blocks according to the coordinate information of the text blocks corresponding to two nodes.

[0089] Figure 3 A schematic diagram of a fully connected directed graph containing 3 nodes.

[0090] Figure 4 It is a schematic diagram of a device for identifying picture categories provided in the second embodiment of the present application.

[0091] Figure 5 It is a schematic diagram of an electronic device provided in the third embodiment of the present application. DETAILED DESCRIPTION

[0092] Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention, so the present invention is not limited to the specific implementation disclosed below.

[0093] In order to present the present application more clearly, the application scenario of the method for identifying the image category provided in the first embodiment of the present application is briefly introduced.

[0094] The image category recognition method provided in the first embodiment of the present application can be applied to the scene where the client and the server interact, such as Figure 1A When it is necessary to identify the category of a target image, the client usually first establishes a connection with the server. After the connection is established, the client sends the target image to the server. After the server receives the target image, the server obtains a text block containing coordinate information and text content based on the target image; then generates a graph structure based on the text block; then obtains vector feature information of the graph structure based on the graph structure; inputs the vector feature information into a vector classification model to obtain the category of the target image, provides the category of the target image to the client, and the client receives the category of the target image.

[0095] The first embodiment of the present application provides a method for identifying a picture category. Figure 1 Make an introduction.

[0096] like Figure 1 As shown, in step S101, a target picture is obtained.

[0097] The target image is an image of the category to be obtained. The target image may be a document image containing text information, such as an ID card image or a marriage certificate image. The target image may be an image obtained by scanning an original image (e.g., an original ID card), or may be an image obtained by photographing the original image.

[0098] The image category recognition method can be run on the server, and the target image can be a target image obtained from the client. When the user needs to obtain the category of the target image, the target image can be uploaded to the server through the client, and the server obtains the target image sent by the client. The image category recognition method can also be run on the client.

[0099] like Figure 1 As shown, in step S102, a text block containing coordinate information and text content is obtained according to the target image.

[0100] In specific implementation, general OCR (optical character recognition) technology can be used to obtain a text block containing coordinate information and text content according to the target image. OCR technology refers to the technology of locating text coordinate information from the target image and identifying text content.

[0101] The text block refers to a rectangle containing text content.

[0102] The coordinate information of the text block refers to the coordinate information of the four vertices of the rectangle in the target image.

[0103] For example, the text content of a text block is "Shanghai XXX", and the coordinates of the four vertices of the rectangle containing the text content are: (86,162), (337,162), (337,182), (86,182).

[0104] According to a target image, multiple text blocks containing coordinate information and text content can be obtained. For example, according to an ID card image, 21 text blocks can be obtained, which are represented as text block 0, text block 1, and up to text block 20, and the information of each text block includes coordinate information and text content of the text block.

[0105] like Figure 1 As shown, in step S103, a graph structure is generated according to the text block.

[0106] The step of generating a graph structure according to the text block includes:

[0107] Converting text content in the text block into vector feature information of nodes as feature information of nodes of the graph structure corresponding to the text block;

[0108] The relative position information and relative width and height information between the two text blocks are used as the feature information of the edge between the nodes corresponding to the two text blocks;

[0109] A graph structure is generated according to the feature information of the nodes and the feature information of the edges between the nodes.

[0110] In a specific implementation, when converting the text content in a text block into vector feature information of a node, sentence embedding can be used to convert the text content in the text block into vector feature information of a node, wherein sentence embedding is a sentence vector technology, which is a method of representing a piece of text information as vector feature information.

[0111] The relative position information between the two text blocks refers to the position information between the text blocks expressed by relative coordinates. Considering that the positions of various types of certificates in the target image are uncertain, the absolute coordinates such as x1, x2, y1, y2 are discarded, and the relative coordinates such as x2-x1, y2-y1 are used to express the relative position between the two text blocks.

[0112] The relative width and height information between the two text blocks refers to the width and height information obtained by normalizing the width or height of the other text blocks with the width or height information of one of the text blocks as the denominator. Considering that the size of the document (e.g., ID card) in the target image cannot be determined when taking a photo, it is meaningless to use absolute values ​​such as w1, h1, w2, and h2. Here, the width or height information of one of the text blocks is used as the denominator, the width and height values ​​of the two text blocks are normalized, and the relative ratio is used as the feature information.

[0113] The method of using the relative position information and the relative width and height information between the two text blocks as the feature information of the edge between the nodes corresponding to the two text blocks includes:

[0114] According to the coordinate information of the text blocks corresponding to the two nodes, two rectangles corresponding to the text blocks are obtained;

[0115] Normalize the widths and heights of the two rectangles to obtain two normalized rectangles;

[0116] The relative position information between the corresponding vertices of the two rectangles and the normalized width information and height information of the rectangles are used as feature information of the edge between the nodes corresponding to the two text blocks.

[0117] like Figure 2 As shown, according to the coordinate information of the text blocks corresponding to the two nodes, two rectangles corresponding to the text blocks are obtained: rectangle 1 and rectangle 2, the width of rectangle 1 is w1, the height is h1, and the width of rectangle 2 is w2, the height is h2. Then, the height of rectangle 1 is h1 as the denominator, and the widths and heights of the two rectangles are normalized to obtain the two normalized rectangles; the relative distance between the corresponding vertices of the two rectangles and the width information and height information of the normalized rectangles are used as the feature information of the edge between the nodes corresponding to the two text blocks. The feature information of the edge can be a 5-dimensional vector (since h1 must be normalized to 1, it can be omitted), and the 5-dimensional vector includes: the width information w1 / h1 of rectangle 1 after normalization, the width information w2 / h1 of rectangle 2, the height information h2 / h1 of rectangle 2, and the x component (i.e. (x2-x1) / h1) and y component (i.e. (y2-y1) / h1) of the relative position between the upper left vertices of rectangle 1 and rectangle 2.

[0118] The step of obtaining text block feature information of the graph structure according to the graph structure includes:

[0119] The feature information of each node in the graph structure and the feature information of all edges contained in the graph structure are combined and processed as the text block feature information of the graph structure.

[0120] In the first embodiment of the present application, generating a graph structure according to the characteristic information of the nodes and the characteristic information of the edges between the nodes may include: generating a fully connected directed graph according to the characteristic information of the nodes and the characteristic information of the edges between the nodes.

[0121] For example, Figure 3 Schematic diagram of a fully connected directed graph containing 3 nodes.

[0122] In a specific implementation, in addition to generating a fully connected directed graph based on the feature information of the nodes and the feature information of the edges between the nodes, graphs of other structures may also be generated based on the feature information of the nodes and the feature information of the edges between the nodes.

[0123] like Figure 1 As shown, in step S104, vector feature information of the graph structure is obtained according to the graph structure.

[0124] The obtaining the vector feature information of the graph structure according to the graph structure includes:

[0125] Obtaining text block feature information of the graph structure according to the graph structure;

[0126] The text block feature information of the graph structure is converted into vector feature information.

[0127] In specific implementation, Graph Embedding can be used to convert the feature information of graph-structured text blocks into vector feature information. Graph Embedding: Graph embedding technology is a method of representing graph-structured data as vector features.

[0128] A document image category recognition scheme in the prior art uses a method of calibrating the keywords of the document, identifying the OCR results of the document image, and calculating the keyword matching rate to perform category recognition. The disadvantage of this scheme is that when the keyword differentiation is insufficient, for example, when 80% of the keywords in the text content of two document images overlap, it is difficult to distinguish the document images. Compared with the method of identifying document image categories based on keywords in the prior art, the first embodiment of the present application introduces relative position and relative width and height in the composition process according to the text block data. Therefore, in the process of obtaining the vector feature information of the graph structure according to the graph structure, not only the knowledge of keywords can be learned, but also the knowledge of the document layout can be learned. Data with the same keywords but different layouts can be distinguished, and the differentiation is stronger.

[0129] like Figure 1 As shown, in step S105, the vector feature information is input into a vector classification model to obtain the category of the target image; the vector classification model is used to obtain the category of the image corresponding to the vector feature information according to the input vector feature information.

[0130] The vector classification model refers to a target vector classification model obtained by training the original vector classification model using the vector feature information corresponding to certain categories of pictures and the category annotations of the vector feature information as a sample set. If the training samples include marriage certificate pictures, ID card pictures, and passport pictures, the vector classification model supports marriage certificate categories, ID card categories, and passport categories. If it is necessary to increase the picture categories supported by the vector classification model, it is necessary to increase the vector feature information corresponding to these categories of pictures and the category annotations of the vector feature information, merge them with the original sample set, and train a new vector classification model.

[0131] The step of inputting the vector feature information into a vector classification model to obtain the category of the target image includes:

[0132] Inputting the vector feature information into a vector classification model, and the vector classification model outputs a probability value that the vector feature information belongs to each category;

[0133] Obtaining the category corresponding to the vector feature information according to the probability value that the vector feature information belongs to each category;

[0134] The category corresponding to the vector feature information is used as the category of the target image.

[0135] The obtaining, according to the probability value of the vector feature information belonging to each category, the category corresponding to the vector feature information includes:

[0136] Selecting the maximum probability value from the probability values ​​that the vector feature information belongs to each category;

[0137] Determining whether the maximum probability value is greater than or equal to a preset probability threshold;

[0138] If so, the picture category corresponding to the maximum probability value is used as the category corresponding to the vector feature information.

[0139] In specific implementation, a Sigmoid classifier can be used as a vector classification model to classify vector feature information. Sigmoid function is often used as an activation function of a neural network to map variables between 0 and 1.

[0140] For example, if the vector feature information is input into the vector classification model, the vector classification model outputs a probability value of 70% that the vector feature information belongs to the ID card category, a probability value of 10% that the vector feature information belongs to the marriage certificate category, and a probability value of 10% that the vector feature information belongs to the passport category, then the largest probability value of 70% is selected from the three probability values, and then it is determined whether the probability value of 70% is greater than a preset probability threshold (if the preset probability threshold is 50%). After determining that the probability value of 70% is greater than the preset probability threshold, the ID card corresponding to the largest probability value of 70% is used as the category corresponding to the vector feature information, that is, the category of the target image is the ID card category; if the vector feature information is input into the vector classification model, the vector classification model outputs a probability value of 10% that the vector feature information belongs to the ID card category, a probability value of 15% that the vector feature information belongs to the marriage certificate category, and a probability value of 15% that the vector feature information belongs to the passport category, then the largest probability value of 15% is selected from the three probability values. After determining that the probability value of 15% is less than the preset probability threshold of 50%, the target image belongs to a category not supported by the vector classification model.

[0141] So far, the introduction of the first embodiment of the present application has been completed. The first embodiment of the present application provides a method for identifying the image category. First, the target image is converted into a text block; then a graph structure is generated according to the text block; then the vector feature information of the graph structure is obtained according to the graph structure, and the document image is classified according to the vector feature information. Since the vector feature information is obtained according to the text block, it will not be disturbed by light and shadow, blur, color tone, shooting angle, etc., and does not require a large amount of data for training, which solves the problem of the existing technology that a large amount of data training is required due to the interference when classifying document images according to pixel features. In addition, compared with the method of identifying the category of document images based on keywords in the prior art, the first embodiment of the present application introduces relative position and relative width and height in the composition process according to the text block data. Therefore, in the process of obtaining the vector feature information of the graph structure according to the graph structure, not only the knowledge of keywords can be learned, but also the knowledge of document layout can be learned, and data with the same keywords but different layouts can be distinguished, with a stronger degree of distinction.

[0142] Corresponding to the method for identifying picture categories provided in the first embodiment of the present application, the second embodiment of the present application further provides a device for identifying picture categories.

[0143] like Figure 4 As shown, the image category recognition device includes:

[0144] The target image obtaining unit 401 is used to obtain the target image;

[0145] A text block obtaining unit 402 is used to obtain a text block containing coordinate information and text content according to the target image;

[0146] A graph structure generating unit 403, configured to generate a graph structure according to the text block;

[0147] A vector feature information obtaining unit 404 is used to obtain vector feature information of a graph structure according to the graph structure;

[0148] The target image category obtaining unit 405 is used to input the vector feature information into a vector classification model to obtain the category of the target image; the vector classification model is used to obtain the category of the image corresponding to the vector feature information according to the input vector feature information.

[0149] Optionally, the vector feature information obtaining unit is specifically used to:

[0150] Obtaining text block feature information of the graph structure according to the graph structure;

[0151] Converting text block feature information of the graph structure into vector feature information;

[0152] Optionally, the graph structure generating unit is specifically used to:

[0153] Converting text content in the text block into vector feature information of nodes as feature information of nodes of the graph structure corresponding to the text block;

[0154] The relative position information and relative width and height information between the two text blocks are used as the feature information of the edge between the nodes corresponding to the two text blocks;

[0155] A graph structure is generated according to the feature information of the nodes and the feature information of the edges between the nodes.

[0156] Optionally, the graph structure generating unit is specifically used to:

[0157] According to the coordinate information of the text blocks corresponding to the two nodes, two rectangles corresponding to the text blocks are obtained;

[0158] Normalize the widths and heights of the two rectangles to obtain two normalized rectangles;

[0159] The relative position information between the corresponding vertices of the two rectangles and the normalized width information and height information of the rectangles are used as feature information of the edge between the nodes corresponding to the two text blocks.

[0160] Optionally, the vector feature information obtaining unit is specifically used to:

[0161] The feature information of each node in the graph structure and the feature information of all edges contained in the graph structure are combined and processed as the text block feature information of the graph structure.

[0162] Optionally, the graph structure generating unit is specifically used to:

[0163] A fully connected directed graph is generated according to the feature information of the nodes and the feature information of the edges between the nodes.

[0164] Optionally, the target image category obtaining unit is specifically used for:

[0165] Inputting the vector feature information into a vector classification model, and the vector classification model outputs a probability value that the vector feature information belongs to each category;

[0166] Obtaining the category corresponding to the vector feature information according to the probability value that the vector feature information belongs to each category;

[0167] The category corresponding to the vector feature information is used as the category of the target image.

[0168] Optionally, the category obtaining unit of the target image is specifically used for:

[0169] Selecting the maximum probability value from the probability values ​​that the vector feature information belongs to each category;

[0170] Determining whether the maximum probability value is greater than or equal to a preset probability threshold;

[0171] If so, the picture category corresponding to the maximum probability value is used as the category corresponding to the vector feature information.

[0172] Optionally, the target image is a document image containing text information.

[0173] It should be noted that for the detailed description of the device provided in the second embodiment of the present application, reference can be made to the relevant description of the first embodiment of the present application, which will not be repeated here.

[0174] Corresponding to the method for identifying the picture category provided in the first embodiment of the present application, the third embodiment of the present application further provides an electronic device.

[0175] like Figure 5 As shown, the electronic device comprises:

[0176] Processor 501; and

[0177] The memory 502 is used to store a program of a method for identifying a picture category. After the device is powered on and the program of the method for identifying a picture category is run by the processor, the following steps are performed:

[0178] Get the target image;

[0179] Obtaining a text block containing coordinate information and text content according to the target image;

[0180] generating a graph structure according to the text block;

[0181] Obtaining vector feature information of the graph structure according to the graph structure;

[0182] The vector feature information is input into a vector classification model to obtain the category of the target image; the vector classification model is used to obtain the category of the image corresponding to the vector feature information according to the input vector feature information.

[0183] Optionally, obtaining the vector feature information of the graph structure according to the graph structure includes:

[0184] Obtaining text block feature information of the graph structure according to the graph structure;

[0185] Converting text block feature information of the graph structure into vector feature information;

[0186] Optionally, generating a graph structure according to the text block includes:

[0187] Converting text content in the text block into vector feature information of nodes as feature information of nodes of the graph structure corresponding to the text block;

[0188] The relative position information and relative width and height information between the two text blocks are used as the feature information of the edge between the nodes corresponding to the two text blocks;

[0189] A graph structure is generated according to the feature information of the nodes and the feature information of the edges between the nodes.

[0190] Optionally, the using the relative position information and the relative width and height information between the two text blocks as the feature information of the edge between the nodes corresponding to the two text blocks includes:

[0191] According to the coordinate information of the text blocks corresponding to the two nodes, two rectangles corresponding to the text blocks are obtained;

[0192] Normalize the widths and heights of the two rectangles to obtain two normalized rectangles;

[0193] The relative distance between the corresponding vertices of the two rectangles and the normalized width information and height information of the rectangles are used as feature information of the edge between the nodes corresponding to the two text blocks.

[0194] Optionally, obtaining text block feature information of the graph structure according to the graph structure includes:

[0195] The feature information of each node in the graph structure and the feature information of all edges contained in the graph structure are combined and processed as the text block feature information of the graph structure.

[0196] Optionally, generating a graph structure according to the feature information of the nodes and the feature information of the edges between the nodes includes:

[0197] A fully connected directed graph is generated according to the feature information of the nodes and the feature information of the edges between the nodes.

[0198] Optionally, inputting the vector feature information into a vector classification model to obtain the category of the target image includes:

[0199] Inputting the vector feature information into a vector classification model, and the vector classification model outputs a probability value that the vector feature information belongs to each category;

[0200] Obtaining the category corresponding to the vector feature information according to the probability value that the vector feature information belongs to each category;

[0201] The category corresponding to the vector feature information is used as the category of the target image.

[0202] Optionally, obtaining the category corresponding to the vector feature information according to the probability value that the vector feature information belongs to each category includes:

[0203] Selecting the maximum probability value from the probability values ​​that the vector feature information belongs to each category;

[0204] Determining whether the maximum probability value is greater than or equal to a preset probability threshold;

[0205] If so, the picture category corresponding to the maximum probability value is used as the category corresponding to the vector feature information.

[0206] Optionally, the target image is a document image containing text information.

[0207] It should be noted that for the detailed description of the electronic device provided in the third embodiment of the present application, reference can be made to the relevant description of the first embodiment of the present application, which will not be repeated here.

[0208] Corresponding to the method for identifying the image category provided in the first embodiment of the present application, the fourth embodiment of the present application further provides a storage device storing a program for the method for identifying the image category, the program being executed by a processor to perform the following steps:

[0209] Get the target image;

[0210] Obtaining a text block containing coordinate information and text content according to the target image;

[0211] generating a graph structure according to the text block;

[0212] Obtaining vector feature information of the graph structure according to the graph structure;

[0213] The vector feature information is input into a vector classification model to obtain the category of the target image; the vector classification model is used to obtain the category of the image corresponding to the vector feature information according to the input vector feature information.

[0214] It should be noted that for the detailed description of the storage device provided in the fourth embodiment of the present application, reference can be made to the relevant description of the first embodiment of the present application, which will not be repeated here.

[0215] Corresponding to the method for identifying the image category provided in the first embodiment of the present application, the fifth embodiment of the present application provides an image search method, including:

[0216] Get the image to be searched;

[0217] According to the picture to be searched, generating a graph structure of the picture to be searched;

[0218] Obtaining vector feature information of the graph structure;

[0219] Using a vector classification model, obtaining the category of the image to be searched;

[0220] According to the category of the picture to be searched, information data of the picture to be searched is output.

[0221] It should be noted that the image search method provided in the fifth embodiment of the present application can refer to the relevant description of the first embodiment of the present application, and only a brief description is given here. After obtaining the category of the image to be searched, the information data of the image to be searched can be obtained according to the identification information of the image to be searched.

[0222] Corresponding to the method for identifying the image category provided in the first embodiment of the present application, the sixth embodiment of the present application provides a method for identifying an image identity card, including:

[0223] Obtain a picture ID card to be identified;

[0224] Generating a graph structure of the picture ID card according to the picture ID card;

[0225] Using a vector classification model, obtaining the category of the picture identity card;

[0226] Outputting confirmation information to the user for confirming whether the category of the picture ID card is correct;

[0227] Get confirmation information entered by the user;

[0228] According to the confirmation information input by the user, the parameters of the vector classification model are adjusted.

[0229] It should be noted that, for the image search method provided in the sixth embodiment of the present application, reference can be made to the relevant description of the first embodiment of the present application, and will not be repeated here.

[0230] Corresponding to the method for identifying the image category provided in the first embodiment of the present application, the seventh embodiment of the present application provides a method for processing an image ticket, including:

[0231] Obtain the image ticket to be identified;

[0232] According to the picture ticket, generating a graph structure of the picture ticket;

[0233] Using a vector classification model, obtaining the category of the image ticket;

[0234] If the category of the picture ticket is an invoice, the amount of the picture ticket is counted to obtain the total amount of the invoice.

[0235] It should be noted that, for the image search method provided in the seventh embodiment of the present application, reference can be made to the relevant description of the first embodiment of the present application, and no further details will be given here.

[0236] Corresponding to the method for identifying a picture category provided in the first embodiment of the present application, the eighth embodiment of the present application provides a method for identifying a picture category, including:

[0237] Get the target image;

[0238] Obtaining a text block containing coordinate information and text content according to the target image;

[0239] According to the target image, generating a graph structure of the target image;

[0240] Obtaining vector feature information of the graph structure according to the graph structure;

[0241] Inputting the vector feature information into a vector classification model to obtain a candidate category of the target image; the vector classification model is used to obtain the category of the image corresponding to the vector feature information according to the input vector feature information;

[0242] It is determined whether the candidate category of the target image matches the text block; if so, the candidate category of the target image is determined as the category of the target image.

[0243] It should be noted that, for the image search method provided in the eighth embodiment of the present application, reference can be made to the relevant description of the first embodiment of the present application, and will not be repeated here.

[0244] Corresponding to the method for identifying picture categories provided in the first embodiment of the present application, the ninth embodiment of the present application provides a method for constructing a knowledge graph, including:

[0245] In a training process of a vector classification model for image classification, obtaining training features of the vector classification model;

[0246] According to the training features, obtaining entity information of the image and the relationship between entities;

[0247] Construct an image knowledge graph based on the entity information and the relationships between the entities.

[0248] During the training process of the vector classification model for image classification, the training features of the vector classification model can be obtained. For example, the image can be an ID card, an invoice, vehicle information, etc. Based on these training features, entity information and the relationship between entities can be obtained. Finally, based on the entity information and the relationship between the entities, a picture knowledge graph is constructed. For example, if a picture includes several attributes such as a unified social credit code, vehicle brand, vehicle usage time, vehicle usage status, and vehicle license plate, it can be reflected that the picture corresponds to an entity called "legal person" and an entity called "vehicle".

[0249] The entity "legal person" and "vehicle" can be obtained by experts who build knowledge graphs based on their own knowledge summary through picture content, or can be obtained through machine deduction based on an initial knowledge base; in addition, it can also be known that "legal person" has the attribute of "unified social credit code" and "vehicle" has attributes such as "brand", "usage time", "license plate", and "usage status".

[0250] Next, through the picture including several attributes such as unified social credit code, vehicle license plate, vehicle usage time, etc., we can know that there is an "ownership" relationship between "legal person" and "vehicle".

[0251] Through several attributes included in the picture, such as the unified social credit code, vehicle license plate, and vehicle usage time, the two entities of "legal person" and "vehicle" are obtained; the "legal person" has the attribute of "unified social credit code", and because this attribute has a one-to-one relationship with the legal entity, this attribute can be used as the primary key information of the "legal person" entity; similarly, the "vehicle" has attributes such as license plate, usage time, and usage status. The use of vehicle license plates can correspond one-to-one with a specific vehicle, so it can be used as the primary key information of the "vehicle" entity.

[0252] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0253] In a typical configuration, a computing device includes one or more processors (CPU), memory-mapped input / output interfaces, network interfaces, and memory.

[0254] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0255] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0256] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

Claims

1. A method for identifying a picture category, characterized in that: include: Get the target image; Obtaining a text block containing coordinate information and text content according to the target image, wherein the coordinate information is coordinate information of vertices of the text block in the target image; Convert the text content in the text block into vector feature information of nodes as feature information of nodes of the graph structure corresponding to the text block; use the relative position information and relative width and height information between two text blocks as feature information of edges between the nodes corresponding to the two text blocks; generate the graph structure of the target image according to the feature information of the nodes and the feature information of the edges between the nodes; Obtaining vector feature information of the graph structure according to the graph structure, wherein the vector feature information of the graph structure is obtained by converting text block feature information of the graph structure; The vector feature information is input into a vector classification model to obtain the category of the target image; the vector classification model is used to obtain the category of the image corresponding to the vector feature information according to the input vector feature information.

2. The method according to claim 1, characterized in that The method of using the relative position information and the relative width and height information between the two text blocks as the feature information of the edge between the nodes corresponding to the two text blocks includes: According to the coordinate information of the text blocks corresponding to the two nodes, two rectangles corresponding to the text blocks are obtained; Normalize the widths and heights of the two rectangles to obtain two normalized rectangles; The relative position information between the corresponding vertices of the two rectangles and the normalized width information and height information of the rectangles are used as feature information of the edge between the nodes corresponding to the two text blocks.

3. The method according to claim 1 or 2, characterized in that: The step of obtaining text block feature information of the graph structure according to the graph structure includes: The feature information of each node in the graph structure and the feature information of all edges contained in the graph structure are combined and processed as the text block feature information of the graph structure.

4. The method according to claim 1, characterized in that: The generating a graph structure according to the feature information of the nodes and the feature information of the edges between the nodes includes: A fully connected directed graph is generated according to the feature information of the nodes and the feature information of the edges between the nodes.

5. The method according to claim 1, characterized in that The step of inputting the vector feature information into a vector classification model to obtain the category of the target image includes: Inputting the vector feature information into a vector classification model, and the vector classification model outputs a probability value that the vector feature information belongs to each category; Obtaining the category corresponding to the vector feature information according to the probability value that the vector feature information belongs to each category; The category corresponding to the vector feature information is used as the category of the target image.

6. The method according to claim 5, characterized in that The obtaining, according to the probability value of the vector feature information belonging to each category, the category corresponding to the vector feature information includes: Selecting the maximum probability value from the probability values ​​that the vector feature information belongs to each category; Determining whether the maximum probability value is greater than or equal to a preset probability threshold; If so, the picture category corresponding to the maximum probability value is used as the category corresponding to the vector feature information.

7. The method according to claim 1, characterized in that The target image is a document image containing text information.

8. A device for identifying image categories, characterized in that: include: A target image obtaining unit, used for obtaining a target image; A text block obtaining unit, used to obtain a text block containing coordinate information and text content according to the target image, wherein the coordinate information is the coordinate information of the vertices of the text block in the target image; A graph structure generating unit is used to convert the text content in the text block into vector feature information of nodes as feature information of nodes of the graph structure corresponding to the text block; use the relative position information and relative width and height information between two text blocks as feature information of edges between the nodes corresponding to the two text blocks; and generate the graph structure of the target image according to the feature information of the nodes and the feature information of the edges between the nodes; A vector feature information obtaining unit, used to obtain vector feature information of a graph structure according to the graph structure, wherein the vector feature information of the graph structure is obtained by converting text block feature information of the graph structure; The target image category obtaining unit is used to input the vector feature information into a vector classification model to obtain the category of the target image; the vector classification model is used to obtain the category of the image corresponding to the vector feature information based on the input vector feature information.

9. An electronic device, characterized in that: include: processor; as well as The memory is used to store a program of a method for identifying a picture category. After the device is powered on and the program of the method for identifying a picture category is run by the processor, the following steps are performed: Get the target image; Obtaining a text block containing coordinate information and text content according to the target image, wherein the coordinate information is coordinate information of vertices of the text block in the target image; Convert the text content in the text block into vector feature information of nodes as feature information of nodes of the graph structure corresponding to the text block; use the relative position information and relative width and height information between two text blocks as feature information of edges between the nodes corresponding to the two text blocks; generate the graph structure of the target image according to the feature information of the nodes and the feature information of the edges between the nodes; Obtaining vector feature information of the graph structure according to the graph structure, wherein the vector feature information of the graph structure is obtained by converting text block feature information of the graph structure; The vector feature information is input into a vector classification model to obtain the category of the target image; the vector classification model is used to obtain the category of the image corresponding to the vector feature information according to the input vector feature information.

10. A storage device, characterized in that: A program storing a method for identifying a picture category is executed by a processor to perform the following steps: Get the target image; Obtaining a text block containing coordinate information and text content according to the target image, wherein the coordinate information is coordinate information of vertices of the text block in the target image; Convert the text content in the text block into vector feature information of nodes as feature information of nodes of the graph structure corresponding to the text block; use the relative position information and relative width and height information between two text blocks as feature information of edges between the nodes corresponding to the two text blocks; generate the graph structure of the target image according to the feature information of the nodes and the feature information of the edges between the nodes; Obtaining vector feature information of the graph structure according to the graph structure, wherein the vector feature information of the graph structure is obtained by converting text block feature information of the graph structure; The vector feature information is input into a vector classification model to obtain the category of the target image; the vector classification model is used to obtain the category of the image corresponding to the vector feature information according to the input vector feature information.

11. A method for searching an image, characterized in that: include: Get the image to be searched; According to the image to be searched, a text block containing coordinate information and text content is obtained, wherein the coordinate information is the coordinate information of the vertices of the text block in the image to be searched; Converting the text content in the text block into vector feature information of nodes as feature information of nodes of the graph structure corresponding to the text block; using relative position information and relative width and height information between two text blocks as feature information of edges between nodes corresponding to the two text blocks; generating the graph structure of the image to be searched according to the feature information of the nodes and the feature information of the edges between the nodes; Obtaining vector feature information of the graph structure, wherein the vector feature information of the graph structure is converted based on text block feature information of the graph structure; Using a vector classification model, obtaining the category of the image to be searched; According to the category of the picture to be searched, information data of the picture to be searched is output.

12. A method for identifying a picture identity card, characterized in that: include: Obtain a picture ID card to be identified; According to the picture ID card, a text block containing coordinate information and text content is obtained, wherein the coordinate information is the coordinate information of the vertices of the text block in the picture ID card; Convert the text content in the text block into vector feature information of nodes as feature information of nodes in the graph structure corresponding to the text block; use the relative position information and relative width and height information between two text blocks as feature information of edges between the nodes corresponding to the two text blocks; generate the graph structure of the picture ID card according to the feature information of the nodes and the feature information of the edges between the nodes; Using a vector classification model, obtaining the category of the picture identity card; Outputting confirmation information to the user for confirming whether the category of the picture ID card is correct; Get confirmation information entered by the user; According to the confirmation information input by the user, the parameters of the vector classification model are adjusted.

13. A method for processing image receipts, characterized in that: include: Obtain the image ticket to be identified; According to the picture ticket, a text block containing coordinate information and text content is obtained, wherein the coordinate information is the coordinate information of the vertices of the text block in the picture ticket; Convert the text content in the text block into vector feature information of nodes as feature information of nodes in the graph structure corresponding to the text block; use the relative position information and relative width and height information between two text blocks as feature information of edges between the nodes corresponding to the two text blocks; generate the graph structure of the picture ticket according to the feature information of the nodes and the feature information of the edges between the nodes; Using a vector classification model, obtaining the category of the image ticket; If the category of the picture ticket is an invoice, the amount of the picture ticket is counted to obtain the total amount of the invoice.

14. A method for identifying a picture category, characterized in that: include: Get the target image; Obtaining a text block containing coordinate information and text content according to the target image, wherein the coordinate information is coordinate information of vertices of the text block in the target image; Convert the text content in the text block into vector feature information of nodes as feature information of nodes of the graph structure corresponding to the text block; use the relative position information and relative width and height information between two text blocks as feature information of edges between the nodes corresponding to the two text blocks; generate the graph structure of the target image according to the feature information of the nodes and the feature information of the edges between the nodes; Obtaining vector feature information of the graph structure according to the graph structure, wherein the vector feature information of the graph structure is obtained by converting text block feature information of the graph structure; Inputting the vector feature information into a vector classification model to obtain a candidate category of the target image; the vector classification model is used to obtain the category of the image corresponding to the vector feature information according to the input vector feature information; It is determined whether the candidate category of the target image matches the text block; if so, the candidate category of the target image is determined as the category of the target image.

Images