Image recognition method and device, model training method and device

Abstract

The present specification provides an image recognition method and device, a model training method and device, wherein the image recognition method comprises: extracting a plurality of pictures to be recognized from a set to be recognized, and constructing a corresponding graph network for each picture; inputting the graph network corresponding to the plurality of pictures to be recognized into a feature extraction model to obtain a feature vector of the picture, wherein the feature extraction model is obtained by training a metric learning loss function; and clustering the feature vectors of the plurality of pictures to recognize the picture type. The image recognition method provided in the present specification extracts the feature vector of the picture to be recognized by inputting the graph network of the picture to be recognized into the feature extraction model obtained by training the metric learning loss function, so that the clustering is more accurate, and the picture type is recognized more quickly and accurately.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to image recognition methods and apparatus, and model training methods and apparatus. Background Technology

[0002] With the development of optical character recognition technology, this technology has been gradually applied to the recognition of image types, such as card and certificate classification.

[0003] In reality, the types of images are endless. Although various applications of optical character recognition generate a large amount of data, this data is currently idle due to a lack of ability to process unknown data. Furthermore, existing methods for discovering new image types mainly rely on manual methods or rules. These methods, such as keyword searches, are needed to find new image types that conform to the definition from a large image database. This approach is time-consuming, labor-intensive, and has poor robustness, facing the problem of a large number of noisy images and only a small portion of effective images.

[0004] This necessitates faster and more accurate operations or processing to identify image types. Summary of the Invention

[0005] In view of this, embodiments of this specification provide an image recognition method and apparatus. This specification also relates to a model training method and apparatus, a computing device, and a computer-readable storage medium, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this specification, an image recognition method is provided, comprising:

[0007] Extract multiple images to be identified from the set of images to be identified, and construct a corresponding graph network for each image;

[0008] The graph network corresponding to multiple images to be identified is input into the feature extraction model to obtain the feature vector of the image, wherein the feature extraction model is trained by a metric learning loss function;

[0009] Cluster the feature vectors of the multiple images to identify the image type.

[0010] Optionally, before extracting multiple images to be identified from the set to be identified, the process also includes:

[0011] Receive the image to be identified and add it to the set to be identified.

[0012] Optionally, the feature vectors of the plurality of images are clustered to identify the image type, including:

[0013] The feature vectors of the multiple images are clustered using a density-based clustering algorithm, and the images corresponding to the vectors that are merged into clusters are identified as the same image type.

[0014] Optionally, the feature extraction model is trained through the following steps:

[0015] Construct multiple triplet training samples consisting of an initial sample image, a positive sample image, and a negative sample image corresponding to the initial sample image, the positive sample image, and the negative sample image;

[0016] The multiple triplet training samples are input into the feature extraction model to obtain the feature vectors of the initial sample image, positive sample image, and negative sample image in each triplet training sample.

[0017] Calculate the first distance between the feature vectors of the initial sample image and the negative sample image in each triplet training sample, and the second distance between the feature vectors of the initial sample image and the positive sample image. Input the first distance and the second distance into the metric learning loss function, and train the feature extraction model according to the output of the loss function until the loss function tends to stabilize.

[0018] Optionally, constructing a corresponding graph network for each image includes:

[0019] Perform optical character recognition on each image to obtain text blocks;

[0020] The word embeddings of the text blocks are set as nodes of the graph network, and the relative positional relationships between pairs of text blocks are set as edges of the graph network.

[0021] Optionally, the graph network corresponding to multiple images to be identified is input into the feature extraction model to obtain the feature vector of the image, including:

[0022] Graph convolution encoding is performed on the nodes and edges of the graph network to obtain the first dimension vector of the nodes and edges;

[0023] The first dimension vector of the graph network is averaged and pooled to obtain the feature vector of the image.

[0024] Optionally, the image recognition method further includes:

[0025] The system receives images of the identified types and performs type merging and / or filtering, and labels the images with their types based on the results of the merging and / or filtering.

[0026] Optionally, the image recognition method further includes:

[0027] Images whose types are not identified are stored in the set to be identified.

[0028] According to a second aspect of the embodiments of this specification, an image recognition method is provided, comprising:

[0029] Based on the user's request, display the image input interface to the user;

[0030] Receive multiple images to be recognized from the user based on the image input interface, and construct a corresponding graph network for each image;

[0031] The graph network corresponding to multiple images to be identified is input into the feature extraction model to obtain the feature vector of the image, wherein the feature extraction model is trained by a metric learning loss function;

[0032] The feature vectors of the multiple images are clustered to identify the image type and the result is returned to the user.

[0033] According to a third aspect of the embodiments of this specification, an image recognition method is provided, comprising:

[0034] Receive a call request sent by a user, wherein the call request carries multiple images to be identified;

[0035] Build a corresponding graph network for each image;

[0036] The graph network corresponding to multiple images to be identified is input into the feature extraction model to obtain the feature vector of the image, wherein the feature extraction model is trained by a metric learning loss function;

[0037] The feature vectors of the multiple images are clustered to identify the image type and the result is returned to the user.

[0038] According to a fourth aspect of the embodiments of this specification, a model training method is provided, comprising:

[0039] Construct multiple triplet training samples consisting of an initial sample image, a positive sample image, and a negative sample image corresponding to the initial sample image, the positive sample image, and the negative sample image;

[0040] The multiple triplet training samples are input into the feature extraction model to obtain the feature vectors of the initial sample image, positive sample image, and negative sample image in each triplet training sample.

[0041] Calculate the first distance between the feature vectors of the initial sample image and the negative sample image in each triplet training sample, and the second distance between the feature vectors of the initial sample image and the positive sample image. Input the first distance and the second distance into the metric learning loss function, and train the feature extraction model according to the output of the loss function until the loss function tends to stabilize.

[0042] According to a fifth aspect of the embodiments of this specification, an image recognition device is provided, comprising:

[0043] The first building module is configured to extract multiple images to be identified from the set of images to be identified, and build a corresponding graph network for each image;

[0044] The first acquisition module is configured to input the graph network corresponding to multiple images to be identified into the feature extraction model to obtain the feature vector of the image, wherein the feature extraction model is trained by a metric learning loss function;

[0045] The first clustering module is configured to cluster the feature vectors of the multiple images to identify the image type.

[0046] Optionally, the image recognition device further includes:

[0047] The add module is configured to receive images to be recognized and add them to the set of images to be recognized.

[0048] Optionally, the first clustering module is further configured to:

[0049] The feature vectors of the multiple images are clustered using a density-based clustering algorithm, and the images corresponding to the vectors that are merged into clusters are identified as the same image type.

[0050] Optionally, the feature extraction model is trained through the following steps:

[0051] Construct multiple triplet training samples consisting of an initial sample image, a positive sample image, and a negative sample image corresponding to the initial sample image, the positive sample image, and the negative sample image;

[0052] The multiple triplet training samples are input into the feature extraction model to obtain the feature vectors of the initial sample image, positive sample image, and negative sample image in each triplet training sample.

[0053] Calculate the first distance between the feature vectors of the initial sample image and the negative sample image in each triplet training sample, and the second distance between the feature vectors of the initial sample image and the positive sample image. Input the first distance and the second distance into the metric learning loss function, and train the feature extraction model according to the output of the loss function until the loss function tends to stabilize.

[0054] Optionally, the first building module is further configured to:

[0055] Perform optical character recognition on each image to obtain text blocks;

[0056] The word embeddings of the text blocks are set as nodes of the graph network, and the relative positional relationships between pairs of text blocks are set as edges of the graph network.

[0057] Optionally, the first obtaining module is further configured to:

[0058] Graph convolution encoding is performed on the nodes and edges of the graph network to obtain the first dimension vector of the nodes and edges;

[0059] The first dimension vector of the graph network is averaged and pooled to obtain the feature vector of the image.

[0060] Optionally, the image recognition device further includes:

[0061] The type labeling module is configured to receive images of identified types and / or perform type merging and / or filtering, and to label the image types according to the results of the merging and / or filtering.

[0062] Optionally, the image recognition device further includes:

[0063] The storage module is configured to store images of unidentified types into the set to be identified.

[0064] According to a sixth aspect of the embodiments of this specification, an image recognition device is provided, comprising:

[0065] The display module is configured to show the user an image input interface based on the user's request.

[0066] The second construction module is configured to receive multiple images to be identified by the user based on the image input interface, and construct a corresponding graph network for each image;

[0067] The second acquisition module is configured to input the graph network corresponding to multiple images to be identified into the feature extraction model to obtain the feature vector of the image, wherein the feature extraction model is trained by a metric learning loss function;

[0068] The second clustering module is configured to cluster the feature vectors of the multiple images, identify the image type, and return the results to the user.

[0069] According to a seventh aspect of the embodiments of this specification, an image recognition apparatus is provided, comprising:

[0070] The receiving module is configured to receive a call request sent by a user, wherein the call request carries multiple images to be identified;

[0071] The third building module is configured to build a corresponding graph network for each image;

[0072] The third acquisition module is configured to input the graph network corresponding to multiple images to be identified into the feature extraction model to obtain the feature vector of the image, wherein the feature extraction model is trained by a metric learning loss function;

[0073] The third clustering module is configured to cluster the feature vectors of the multiple images, identify the image type, and return the results to the user.

[0074] According to an eighth aspect of the embodiments of this specification, a model training apparatus is provided, comprising:

[0075] The fourth construction module is configured to construct multiple triplet training samples consisting of the initial sample image, the positive sample image, the negative sample image, and the corresponding initial sample graph network, positive sample graph network, and negative sample graph network.

[0076] The fourth acquisition module is configured to input the multiple triplet training samples into the feature extraction model to obtain the feature vectors of the initial sample image, positive sample image, and negative sample image in each triplet training sample.

[0077] The computation module is configured to calculate a first distance between the feature vectors of the initial sample image and the negative sample image in each triplet training sample, and a second distance between the feature vectors of the initial sample image and the positive sample image. The first distance and the second distance are input into a metric learning loss function, and the feature extraction model is trained based on the output of the loss function until the loss function tends to stabilize.

[0078] According to a ninth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0079] Memory and processor;

[0080] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, wherein the processor executes the computer-executable instructions to implement the steps of the image recognition method or the steps of the model training method.

[0081] According to a tenth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the image recognition method or the steps of the model training method.

[0082] The image recognition method provided in this specification extracts multiple images to be recognized from a set of images to be recognized, constructs a corresponding graph network for each image, and then inputs the graph networks corresponding to the multiple images to be recognized into a feature extraction model pre-trained using a metric learning loss function to obtain feature vectors for the images. The feature vectors of the multiple images are then clustered to identify the image type. Because the image recognition method provided in this specification inputs the graph networks of the images to be recognized into the feature extraction model trained using a metric learning loss function to extract feature vectors from the images to be recognized, the clustering is more accurate, thus enabling faster and more accurate identification of image types. Attached Figure Description

[0083] Figure 1 This specification illustrates an example diagram of a specific application scenario of an image recognition method provided in one embodiment.

[0084] Figure 2 A flowchart of a first image recognition method provided in an embodiment of this specification is shown;

[0085] Figure 3 A flowchart of a model training method provided in one embodiment of this specification is shown;

[0086] Figure 4 An example processing flowchart of a first image recognition method provided in an embodiment of this specification is shown;

[0087] Figure 5 A flowchart illustrating an image recognition method for recognizing new card types, provided in one embodiment of this specification, is shown.

[0088] Figure 6 A flowchart of a second image recognition method provided in one embodiment of this specification is shown;

[0089] Figure 7 A flowchart of a third image recognition method provided in one embodiment of this specification is shown;

[0090] Figure 8 A schematic diagram of the structure of a first image recognition device provided in an embodiment of this specification is shown;

[0091] Figure 9 A schematic diagram of the structure of a second image recognition device provided in one embodiment of this specification is shown;

[0092] Figure 10 A schematic diagram of the structure of a third image recognition device provided in one embodiment of this specification is shown;

[0093] Figure 11 This specification shows a schematic diagram of the structure of a model training device according to an embodiment of the present specification;

[0094] Figure 12 A structural block diagram of a computing device provided in one embodiment of this specification is shown. Detailed Implementation

[0095] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0096] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0097] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0098] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0099] Optical Character Recognition (OCR) refers to the process of translating printed characters into computer text using character recognition methods.

[0100] OCR Text Block: A block-shaped text region (containing single or multiple characters) obtained by using text detection and recognition technology and line aggregation.

[0101] Word embedding: A word embedding is a real-valued vector representing each word in a predetermined vector space.

[0102] Metric learning: The object of metric learning is usually the distance between sample feature vectors. The purpose of metric learning is to reduce or limit the distance between samples of the same class, while increasing the distance between samples of different classes, through training and learning.

[0103] This specification provides an image recognition method, and also relates to an image recognition device, a model training method and device, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0104] Figure 1 This is an example diagram illustrating a specific application scenario of an image recognition method provided in one embodiment of this specification.

[0105] See Figure 1 The application scenarios include: terminal 102 and server 104.

[0106] Terminals can be, for example, smartphones, portable terminals, mobile terminals, personal digital assistants (PDAs), portable multimedia player (PMP) terminals, telematics terminals, navigation terminals, personal computers (PCs), laptops, tablet PCs, ultrabooks, wearable devices (e.g., watch terminals (smartwatches), glasses terminals (smart glasses), or head-mounted displays (HMDs)), Internet Protocol Television (IPTV) terminals, smart TVs, digital broadcasting terminals, televisions, 3D televisions, home theater systems, audio-visual navigation (AVN) terminals, audio / video (A / V) systems, or flexible terminals, but this disclosure is not limited thereto and may include any device on which applications can be installed.

[0107] Specifically, such as Figure 1 As shown, terminal 102 sends multiple images to be identified to server 104. The images to be identified include images with text content of known types (card / certificate images and non-card / certificate images) and images with text content of unknown types. Known card / certificate images include various ID card images; known non-card / certificate images include business card images, bookmark images, and product tag images; unknown images with text content are those whose type cannot be identified based on the semantics and structure of their text content. Server 104 receives the images to be identified, constructs a corresponding graph network for each received image, and inputs the graph networks corresponding to the multiple images to be identified into a feature extraction model trained using a metric learning loss function to obtain the image feature vectors. The feature vectors of images satisfy the following condition: the Euclidean distance between feature vectors of images of the same type is small, and the Euclidean distance between feature vectors of images of different types is large. Then, the feature vectors of multiple images are clustered to identify the image type. Figure 1As shown, after clustering, cluster A is formed by the feature vectors of ID card images, cluster B is formed by the feature vectors of product tag images, and clusters C and D are formed by the feature vectors of images with text content of unknown image types. Each cluster defines an image type. Clusters other than those formed by known image types are considered newly identified image types. The final identified image types are a, b, c, and d.

[0108] Figure 2 A flowchart of a first image recognition method provided in an embodiment of this specification is shown, specifically including steps 202 to 206.

[0109] Step 202: Extract multiple images to be identified from the set of images to be identified, and construct a corresponding graph network for each image.

[0110] The set to be identified is a collection of images to be identified. This set is obtained by receiving images to be identified and adding them to the collection. The images to be identified include images with text content of known image types (card / certificate images and non-card / certificate images), and images with text content of unknown image types. Known image types include card / certificate images such as various ID card images, and known image types include non-card / certificate images such as business cards, bookmarks, and product tags. Images of known image types can be identified based on the semantics and structure of their text content; for example, common ID cards can be identified. Images of unknown image types with text content cannot be identified based on the semantics and structure of their text content. The set to be identified can include both known and unknown image types, or it can include only images of known or unknown image types.

[0111] Building a corresponding graph network for each image can be achieved as follows:

[0112] Perform optical character recognition on each image to obtain text blocks;

[0113] The word embeddings of the text blocks are set as nodes of the graph network, and the relative positional relationships between pairs of text blocks are set as edges of the graph network.

[0114] A text block is a block-shaped distributed text region obtained through text detection and recognition technology and line aggregation. Text blocks can be divided according to the spacing between characters. Each character content, such as Chinese characters, letters, or words, in the distributed text region corresponds to a word vector. The word embedding of the text block can be obtained based on the word vectors of all text content in the distributed text region, and the obtained word embeddings are used as nodes in a graph network. In one embodiment, the word embedding of the text block is obtained by averaging the word vectors of all text content. The word vectors of the text content can be set manually or learned and updated through a neural network. The word vector dimension is preferably set to 100 or higher, for example, a dimension of 100 to 300.

[0115] The edges of the graph network represent the relative positional relationships between pairs of text blocks. If the coordinates of two text blocks are (x1, y1, w1, h1) and (x2, y2, w2, h2), where x1 and y1 are the x and y coordinates of the top-left corner of the first text block, w1 and h1 are the width and height of the first text block, x2 and y2 are the x and y coordinates of the top-left corner of the second text block, and w2 and h2 are the width and height of the second text block, then the relative positional relationship between the first and second text blocks can be expressed as w1 / h1, (x2-x1) / h1, (y2-y1) / h1, w2 / h1.

[0116] Step 204: Input the graph network corresponding to the multiple images to be identified into the feature extraction model to obtain the feature vector of the image, wherein the feature extraction model is trained by a metric learning loss function.

[0117] Feature extraction models are any neural network models, such as convolutional neural networks, that can perform feature extraction. The metric learning loss function can be a contrastive (binary) loss function, a triplet loss function, a quadruplet loss function, or similar loss functions. The feature vectors of an image are multi-dimensional vectors.

[0118] In one embodiment, the graph network corresponding to multiple images to be identified is input into a feature extraction model to obtain the feature vector of the images, which can be achieved as follows:

[0119] Graph convolution encoding is performed on the nodes and edges of the graph network to obtain the first dimension vector of the nodes and edges;

[0120] The first dimension vector of the graph network is averaged and pooled to obtain the feature vector of the image.

[0121] Specifically, after performing graph convolution operations on nodes and edges, high-dimensional vectors are obtained. Each node corresponds to a high-dimensional vector. The high-dimensional vectors of multiple nodes are averaged and pooled to obtain the feature vector of the image.

[0122] Images containing text can be described using three feature dimensions: image, semantic, and structural. Image features of the same type of image are complex and variable, influenced by background, lighting, and other factors, making them unstable and lacking good generalization performance. A feature extraction model trained by combining semantic and structural features with a metric learning loss function minimizes the distance between images of the same type in the feature space, while maximizing the distance between images of different types, and exhibits good generalization ability.

[0123] The following text combines Figure 3 The training process of the feature extraction model is illustrated using the triplet loss function as an example. Figure 3 As shown, the training process includes steps 302 to 306.

[0124] Step 302: Construct multiple triplet training samples consisting of the initial sample image, the positive sample image, the negative sample image, and the corresponding initial sample image network, positive sample image network, and negative sample image network.

[0125] Step 304: Input the multiple triplet training samples into the feature extraction model to obtain the feature vectors of the initial sample image, positive sample image, and negative sample image in each triplet training sample.

[0126] Step 306: Calculate the first distance between the feature vectors of the initial sample image and the negative sample image in each triplet training sample, and the second distance between the feature vectors of the initial sample image and the positive sample image. Input the first distance and the second distance into the metric learning loss function, and train the feature extraction model according to the output of the loss function until the loss function tends to stabilize.

[0127] The initial sample images are randomly selected images containing text. Positive sample images are of the same image type as the initial sample images, while negative sample images are of a different image type. Assuming 1000 initial sample images are selected, 1000... 3 There are three triples. All of these triples can be used for model training, or some triples can be randomly selected for model training.

[0128] In one embodiment, image cards are used to construct a triplet training model. For example, one image of an ID card is selected as the initial sample image, another image of the same type is selected as the positive sample image, and images of different types of ID cards are selected as negative sample images. Optical character recognition is performed on each image to obtain text blocks. The word embeddings of the text blocks are set as nodes in the graph network, and the relative positional relationships between pairs of text blocks are set as edges in the graph network, thereby obtaining the initial sample graph network, the positive sample graph network, and the negative sample graph network. The initial sample graph network, the positive sample graph network, and the negative sample graph network are input into the feature extraction model to obtain feature vectors of the three ID card images. The distance between the feature vectors of the ID card image used as the initial sample image and the ID card image used as the positive sample image, and the distance between the feature vectors of the ID card image used as the initial sample image and the ID card image used as the negative sample image are calculated. The two distances are input into the metric learning loss function, and the feature extraction model is trained based on the output of the loss function until the loss function tends to stabilize.

[0129] Specifically, the triplet loss function can be:

[0130]

[0131] Where i is the triplet number, N is the number of triplets, and x a It is the feature vector of the initial sample image, x p It is the feature vector of the positive sample image, x n It is the feature vector of the negative sample image. It is the Euclidean distance between the feature vectors of the initial sample image and the positive sample image. α is the Euclidean distance between the feature vectors of the initial sample image and the negative sample image, and α is the minimum margin between the Euclidean distance between the feature vectors of the initial sample image and the negative sample image and the Euclidean distance between the feature vectors of the initial sample image and the positive sample image. The specific parameter value of α can be determined according to the model performance.

[0132] After iteratively calculating and updating the parameters of the feature extraction model based on the loss function, the value of the loss function decreases from an initial large value until it stabilizes. Stabilization means the loss function no longer decreases, reaching convergence (e.g., approaching zero), thus completing the training of the feature extraction model and obtaining the trained model. Specifically, the parameters of the feature extraction model are optimized using gradient descent. In each iteration, the data from the forward propagation of samples to the output layer nodes is calculated, the gradient is obtained, and backpropagation is performed, updating the feature extraction model parameters during the backpropagation process. After multiple iterations, the objective function converges, resulting in stable model parameters, indicating that the training process of the feature extraction model has converged.

[0133] After training the feature extraction model using the triplet loss function, the feature vectors output by the feature extraction model can achieve small Euclidean distance between feature vectors of the same type of image and large Euclidean distance between feature vectors of different types of images.

[0134] Step 206: Cluster the feature vectors of the multiple images to identify the image types.

[0135] Density-based clustering algorithms, K-means, hierarchical clustering, and other methods can be used to cluster the feature vectors of images to identify image types. When the set to be identified includes images of known and unknown image types, the identified image types are various known card / certificate images, various known non-card / certificate images, and one or more unknown image types. When the set to be identified only includes images of known image types, the identified image types are various known image types such as ID cards, business cards, bookmarks, and tags. When the set to be identified only includes images of unknown image types, the identified image types are one or more new image types.

[0136] The image recognition method provided in this manual extracts feature vectors from the image by inputting the graph network of the image to be recognized into a feature extraction model trained through a metric learning loss function. This makes clustering more accurate, thus enabling faster and more accurate identification of image types.

[0137] In one embodiment, step 206 is implemented as follows:

[0138] The feature vectors of the multiple images are clustered using a density-based clustering algorithm, and the images corresponding to the vectors that are merged into clusters are identified as the same image type.

[0139] The following explanation uses the density-based clustering of applications with noise (DBSCAN) algorithm as an example.

[0140] After obtaining feature vectors from multiple images, the DBSCAN clustering algorithm is used to cluster these feature vectors. The DBSCAN clustering algorithm mainly consists of two steps: First, finding core points to form temporary clusters. Core points must satisfy the condition that the number of points within a radius R is greater than or equal to the minimum number of points (MinPoints). Points whose density reaches the core point form temporary clusters. The specific values ​​of the radius R and the minimum number of points can be adjusted based on the clustering effect to select the most suitable parameters. Second, for each temporary cluster, checking if any of its points are core points. If so, the temporary clusters are merged until all points in the current temporary cluster have been merged. Then, the cluster is upgraded to a single cluster. This involves checking if there are other core points within the temporary clusters. If so, their density is achievable. The point densities of these two temporary clusters are connected, and the feature vectors with the highest density connection are found; these are the single clusters. The remaining unmerged points are considered images whose type could not be identified. The DBSCAN clustering algorithm does not require specifying the number of clusters and is robust to noise. By combining semantic and structural features with a feature extraction model trained using a metric learning loss function, images of the same type are less distant in the feature space, while images of different types are more distant. Combined with a density-based clustering algorithm, this enables fast and accurate identification of image types.

[0141] In one embodiment of this specification, the image recognition method further includes:

[0142] The system receives images of the identified types and performs type merging and / or filtering, and labels the images with their types based on the results of the merging and / or filtering.

[0143] After identifying the image type, there may be cases where the identified image type is too granular or misclassified. Category merging and filtering operations can be performed to obtain the required defined type. This operation can be performed manually. After the image recognition method identifies the image and then human intervention is performed, the workload of manual labor will be greatly reduced and the time spent on identifying the image type will be reduced.

[0144] Figure 4 An example processing flowchart of a first image recognition method provided in an embodiment of this specification is shown, such as... Figure 4 As shown, the image recognition method specifically includes steps 402 to 412.

[0145] Step 402: Receive multiple images to be identified and add them to the identification set.

[0146] The images in the set to be identified include images of known image types with text content (card / certificate images and non-card / certificate images), and may also include images of unknown image types with text content. Known image types include card / certificate images such as various ID card images, and known image types include non-card / certificate images such as business card images, bookmark images, and product tag images. Images of known image types can be identified based on the semantics and structure of their text content; for example, common ID card images can be identified. Images of unknown image types with text content cannot be identified based on the semantics and structure of their text content. This embodiment uses an example where the set to be identified includes one type of ID card image, a product tag image, and images of unknown image types with text content.

[0147] Step 404: Extract multiple images to be identified from the set of images to be identified, and construct a corresponding graph network for each image.

[0148] Extract images with text content from the set to be identified, including one type of ID card image, one type of product tag image, and one type of unknown image. Obtain text blocks by performing optical character recognition on each image. Set the word embeddings of the text blocks as nodes of the graph network, and set the relative positional relationships between pairs of text blocks as edges of the graph network, thus constructing a graph network for each image.

[0149] Step 406: Input the graph network corresponding to the multiple images to be identified into the feature extraction model to obtain the feature vector of the image, wherein the feature extraction model is trained by a metric learning loss function.

[0150] For each image, graph convolution encoding is performed on the nodes and edges of the graph network to obtain high-dimensional vectors of the nodes and edges; average pooling is then performed on the high-dimensional vectors of the graph network to obtain the feature vector of each image.

[0151] Step 408: Use a density-based clustering algorithm to cluster the feature vectors of multiple images to identify the image type.

[0152] Specifically, after clustering, we obtain clusters formed by feature vectors of document images, clusters formed by feature vectors of product tag images, and one or more clusters formed by feature vectors of images with text content of unknown image types. Each cluster defines an image type, and clusters other than those formed by known image types are considered newly identified image types.

[0153] Step 410: Receive the type merging and / or filtering of the identified image types, and label the images according to the merging and / or filtering results.

[0154] The image types obtained after clustering can be merged or filtered manually. Type labeling can be performed after merging and / or filtering. New image types can be defined when merging and / or filtering new image types.

[0155] Step 412: Store images whose image types are not identified into the unidentified image set.

[0156] Images of unknown type that have not been identified will be returned to the set to be identified.

[0157] The image recognition method according to this embodiment can quickly and accurately identify image types, accumulate data for subsequent classification and structuring tasks, and greatly reduce annotation costs.

[0158] The following is in conjunction with the appendix Figure 5 Taking the image recognition method provided in this specification as an example of its application in recognizing new card and document types, the image recognition method will be further explained. Among other things, Figure 5 The present specification shows a processing flowchart of an image recognition method for recognizing new card types, according to an embodiment of the present specification, specifically including steps 502 to 512.

[0159] Step 502: Receive the unknown type of card image to be identified and add it to the card image set of unknown type.

[0160] Add the card images whose types cannot be recognized to the card image collection.

[0161] Step 504: Extract multiple card images to be identified from the set of card images of unknown type, and construct a corresponding graph network for each card image.

[0162] The graph network is constructed as follows: text blocks are obtained by performing optical character recognition on each card image; the word embeddings of the text blocks are set as nodes of the graph network, and the relative positional relationships between pairs of text blocks are set as edges of the graph network.

[0163] Step 506: Input the graph network corresponding to the multiple card images to be identified into the feature extraction model to obtain the feature vector of each card image, wherein the feature extraction model is trained by a metric learning loss function.

[0164] In one embodiment, the feature vector of the card image is obtained by performing graph convolution encoding on the nodes and edges of the graph network to obtain high-dimensional vectors of the nodes and edges; and then performing average pooling on the high-dimensional vectors of the graph network to obtain the feature vector of the card image.

[0165] Step 508: Use a density-based clustering algorithm to cluster the feature vectors of multiple card images to identify new card types.

[0166] Specifically, the card images corresponding to the feature vectors of the merged clusters are identified as the same new image type.

[0167] Step 510: Receive the card images of the identified new card types and perform type merging and / or filtering, and label the card images according to the merging and / or filtering results.

[0168] Multiple new image types obtained through clustering can be manually merged or filtered, and then the new image types can be labeled or new card / certificate types can be defined.

[0169] Step 512: Store the card images whose image types are not recognized into the card image collection.

[0170] By receiving unknown card images to be identified and adding them to a set of card images of unknown types, and returning unidentified card images to the set to be identified, an image recognition method can be executed to discover new types of cards in the card image set when a predetermined number of images have been found. This enables the discovery of new types of cards using unknown card images.

[0171] The image recognition method for identifying new card and document types according to this embodiment can quickly discover and define new card and document types, accumulate data for subsequent classification and structuring tasks, and greatly reduce annotation costs.

[0172] Figure 6 A flowchart of a second image recognition method provided in an embodiment of this specification is shown, including steps 602 to 608.

[0173] Step 602: Display the image input interface to the user based on the user's request.

[0174] Specifically, upon receiving a user's request, the system determines the image input interface based on the request and displays the image input interface to the user, who can then input images through this interface.

[0175] In practical applications, users include, but are not limited to, merchants or individual users.

[0176] Step 604: Receive multiple images to be identified by the user based on the image input interface, and construct a corresponding graph network for each image.

[0177] Step 606: Input the graph network corresponding to the multiple images to be identified into the feature extraction model to obtain the feature vector of the image, wherein the feature extraction model is trained by a metric learning loss function.

[0178] Step 608: Cluster the feature vectors of the multiple images, identify the image types, and return the results to the user.

[0179] The image recognition method provided in this embodiment extracts feature vectors from the image to be recognized by training a feature extraction model through a metric learning loss function on the graph network input of the image to be recognized. This makes clustering more accurate, thereby identifying the image type more quickly and accurately and returning the image type to the user, thus improving the user experience.

[0180] It should be noted that the image recognition method provided in the embodiments of this specification is combined with the above. Figure 2 For the parts corresponding to the embodiments of the image recognition method described above, please refer to the detailed description in the embodiments of the image recognition method described above, and they will not be repeated here.

[0181] Figure 7 A flowchart of a third image recognition method provided in an embodiment of this specification is shown, including steps 702 to 708.

[0182] Step 702: Receive a call request sent by the user, wherein the call request carries multiple images to be identified.

[0183] Step 704: Construct a corresponding graph network for each image.

[0184] Step 706: Input the graph network corresponding to the multiple images to be identified into the feature extraction model to obtain the feature vector of the image, wherein the feature extraction model is trained by a metric learning loss function.

[0185] Step 708: Cluster the feature vectors of the multiple images, identify the image types, and return the results to the user.

[0186] In practical applications, the image recognition method is applied to a local server and provides an API interface for users. After receiving an API call request from a user, the image recognition method extracts feature vectors from the images to be recognized based on the multiple images to be recognized carried in the user's call request. This is achieved by using a feature extraction model trained with a metric learning loss function to make clustering more accurate, thereby identifying the image type more quickly and accurately and returning the image type to the user, thus improving the user experience.

[0187] It should be noted that the image recognition method provided in this embodiment is combined with the above-described method. Figure 2 For the parts corresponding to the embodiments of the image recognition method described above, please refer to the detailed description in the embodiments of the image recognition method described above, and they will not be repeated here.

[0188] Corresponding to the above method embodiments, this specification also provides embodiments of image recognition devices. Figure 8A schematic diagram of the structure of a first image recognition device provided in an embodiment of this specification is shown. Figure 8 As shown, the device includes:

[0189] The first building module 802 is configured to extract multiple images to be identified from the set of images to be identified, and to build a corresponding graph network for each image;

[0190] The first acquisition module 804 is configured to input the graph network corresponding to multiple images to be identified into the feature extraction model to obtain the feature vector of the image, wherein the feature extraction model is trained by a metric learning loss function;

[0191] The first clustering module 806 is configured to cluster the feature vectors of the multiple images to identify the image type.

[0192] The image recognition device provided in this manual extracts feature vectors from the image by inputting the graph network of the image to be recognized into a feature extraction model trained through a metric learning loss function. This makes clustering more accurate, thus enabling faster and more accurate identification of image types.

[0193] Optionally, the image recognition device further includes:

[0194] The add module is configured to receive images to be recognized and add them to the set of images to be recognized.

[0195] Optionally, the first clustering module is further configured to:

[0196] The feature vectors of the multiple images are clustered using a density-based clustering algorithm, and the images corresponding to the vectors that are merged into clusters are identified as the same image type.

[0197] Optionally, the feature extraction model is trained through the following steps:

[0198] Construct multiple triplet training samples consisting of an initial sample image, a positive sample image, and a negative sample image corresponding to the initial sample image, the positive sample image, and the negative sample image;

[0199] The multiple triplet training samples are input into the feature extraction model to obtain the feature vectors of the initial sample image, positive sample image, and negative sample image in each triplet training sample.

[0200] Calculate the first distance between the feature vectors of the initial sample image and the negative sample image in each triplet training sample, and the second distance between the feature vectors of the initial sample image and the positive sample image. Input the first distance and the second distance into the metric learning loss function, and train the feature extraction model according to the output of the loss function until the loss function tends to stabilize.

[0201] Optionally, the first building module is further configured to:

[0202] Perform optical character recognition on each image to obtain text blocks;

[0203] The word embeddings of the text blocks are set as nodes of the graph network, and the relative positional relationships between pairs of text blocks are set as edges of the graph network.

[0204] Optionally, the first obtaining module is further configured to:

[0205] Graph convolution encoding is performed on the nodes and edges of the graph network to obtain the first dimension vector of the nodes and edges;

[0206] The first dimension vector of the graph network is averaged and pooled to obtain the feature vector of the image.

[0207] Optionally, the image recognition device further includes:

[0208] The type labeling module is configured to receive images of identified types and / or perform type merging and / or filtering, and to label the image types according to the results of the merging and / or filtering.

[0209] Optionally, the image recognition device further includes:

[0210] The storage module is configured to store images whose image types are not identified into the set to be identified.

[0211] The above is an illustrative scheme of an image recognition device according to this embodiment. It should be noted that the technical solution of this image recognition device is combined with the above-described... Figure 2 The technical solutions of the image recognition methods described herein belong to the same concept. For details not described in detail in the technical solutions of the image recognition devices, please refer to the description of the technical solutions of the image recognition methods described above.

[0212] This specification also provides an embodiment of an image recognition device. Figure 9 A schematic diagram of the structure of a second image recognition device provided in one embodiment of this specification is shown. For example... Figure 9 As shown, the device includes:

[0213] Display module 902 is configured to display an image input interface to the user based on the user's request;

[0214] The second construction module 904 is configured to receive multiple images to be identified by the user based on the image input interface, and construct a corresponding graph network for each image;

[0215] The second acquisition module 906 is configured to input the graph network corresponding to the multiple images to be identified into the feature extraction model to obtain the feature vector of the image, wherein the feature extraction model is trained by a metric learning loss function;

[0216] The second clustering module 908 is configured to cluster the feature vectors of the multiple images, identify the image type, and return the results to the user.

[0217] The image recognition device provided in this embodiment takes the graph network of the image to be recognized as input and uses the feature extraction model trained by the metric learning loss function to extract feature vectors from the image to be recognized, making clustering more accurate, thereby identifying the image type more quickly and accurately and returning the image type to the user, thus improving the user experience.

[0218] The above is an illustrative scheme of an image recognition device according to this embodiment. It should be noted that the technical solution of this image recognition device is combined with the above-described... Figure 6 The technical solutions of the image recognition methods described herein belong to the same concept. For details not described in detail in the technical solutions of the image recognition devices, please refer to the description of the technical solutions of the image recognition methods described above.

[0219] This specification also provides an embodiment of an image recognition device. Figure 10 A schematic diagram of the structure of a third image recognition device provided in one embodiment of this specification is shown. Figure 10 As shown, the device includes:

[0220] The receiving module 1002 is configured to receive a call request sent by a user, wherein the call request carries multiple images to be identified;

[0221] The third building module 1004 is configured to build a corresponding graph network for each image;

[0222] The third acquisition module 1006 is configured to input the graph network corresponding to the multiple images to be identified into the feature extraction model to obtain the feature vector of the image, wherein the feature extraction model is trained by a metric learning loss function;

[0223] The third clustering module 1008 is configured to cluster the feature vectors of the multiple images, identify the image type, and return the results to the user.

[0224] The image recognition device provided in this embodiment takes the graph network of the image to be recognized as input and uses the feature extraction model trained by the metric learning loss function to extract feature vectors from the image to be recognized, making clustering more accurate, thereby identifying the image type more quickly and accurately and returning the image type to the user, thus improving the user experience.

[0225] The above is an illustrative scheme of an image recognition device according to this embodiment. It should be noted that the technical solution of this image recognition device is combined with the above-described... Figure 7 The technical solutions of the image recognition methods described herein belong to the same concept. For details not described in detail in the technical solutions of the image recognition devices, please refer to the description of the technical solutions of the image recognition methods described above.

[0226] This specification also provides a model training device in its embodiments. Figure 11 A schematic diagram of a model training device according to an embodiment of this specification is shown. Figure 11 As shown, the device includes:

[0227] The fourth construction module 1102 is configured to construct multiple triplet training samples consisting of an initial sample image, a positive sample image, and a negative sample image corresponding to the initial sample image, the positive sample image, and the negative sample image.

[0228] The fourth obtaining module 1104 is configured to input the multiple triple training samples into the feature extraction model to obtain the feature vectors of the initial sample image, positive sample image, and negative sample image in each triple training sample.

[0229] The calculation module 1106 is configured to calculate a first distance between the feature vectors of the initial sample image and the negative sample image in each triplet training sample and a second distance between the feature vectors of the initial sample image and the positive sample image, input the first distance and the second distance into the metric learning loss function, and train the feature extraction model according to the output of the loss function until the loss function tends to stabilize.

[0230] The above is an illustrative scheme of a model training device according to this embodiment. It should be noted that the technical solution of this model training device is combined with the above-described approach. Figure 3 The technical solutions for the model training methods described herein belong to the same concept. For details not described in the technical solution of the model training device, please refer to the description of the technical solution of the model training method described above.

[0231] Figure 12 A structural block diagram of a computing device 1200 according to an embodiment of this specification is shown. The components of the computing device 1200 include, but are not limited to, a memory 1210 and a processor 1220. The processor 1220 is connected to the memory 1210 via a bus 1230, and a database 1250 is used to store data.

[0232] The computing device 1200 also includes an access device 1240, which enables the computing device 1200 to communicate via one or more networks 1260. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 1240 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0233] In one embodiment of this specification, the aforementioned components of the computing device 1200 and Figure 12 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 12 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0234] The computing device 1200 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 1200 can also be a mobile or stationary server.

[0235] The processor 1220 is configured to execute the following computer-executable instructions, wherein when the processor executes the computer-executable instructions, it implements the steps of the image recognition method or the steps of the model training method.

[0236] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the image recognition method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the image recognition method described above.

[0237] An embodiment of this specification also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the image recognition method or the steps of the model training method.

[0238] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the image recognition method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the image recognition method described above.

[0239] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0240] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0241] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this specification is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this specification. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this specification.

[0242] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0243] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. These embodiments have been selected and specifically described in this specification to better explain the principles and practical applications of this specification, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An image recognition method applied to images containing text content, comprising: Multiple images to be identified are extracted from the set to be identified, and a corresponding graph network is constructed for each image. The nodes of the graph network are word embeddings of text blocks in the image, and the edges of the graph network are the relative positional relationship features of pairs of text blocks in the image. The graph network corresponding to multiple images to be identified is input into the feature extraction model to obtain the feature vector of the image. The feature extraction model is trained by combining semantic and structural features and using a metric learning loss function to calculate the first distance between the feature vectors of the initial sample image and the negative sample image in the triplet training samples, and the second distance between the feature vectors of the initial sample image and the positive sample image. The positive sample image and the initial sample image are of the same image type, and the negative sample image and the initial sample image are of different image types. Cluster the feature vectors of the multiple images to identify the image type.

2. The image recognition method according to claim 1, before extracting multiple images to be recognized from the set to be recognized, further includes: Receive the image to be identified and add it to the set to be identified.

3. The image recognition method according to claim 1 or 2, wherein, Clustering the feature vectors of the multiple images to identify image types includes: The feature vectors of the multiple images are clustered using a density-based clustering algorithm, and the images corresponding to the vectors that are merged into clusters are identified as the same image type.

4. The image recognition method according to claim 1 or 2, wherein, The feature extraction model is trained through the following steps: Construct multiple triplet training samples consisting of an initial sample image, a positive sample image, and a negative sample image corresponding to the initial sample image, the positive sample image, and the negative sample image; The multiple triplet training samples are input into the feature extraction model to obtain the feature vectors of the initial sample image, positive sample image, and negative sample image in each triplet training sample. Calculate the first distance between the feature vectors of the initial sample image and the negative sample image in each triplet training sample, and the second distance between the feature vectors of the initial sample image and the positive sample image. Input the first distance and the second distance into the metric learning loss function, and train the feature extraction model according to the output of the loss function until the loss function tends to stabilize.

5. The image recognition method according to claim 1 or 2, wherein, The process of constructing a corresponding graph network for each image includes: Perform optical character recognition on each image to obtain text blocks; The word embeddings of the text blocks are set as nodes of the graph network, and the relative positional relationships between pairs of text blocks are set as edges of the graph network.

6. The image recognition method according to claim 5, wherein, The graph network corresponding to multiple images to be identified is input into the feature extraction model to obtain the feature vector of the image, including: Graph convolution encoding is performed on the nodes and edges of the graph network to obtain the first dimension vector of the nodes and edges; The first dimension vector of the graph network is averaged and pooled to obtain the feature vector of the image.

7. The image recognition method according to claim 1 or 2 further includes: The system receives images of the identified types and performs type merging and / or filtering, and labels the images with their types based on the results of the merging and / or filtering.

8. The image recognition method according to claim 1 or 2, further comprising: Images whose image types are not identified are stored in the set to be identified.

9. An image recognition method applied to images containing text content, comprising: Based on the user's request, display the image input interface to the user; The system receives multiple images to be identified by the user based on the image input interface, and constructs a corresponding graph network for each image. The nodes of the graph network are word embeddings of text blocks in the image, and the edges of the graph network are the relative positional relationship features of pairs of text blocks in the image. The graph network corresponding to multiple images to be identified is input into the feature extraction model to obtain the feature vector of the image. The feature extraction model is trained by combining semantic and structural features and using a metric learning loss function to calculate the first distance between the feature vectors of the initial sample image and the negative sample image in the triplet training samples, and the second distance between the feature vectors of the initial sample image and the positive sample image. The positive sample image and the initial sample image are of the same image type, and the negative sample image and the initial sample image are of different image types. The feature vectors of the multiple images are clustered to identify the image type and the result is returned to the user.

10. An image recognition method applied to images containing text content, comprising: Receive a call request sent by a user, wherein the call request carries multiple images to be identified; A corresponding graph network is constructed for each image, wherein the nodes of the graph network are the word embeddings of the text blocks in the image, and the edges of the graph network are the relative positional relationship features of pairs of text blocks in the image; The graph network corresponding to multiple images to be identified is input into the feature extraction model to obtain the feature vector of the image. The feature extraction model is trained by combining semantic and structural features and using a metric learning loss function to calculate the first distance between the feature vectors of the initial sample image and the negative sample image in the triplet training samples, and the second distance between the feature vectors of the initial sample image and the positive sample image. The positive sample image and the initial sample image are of the same image type, and the negative sample image and the initial sample image are of different image types. The feature vectors of the multiple images are clustered to identify the image type and the result is returned to the user.

11. A model training method, comprising: Multiple triplet training samples are constructed, consisting of an initial sample image, a positive sample image, and a negative sample image corresponding to the initial sample image, the positive sample image, and the negative sample image. The process of constructing the graph network includes performing optical character recognition on each image to obtain text blocks, setting the word embeddings of the text blocks as nodes of the graph network, and setting the relative positional relationship between pairs of text blocks as edges of the graph network, thereby obtaining the initial sample graph network, the positive sample graph network, and the negative sample graph network. The multiple triplet training samples are input into the feature extraction model, and the feature vectors of the initial sample image, positive sample image, and negative sample image in each triplet training sample are obtained by combining semantic and structural features. The positive sample image and the initial sample image are of the same image type, and the negative sample image and the initial sample image are of different image types. Calculate the first distance between the feature vectors of the initial sample image and the negative sample image in each triplet training sample, and the second distance between the feature vectors of the initial sample image and the positive sample image. Input the first distance and the second distance into the metric learning loss function, and train the feature extraction model according to the output of the loss function until the loss function tends to stabilize.

12. An image recognition device, applied to an image containing text content, comprising: The first construction module is configured to extract multiple images to be identified from the set of images to be identified, and construct a corresponding graph network for each image. The nodes of the graph network are word embeddings of text blocks in the image, and the edges of the graph network are the relative positional relationship features of pairs of text blocks in the image. The first acquisition module is configured to input the graph network corresponding to multiple images to be identified into a feature extraction model to obtain the feature vector of the image. The feature extraction model is trained by combining semantic and structural features and using a metric learning loss function to calculate the first distance between the feature vectors of the initial sample image and the negative sample image in the triplet training samples, and the second distance between the feature vectors of the initial sample image and the positive sample image. The positive sample image and the initial sample image are of the same image type, and the negative sample image and the initial sample image are of different image types. The first clustering module is configured to cluster the feature vectors of the multiple images to identify the image type.

13. The image recognition device according to claim 12, further comprising: The add module is configured to receive images to be recognized and add them to the set of images to be recognized.

14. The image recognition device according to claim 12 or 13, wherein, The first clustering module is also configured as follows: The feature vectors of the multiple images are clustered using a density-based clustering algorithm, and the images corresponding to the vectors that are merged into clusters are identified as the same image type.

15. The image recognition device according to claim 12 or 13, wherein, The feature extraction model is trained through the following steps: Construct multiple triplet training samples consisting of an initial sample image, a positive sample image, and a negative sample image corresponding to the initial sample image, the positive sample image, and the negative sample image; The multiple triplet training samples are input into the feature extraction model, and the feature vectors of the initial sample image, positive sample image, and negative sample image in each triplet training sample are obtained by combining semantic and structural features. Calculate the first distance between the feature vectors of the initial sample image and the negative sample image in each triplet training sample, and the second distance between the feature vectors of the initial sample image and the positive sample image. Input the first distance and the second distance into the metric learning loss function, and train the feature extraction model according to the output of the loss function until the loss function tends to stabilize.

16. The image recognition device according to claim 12 or 13, wherein, The first building module is also configured as follows: Perform optical character recognition on each image to obtain text blocks; The word embeddings of the text blocks are set as nodes of the graph network, and the relative positional relationships between pairs of text blocks are set as edges of the graph network.

17. The image recognition device according to claim 16, wherein, The first obtaining module is also configured to: Graph convolution encoding is performed on the nodes and edges of the graph network to obtain the first dimension vector of the nodes and edges; The first dimension vector of the graph network is averaged and pooled to obtain the feature vector of the image.

18. The image recognition device according to claim 12 or 13, further comprising: The type labeling module is configured to receive images of identified types and / or perform type merging and / or filtering, and to label the image types according to the results of the merging and / or filtering.

19. The image recognition device according to claim 12 or 13, further comprising: The storage module is configured to store images whose image type is not identified into the set to be identified.

20. An image recognition device, applied to an image containing text content, comprising: The display module is configured to show the user an image input interface based on the user's request. The second construction module is configured to receive multiple images to be identified by the user based on the image input interface, and construct a corresponding graph network for each image, wherein the nodes of the graph network are word embeddings of text blocks in the image, and the edges of the graph network are the relative positional relationship features of pairs of text blocks in the image. The second acquisition module is configured to input the graph network corresponding to multiple images to be identified into the feature extraction model to obtain the feature vector of the image. The feature extraction model is trained by combining semantic and structural features and using a metric learning loss function to calculate the first distance between the feature vectors of the initial sample image and the negative sample image in the triplet training samples, and the second distance between the feature vectors of the initial sample image and the positive sample image. The positive sample image and the initial sample image are of the same image type, and the negative sample image and the initial sample image are of different image types. The second clustering module is configured to cluster the feature vectors of the multiple images, identify the image type, and return the results to the user.

21. An image recognition device, applied to an image containing text content, comprising: The receiving module is configured to receive a call request sent by a user, wherein the call request carries multiple images to be identified; The third construction module is configured to construct a corresponding graph network for each image, wherein the nodes of the graph network are word embeddings of text blocks in the image, and the edges of the graph network are the relative positional relationship features of pairs of text blocks in the image. The third acquisition module is configured to input the graph network corresponding to multiple images to be identified into the feature extraction model to obtain the feature vector of the image. The feature extraction model is trained by combining semantic and structural features and using a metric learning loss function to calculate the first distance between the feature vectors of the initial sample image and the negative sample image in the triplet training samples, and the second distance between the feature vectors of the initial sample image and the positive sample image. The positive sample image and the initial sample image are of the same image type, and the negative sample image and the initial sample image are of different image types. The third clustering module is configured to cluster the feature vectors of the multiple images, identify the image type, and return the results to the user.

22. A model training device, comprising: The fourth construction module is configured to construct multiple triplet training samples consisting of an initial sample graph network, a positive sample graph network, and a negative sample graph network corresponding to the initial sample image, positive sample image, and negative sample image. The process of constructing the graph network includes performing optical character recognition on each image to obtain text blocks, setting the word embeddings of the text blocks as nodes of the graph network, and setting the relative positional relationship between pairs of text blocks as edges of the graph network, thereby obtaining the initial sample graph network, the positive sample graph network, and the negative sample graph network. The fourth acquisition module is configured to input the multiple triple training samples into the feature extraction model, and combine semantic and structural features to obtain the feature vectors of the initial sample image, positive sample image, and negative sample image in each triple training sample, wherein the positive sample image and the initial sample image are of the same image type, and the negative sample image and the initial sample image are of different image types. The computation module is configured to calculate a first distance between the feature vectors of the initial sample image and the negative sample image in each triplet training sample, and a second distance between the feature vectors of the initial sample image and the positive sample image. The first distance and the second distance are input into a metric learning loss function, and the feature extraction model is trained based on the output of the loss function until the loss function tends to stabilize.

23. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, wherein when the processor executes the computer-executable instructions, it implements the steps of the image recognition method according to any one of claims 1 to 10, or implements the steps of the model training method according to claim 11.

24. A computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the image recognition method of any one of claims 1 to 10, or the steps of the model training method of claim 11.

Images