Image search method, device, and apparatus

Abstract

The embodiment of the present invention provides an image search method, device, and equipment, the method comprising: obtaining a search input image, wherein the search input image contains multiple first text blocks; selecting a to-be-processed image from a search target database, wherein the to-be-processed image contains multiple second text blocks; generating a first graph structure feature according to the multiple first text blocks, and generating a second graph structure feature according to the multiple second text blocks; determining that the first graph structure feature and the second graph structure feature meet a preset condition; and outputting the to-be-processed image as a search result. In this method, based on the idea that images of the same type have a high degree of similarity in graph structure features, by comparing the graph structure features of each to-be-processed image with the search input image, it is possible to accurately find the to-be-processed image that belongs to the same category as the search input image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image search method, device and equipment. Background Art

[0002] As image processing technology is applied to more and more fields, the number of images that need to be processed is also increasing. At present, in the process of image processing, it is often necessary to search for the desired image from a database that stores various images. For example, searching for ID card images from a database that stores images of train tickets, ID cards, orders, etc. However, the more images stored in the database, the more difficult it is to search for a certain category of images in the database.

[0003] Therefore, how to accurately search for a certain type of image that one wants to use from a variety of images has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present invention provide an image search method, device, and equipment for accurately searching for a certain type of image that is desired to be used.

[0005] In a first aspect, an embodiment of the present invention provides an image search method, the image search method comprising:

[0006] Acquire a search input image, wherein the search input image contains a plurality of first text blocks;

[0007] Selecting an image to be processed from a search target database, wherein the image to be processed includes a plurality of second text blocks;

[0008] Generate a first graph structure feature according to the plurality of first text blocks, and generate a second graph structure feature according to the plurality of second text blocks;

[0009] Determine the difference between the first graph structural feature and the second graph structural feature;

[0010] The image to be processed is output as the search result.

[0011] In a second aspect, an embodiment of the present invention provides an image search device, the image search device comprising:

[0012] An acquisition module is used to acquire a search input image, wherein a plurality of first text blocks contained in the search input image are retrieved; an image to be processed and a plurality of second text blocks contained in the image to be processed are selected from a search target database;

[0013] The search module is used to generate a first graph structure feature according to a plurality of first text blocks, and to generate a second graph structure feature according to a plurality of second text blocks; determine that the first graph structure feature and the second graph structure feature meet a preset condition; and output an image to be processed as a search result.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the image search method in the first aspect.

[0015] An embodiment of the present invention provides a non-transitory machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor can at least implement the method in the first aspect.

[0016] In the embodiment of the present invention, since multiple images belonging to the same category will show certain similarities in graph structure features, when it is necessary to search for images similar to a certain retrieval input image from a retrieval target database storing a large number of images, first, multiple first text blocks contained in the retrieval input image are annotated, a to-be-processed image is selected from the retrieval target database, and multiple second text blocks contained in the to-be-processed image are identified, so as to generate first graph structure features corresponding to the retrieval input image according to the multiple first text blocks, and generate second graph structure features corresponding to the to-be-processed image according to the multiple second text blocks. By comparing the first graph structure features and the second graph structure features, it can be determined whether the first graph structure features and the second graph structure features meet the preset conditions, that is, whether the to-be-processed image has similarities with the retrieval input image in graph structure features, so that all images similar to the retrieval input image in the retrieval target database can be searched out as the search results of the image search. In this scheme, based on the idea that images of the same type have a high degree of similarity in graph structure features, by comparing the graph structure features of each to-be-processed image with the retrieval input image, it is possible to accurately find out the to-be-processed image belonging to the same category as the retrieval input image. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A flowchart of an image search method provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of retrieving an input image provided by an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of a graph structure provided by an embodiment of the present invention;

[0021] Figure 4a A schematic diagram of an image to be processed provided by an embodiment of the present invention;

[0022] Figure 4b A schematic diagram of another graph structure provided by an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of another graph structure provided by an embodiment of the present invention;

[0024] Figure 6 A schematic diagram of another graph structure provided by an embodiment of the present invention;

[0025] Figure 7 A schematic diagram of the principle of an image search process provided by an embodiment of the present invention;

[0026] Figure 8 A schematic diagram of another principle of an image search process provided by an embodiment of the present invention;

[0027] Fig. 9 A schematic diagram of the structure of an image search device provided by an embodiment of the present invention;

[0028] Fig.10 For Fig. 9 A schematic structural diagram of an electronic device corresponding to the image search device provided in the illustrated embodiment. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0031] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0032] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0033] The image search solution provided in the embodiment of the present invention can be executed by an electronic device, which can be a terminal device such as a smart phone, a tablet computer, a PC, a laptop, etc., or a server. The server can be a physical server including an independent host, or a virtual server hosted by a host cluster, or a cloud server.

[0034] The image search solution provided by the embodiment of the present invention is applicable to the scenario of searching for an image to be used from a database storing a large number of images. In practical applications, the image to be used is, for example, a ticket image with fixed content, such as a value-added tax invoice, an itinerary, a train ticket, and a return application form.

[0035] The following examples are used to illustrate the actual requirements of the image search provided by the embodiments of the present invention.

[0036] For example, the goal of image search is to search for itineraries from reimbursement vouchers. Taking the company's transportation expense reimbursement scenario as an example, in order to calculate the company's transportation expenses incurred during a certain period of time, it is necessary to first search for itineraries from a large number of reimbursement vouchers uploaded by employees to the office backend database for verification.

[0037] For another example, the goal of image search is to search for real estate loan application forms from audit documents. Taking the risk control department of a bank as an example, in order to analyze the current application status of real estate loan business, it is necessary to first search for real estate loan application forms from a large number of audit documents stored in the bank's backend database.

[0038] Obviously, if a large number of images are stored in the database, how to search for a certain category of images that one wants to use from the database becomes an urgent problem to be solved, which is also the core purpose of the image search method provided in the embodiment of the present invention.

[0039] In view of the practical needs mentioned above, in summary, the solution idea of ​​the image search method provided by the embodiment of the present invention is: since multiple images belonging to the same category will show certain similarities in graph structure features, it is possible to determine whether different images belong to the same category by comparing the graph structure features corresponding to different images. Here, the similarity of multiple images in graph structure features can be understood as the similarity of multiple images in text content and the positional relationship between text contents. Based on the above idea, when it is necessary to search for an image similar to a certain retrieval input image from a retrieval target database storing a large number of images, by comparing the graph structure features of each image to be processed (referring to each image stored in the retrieval target database) with the retrieval input image, it can be determined whether each image to be processed has similarities with the retrieval input image in graph structure features, thereby accurately finding the image to be processed that belongs to the same category as the retrieval input image as the search result of the image search.

[0040] The execution process of the image search method is described below in conjunction with the following embodiments.

[0041] Figure 1 A flowchart of an image search method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method comprises the following steps:

[0042] 101. Obtain a retrieval input image, wherein the retrieval input image includes a plurality of first text blocks.

[0043] 102. Select an image to be processed from a retrieval target database, wherein the image to be processed includes a plurality of second text blocks.

[0044] In an embodiment of the present invention, for a search task of searching for an image of a certain target category from a database storing a large number of images of various categories, it is first necessary to set a retrieval input image corresponding to the target category, and then search the database for an image that meets the similarity requirement with the retrieval input image in terms of graph structure features. In this article, a database storing a large number of images of various categories is referred to as a retrieval target database.

[0045] To ensure the accuracy of image search results, the retrieval input image should have good image quality, such as image clarity and brightness should be good, and the background should be as simple as possible.

[0046] Optionally, taking the current image search task of searching for an ID card image from a large number of images as an example, at this time, the relevant staff can take a photo of an ID card image with better quality as the retrieval input image corresponding to the image search task.

[0047] Or, optionally, the staff may also pre-set images with better image quality corresponding to a certain image search task as retrieval input images based on the image search tasks that may be encountered in actual applications. Thus, for a certain image search task that needs to be performed currently, the pre-set images corresponding to the image search task can be directly used as retrieval input images.

[0048] After obtaining the retrieval input image, in order to construct a graph structure corresponding to the retrieval input image, it is also necessary to mark out multiple text blocks contained in the retrieval input image, which are referred to as multiple first text blocks for distinction.

[0049] It is worth noting that if the retrieval input image is obtained through the above-mentioned method of "directly using the pre-set image corresponding to the current image search task as the retrieval input image", then it should be understood that the retrieval input image has been pre-annotated with multiple first text blocks.

[0050] Among them, multiple first text blocks refer to various areas in the search input image that contain independent fixed text content. In other words, the text content contained in the search input image can be divided into fixed text content and non-fixed text content. Fixed content is often irrelevant to the user, while non-fixed content is often related to the user. Here, only the fixed text content contained therein is marked, and multiple first text blocks can be obtained. Simply put, a first text block can be understood as a word or sentence.

[0051] The following example illustrates how to annotate the first text block of a retrieval input image: Assume that the retrieval input image is Figure 2 The illustrated account opening license contains fixed text contents including: "Account opening license", "Approval number", "Number", "After review", "Meet the account opening conditions, approved", "Open a basic deposit account.", "Legal representative (head of the unit)", "Opening bank", "Account number", "Issuing authority (stamp)", etc. These fixed text contents are marked in text boxes to obtain Figure 3 The diagram illustrates a plurality of first text blocks of different sizes and positions.

[0052] Depend on Figure 3 It can be seen that after the above-mentioned annotation processing of the retrieval input image, on the one hand, the various fixed text contents contained in the retrieval input image can be obtained, and on the other hand, the positional relationship between these fixed text contents can be obtained. These fixed text contents and the positional relationship between each other will be the key to the subsequent construction of the first graph structure feature corresponding to the retrieval input image.

[0053] In order to search for images belonging to the same category as the retrieval input image in a retrieval target database storing images of various categories, it is also necessary to identify the text blocks contained in the image to be processed selected from the retrieval target database, which are called multiple second text blocks for distinction. In practical applications, the method of selecting the image to be processed can be random selection or other image selection methods.

[0054] For the image to be processed, multiple second text blocks contained in the image to be processed can be obtained by using optical character recognition (OCR) software.

[0055] In this embodiment, the second text blocks refer to the regions containing independent text content in the image to be processed. That is to say, the text content contained in the image to be processed can also be divided into fixed text content irrelevant to the user and non-fixed text content related to the user. Here, the text content in the image to be processed is identified to determine the text content included in the image to be processed, and a plurality of second text blocks are obtained. Thus, the second text blocks are divided according to the layout information (such as line information, distance information between adjacent characters, etc.) and semantic relevance of the text content in the image to be processed.

[0056] To facilitate understanding of the meaning of the second text block, for example, assume that the image to be processed is Figure 4a The account opening license shown is Figure 4a The account opening license shown in the figure is subjected to OCR recognition, and non-text content such as background graphics, borders, official seals, etc. included in the account opening license is filtered out, and the remaining text content is divided according to distribution characteristics, where the text content includes: "Account opening license", "Approval number", "Number", "After review," "XX Co., Ltd.", "Legal representative", "(Head of unit)", "Opening bank", "Account number", "Issuing authority", "XX institution special seal", etc., and the Figure 4b Schematic diagram of multiple second text blocks with different sizes and positions.

[0057] Depend on Figure 4b It can be seen that after the above-mentioned recognition processing of the image to be processed, the various text contents contained in the image to be processed and the positional relationship between these text contents can be obtained. These text contents and the positional relationship between each other will also be the key to the subsequent construction of the second graph structure feature corresponding to the image to be processed.

[0058] 103. Generate a first graph structure feature according to the multiple first text blocks, and generate a second graph structure feature according to the multiple second text blocks.

[0059] The generating of the first graph structure feature according to the plurality of first text blocks may be implemented as follows:

[0060] According to the text content contained in each of the multiple first text blocks, the semantic vectors corresponding to each of the multiple first text blocks are generated; according to the positional relationship between the multiple first text blocks, the positional relationship vectors corresponding to different first text blocks are generated; and the first graph structure features composed of multiple groups of first feature vectors are generated, wherein each group of first feature vectors is composed of the semantic vector corresponding to the first text block i, the semantic vector corresponding to the first text block j, and the positional relationship vector between the first text block i and the first text block j. The first text block i is any one of the multiple first text blocks, and the first text block j is any one of the multiple first text blocks except the first text block i.

[0061] From the above-mentioned method of acquiring the first graph structure feature, it can be seen that the first graph structure feature includes both semantic features corresponding to multiple first text blocks and positional relationship features between multiple first text blocks, thereby promoting the accuracy of image search results.

[0062] Optionally, generating semantic vectors corresponding to each of the plurality of first text blocks according to the text contents respectively contained in the plurality of first text blocks may be implemented as follows:

[0063] Words contained in each of the plurality of first text blocks are encoded to obtain corresponding word vectors; and a mean value of the plurality of word vectors corresponding to each first text block is calculated to obtain a semantic vector corresponding to each first text block.

[0064] Specifically, in practical applications, a large number of corpus samples can be obtained in advance, and a number of words can be obtained by performing word segmentation on the large number of corpus samples. Then, the words can be sorted according to the number of occurrences of each word, for example, sorted from most to least according to the number of occurrences. Assuming that there are N words in total, a word list consisting of N words arranged in sequence can be generated. In addition, for each word, a word vector conversion can be performed according to an existing word vector conversion algorithm. Each word is assumed to be represented as an M-dimensional row vector, so that an N*M-dimensional word vector matrix is ​​finally obtained, wherein the k-th row word vector of the word vector matrix corresponds to the k-th word in the word list.

[0065] Based on this, for the first text block i among multiple first text blocks, the words contained in the first text block i can be encoded according to the correspondence between the word vector matrix and the word table to convert the words contained in the first text block i into corresponding word vectors.

[0066] Generally speaking, the first text block i usually includes multiple words, each word corresponds to a word vector, so the first text block i corresponds to multiple word vectors, the mean of the multiple word vectors is calculated, and the mean calculation result is used as the semantic vector of the first text block i.

[0067] It is worth noting that the method of generating semantic vectors corresponding to each of the multiple first text blocks is not limited to the implementation method exemplified above. Other methods can also be used to obtain the semantic vectors corresponding to each of the multiple first text blocks, such as obtaining the semantic vector through a neural network model.

[0068] Optionally, generating corresponding position relationship vectors between different first text blocks according to the positional relationships between the multiple first text blocks may be implemented as follows: for a first text block i and a first text block j, determining that the corresponding position relationship vector between the first text block i and the first text block j includes at least one of the following ratios:

[0069] The aspect ratio of the first text block i;

[0070] The ratio of the width of the first text block j to the height of the first text block i;

[0071] The ratio of the horizontal distance between the first text block j and the first text block i to the height of the first text block i;

[0072] The ratio of the vertical distance between the first text block j and the first text block i to the height of the first text block i.

[0073] In practical applications, it is understood that when obtaining multiple first text blocks contained in the search input image, the position coordinates of each first text block are also marked. The position coordinates of each first text block are represented by the four vertex coordinates of the corresponding rectangular box. Based on these four vertex coordinates, the length, width, and height of each first text block can be obtained.

[0074] Based on this, it is assumed that according to the position coordinates corresponding to the first text block i, the width of the first text block i is calculated to be W i , height H i , so that the aspect ratio of the first text block i is W i / H i .

[0075] Assume that according to the position coordinates corresponding to the first text block j, the width of the first text block j is calculated to be W j , so that the ratio of the width of the first text block j to the height of the first text block i is W j / H i .

[0076] Assume that the horizontal distance between the first text blocks i and j is calculated as X according to the position coordinates corresponding to the first text blocks i and j. ji , so that the ratio of the horizontal distance to the height of the first text block i is X ji / H i Among them, the horizontal distance Xji The calculation method is as follows: for example, the coordinates of the center point of the rectangular box corresponding to each first text block are calculated, and the horizontal distance is obtained by subtracting the horizontal coordinates of the center point coordinates of the two text blocks.

[0077] Specifically, the horizontal distance X ji The calculation method can be implemented as follows: Assume that the coordinates of the center point of the rectangular box corresponding to the first text block i are (x i1 ,y i1 ), the coordinates of the center point of the rectangular box corresponding to the first text block j are (x j1 ,y j1 ), then the horizontal distance X between the first text block i and the first text block j ji , which can be expressed as: X ji =|x j1 -x i1 |.

[0078] Assume that the vertical distance between the first text blocks i and j is calculated as Y according to the position coordinates corresponding to the first text blocks i and j. ji , so that the ratio of the vertical distance to the height of the first text block i is Y ji / H i Among them, the vertical distance Y ji The calculation method is: for example, the coordinates of the center point of the rectangular box corresponding to each first text block are calculated, and the vertical distance is obtained by subtracting the vertical coordinates of the center point coordinates of the two text blocks.

[0079] Specifically, the vertical distance Y ji The calculation method can be implemented as follows: Assume that the coordinates of the center point of the rectangular box corresponding to the first text block i are (x i1 ,y i1 ), the coordinates of the center point of the rectangular box corresponding to the first text block j are (x j1 ,y j1 ), then the vertical distance Y between the first text block i and the first text block j ji , which can be expressed as: ji =|y j1 -y i1 |.

[0080] Optionally, the above four ratios are used to represent the positional relationship between the first text block i and the first text block j, and the positional relationship vector corresponding to the first text block i and the first text block j is obtained, which can be expressed as: R ij =[W i / H i , W j / H i , X ji / H i , Yji / H i ].

[0081] Of course, other combinations of the above four ratios may also be used to generate the corresponding position relationship vector between the first text block i and the first text block j, and the present invention is not limited thereto.

[0082] Obviously, annotating the retrieval input image with multiple first text blocks and generating semantic vectors of the multiple first text blocks and position relationship vectors corresponding to different first text blocks is actually a process of describing the retrieval input image as a graph structure, which is composed of nodes and edges. Among them, the semantic vectors corresponding to the multiple first text blocks are regarded as nodes of the graph structure, and the position relationship vectors between different first text blocks are regarded as connecting edges in the graph structure.

[0083] For ease of understanding, combined Figure 5 To illustrate the graph structure corresponding to the retrieved input image. Figure 5 is the graph structure corresponding to the retrieval input image. Assume that the retrieval input image includes four first text blocks, namely, first text block 1 to first text block 4. Figure 5 In , the semantic vectors corresponding to the four first text blocks are represented as t1 to t4, where ti represents the semantic vector corresponding to the first text block i (i.e., the i-th first text block). It can be assumed that there is a connection relationship between each first text block and the remaining first text blocks. Thus, for the first text block 1, it is assumed that the connection edges between the first text block 1 (corresponding to t1) and the other three first text blocks are represented as: R12, R13, R14. Figure 5 Only the fully connected relationship between the first text block 1 and the remaining first text blocks is illustrated.

[0084] The graph structure corresponding to the retrieved input image is represented by the first graph structure feature. For each first text block, the semantic vector corresponding to the first text block, the position relationship vector corresponding to the first text block and other first text blocks, and the semantic vectors corresponding to the other first text blocks constitute a group of first feature vectors corresponding to the first text block. The other first text blocks are any one of the multiple first text blocks except the first text block. Therefore, for multiple first text blocks, by generating multiple groups of first feature vectors corresponding to the multiple first text blocks, the first graph structure feature composed of the multiple groups of first feature vectors can be obtained.

[0085] For a first text block i among the plurality of first text blocks, assuming that the number of the plurality of first text blocks is n, assuming that the first text block j is any one of the n first text blocks except the first text block i. The semantic vector t corresponding to the first text block i is i, the position relationship vector R corresponding to the first text block i and the first text block j ij , and the semantic vector t corresponding to the first text block j j , forming a set of first feature vectors corresponding to the first text block i [t i , R ij , t j Based on this, the (n-1) first feature vectors corresponding to the first text block i are generated. Figure 5 To illustrate the composition of the three groups of first feature vectors corresponding to the first text block 1, namely t1, they are respectively expressed as: [t1, R12, t2], [t1, R13, t3], and [t1, R14, t4].

[0086] The process of generating multiple groups of first feature vectors corresponding to other first text blocks contained in the retrieved input image is consistent with the process of generating multiple groups of first feature vectors corresponding to the first text block 1, and will not be described in detail.

[0087] Finally, a plurality of groups of first feature vectors corresponding to a plurality of first text blocks constitute the first graph structure feature.

[0088] The above describes the process of generating the first graph structure feature corresponding to the retrieval input image for the multiple first text blocks contained in the retrieval input image. The process of generating the second graph structure feature corresponding to the image to be processed is similar, but also has differences.

[0089] Similar to the generation process of the first graph structure feature described above, in an optional implementation, generating the second graph structure feature according to the plurality of second text blocks may be implemented as follows:

[0090] According to the text content contained in each of the plurality of second text blocks, a semantic vector corresponding to each of the plurality of second text blocks is generated; according to the positional relationship between the plurality of second text blocks, a positional relationship vector corresponding to different second text blocks is generated; and a second graph structure feature composed of a plurality of groups of second feature vectors is generated, wherein each group of second feature vectors is composed of a semantic vector corresponding to the second text block p, a semantic vector corresponding to the second text block q, and a positional relationship vector between the second text block p and the second text block q. The second text block p is any one of the plurality of second text blocks, and the second text block q is any one of the plurality of second text blocks except the second text block p.

[0091] The generation process of the second eigenvector is similar to the generation process of the first eigenvector in the previous text, and will not be repeated here.

[0092] From the above-mentioned method of acquiring the second graph structure feature, it can be seen that the second graph structure feature also includes semantic features corresponding to multiple second text blocks and positional relationship features between multiple second text blocks, which can reflect the layout structure characteristics of the text content of the image to be processed, thereby ensuring the accuracy of image search results and also playing a promoting role.

[0093] Different from the above-mentioned method for obtaining the second graph structural feature, in another optional implementation, generating the second graph structural feature according to multiple second text blocks can be implemented as follows:

[0094] A second graph structure feature is generated based on a plurality of first text blocks and a plurality of second text blocks. Specifically, a semantic vector corresponding to each of the plurality of second text blocks is generated based on the text content contained in each of the plurality of second text blocks; a position relationship vector corresponding to different second text blocks is generated based on the position relationship between the plurality of second text blocks; and a second graph structure feature consisting of a plurality of groups of second feature vectors and a plurality of groups of third feature vectors is generated, wherein each group of second feature vectors consists of a semantic vector corresponding to the second text block p, a semantic vector corresponding to the second text block q, and a position relationship vector between the second text block p and the second text block q, and each group of third feature vectors includes a semantic vector corresponding to the first text block i and a semantic vector corresponding to the second text block p. The second text block p is any one of the plurality of second text blocks, the second text block q is any one of the plurality of second text blocks except the second text block p, and the first text block i is any one of the plurality of first text blocks.

[0095] Here, the generation process of the second eigenvector is similar to the generation process of the first eigenvector in the previous text, and will not be repeated here.

[0096] It can be seen from the above implementation that the difference between the second graph structure feature and the first graph structure feature is mainly reflected in: the third feature vector. Among them, each group of third feature vectors includes the semantic vector of a first text block among multiple first text blocks and the semantic vector of a second text block among multiple second text blocks, so that the semantic information of the first text block can be transferred from the retrieved input image to the image to be processed. Among them, in order to keep consistent with the dimensions of the second feature vector and the first feature vector, the third feature vector can also include a position relationship vector. Taking the first text block i and the second text block p as an example, the position relationship vector represents the position relationship between the first text block i and the second text block p. The position relationship vector can be set to a preset value, such as a value of 1, 0, etc.

[0097] For multiple second text blocks, by generating multiple groups of second feature vectors and multiple groups of third feature vectors corresponding to the multiple second text blocks, the second graph structure features composed of the multiple groups of second feature vectors and the multiple groups of third feature vectors can be obtained.

[0098] For ease of understanding, combined Figure 5 and Figure 6 To illustrate the graph structure corresponding to the image to be processed. Figure 6 is the graph structure corresponding to the image to be processed. Assume that the image to be processed includes five second text blocks, namely, second text block 1 to second text block 5. Figure 6 In , the semantic vectors corresponding to the five second text blocks are represented as k1 to k5, respectively, where kp represents the semantic vector corresponding to the second text block p (i.e., the pth second text block). It can be assumed that there is a connection relationship between each second text block and the remaining second text blocks. Thus, for the second text block 1, it is assumed that the connection edges between the second text block 1 (corresponding to k1) and the other four second text blocks are represented as: M12, M13, M14, M15. Figure 6 Only the fully connected relationship between the second text block 1 and the remaining second text blocks is shown. Figure 6 Assume that the second text block 1, i.e., k1, corresponds to the four groups of second feature vectors, which are respectively expressed as: [k1, M12, k2], [k1, M13, k3], [k1, M14, k4], and [k1, M15, k5].

[0099] Combination Figure 5 and Figure 6 To schematically illustrate the multiple sets of third feature vectors corresponding to the multiple second text blocks: Referring to the above description, in Figure 5 In the retrieval, it is assumed that the retrieval input image includes the first text block 1 to the first text block 4, and it is assumed that the semantic information of these four first text blocks are represented as t1 to t4 respectively.

[0100] based on Figure 5 Assume that the semantic information of the first four text blocks is from Figure 5 The retrieval input image is shown to be mapped to Figure 6 In the image to be processed shown, and as Figure 6 As shown, the semantic information of the four first text blocks is still represented as t1 to t4. It can be assumed that there is a connection relationship between each second text block and the four first text blocks. Therefore, for the second text block 1, it is assumed that the connection edges between the second text block 1 (corresponding to k1) and the four first text blocks (corresponding to t1 to t4) are represented as: Z11, Z12, Z13, Z14. Figure 6 Only the fully connected relationship between the second text block 1 and the four first text blocks is illustrated.

[0101] based on Figure 6Assume that the fourth set of third feature vectors corresponding to the second text block 1, i.e., k1, are composed of: [k1, Z11, t1], [k1, Z12, t2], [k1, Z13, t3], [k1, Z14, t4]. In practical applications, Z11, Z12, Z13, and Z14 can be set to preset values, such as 0 or 1.

[0102] The generation process of multiple groups of second feature vectors and multiple groups of third feature vectors corresponding to other second text blocks contained in the image to be processed is consistent with the generation process of multiple groups of second feature vectors and multiple groups of third feature vectors corresponding to the second text block 1, and will not be repeated here.

[0103] Finally, multiple groups of second feature vectors corresponding to multiple second text blocks constitute the second graph structural feature. Alternatively, multiple groups of second feature vectors and multiple groups of third feature vectors corresponding to multiple second text blocks constitute the second graph structural feature.

[0104] It is determined that the first graph structural feature and the second graph structural feature meet a preset condition.

[0105] Output the image to be processed as the search result.

[0106] In this embodiment, the preset condition includes: the difference between the first graph structure feature corresponding to the retrieved input image and the second graph structure feature corresponding to the image to be processed is less than a threshold.

[0107] Specifically, determining that the first graph structure feature and the second graph structure feature meet the preset conditions can be implemented as follows:

[0108] The first graph structure feature is input into the first graph convolutional neural network model to output the encoding vectors corresponding to the multiple first text blocks respectively through the first graph convolutional neural network model; the second graph structure feature is input into the second graph convolutional neural network model to output the encoding vectors corresponding to the multiple second text blocks respectively through the second graph convolutional neural network model; according to the encoding vectors corresponding to the multiple first text blocks respectively and the encoding vectors corresponding to the multiple second text blocks respectively, the difference between the first graph structure feature and the second graph structure feature is determined, and if the difference is less than the set threshold, it is determined that the first graph structure feature and the second graph structure feature meet the preset condition.

[0109] In practical applications, both the first graph convolutional neural network model and the second graph convolutional neural network model can be implemented as a multi-layer structure, which has one or more fully connected layers.

[0110] Combination Figure 5 and Figure 7 To schematically illustrate the process of obtaining the encoding vectors corresponding to the first text blocks: Referring to the above description, in Figure 5In , it is assumed that the semantic vectors corresponding to the four first text blocks, first text block 1 to first text block 4, are represented as t1 to t4 respectively. For first text block 1, it is assumed that the connecting edges between it and the other 13 first text blocks are represented as R12, R13, and R14 respectively. R12, R13, and R14 are the position relationship vectors of first text block 1 relative to the other three first text blocks.

[0111] based on Figure 5 Assumptions such as Figure 7 As shown in , for the first text block 1, the first graph structural feature includes three groups of first feature vectors consisting of three elements as follows: [t1, R12, t2], [t1, R13, t3], [t1, R14, t4]. These three groups of first feature vectors are input into the first graph convolutional neural network model. Through the calculation of the first graph convolutional neural network model, the encoding vector corresponding to the first text block 1 can be obtained for these three groups of first feature vectors, which is assumed to be represented by G t1 . Figure 7 In , each small square represents the encoding result of multiple groups of feature vectors corresponding to a text block, that is, multiple squares represent encoding vectors corresponding to multiple text blocks.

[0112] The calculation process for retrieving the encoding vectors corresponding to other first text blocks contained in the input image is consistent with the calculation process for the encoding vector corresponding to the first text block 1, and will not be repeated here.

[0113] It can be seen that the encoding vectors corresponding to the first text blocks can be obtained through the first graph convolutional neural network model, and then, optionally, the first mean value g of the encoding vectors corresponding to the first text blocks can be determined. t1 , with the first mean g t1 The result of the image feature encoding corresponding to the retrieved input image.

[0114] Similarly, the second graph structure features are input into the second graph convolutional neural network model to output encoding vectors corresponding to multiple second text blocks respectively through the second graph convolutional neural network model.

[0115] Combination Figure 5 , Figure 6 as well as Figure 7 To schematically illustrate the process of obtaining the encoding vectors corresponding to the plurality of second text blocks: Referring to the above description, in Figure 6 In , it is assumed that the semantic vectors corresponding to the five second text blocks, namely, the second text block 1 to the second text block 5, are respectively represented as k1 to k5. For the second text block 1, it is assumed that the connecting edges between it and the other four second text blocks are represented as M12 to M15.

[0116] based on Figure 5 and Figure 6 Assumptions such as Figure 7 As shown in , for the second text block 1, the second graph structure features include the following 4 groups of second feature vectors and 4 groups of third feature vectors. The 4 groups of second feature vectors are expressed as: [k1, M12, k2], [k1, M13, k3], [k1, M14, k4], [k1, M15, k5], and the 4 groups of third feature vectors are expressed as: [k1, Z11, t1], [k1, Z12, t2], [k1, Z13, t3], [k1, Z14, t4]. The 4 groups of second feature vectors and 4 groups of third feature vectors are input into the second graph convolutional neural network model. Through the calculation of the second graph convolutional neural network model, the encoding vector corresponding to the second text block 1 can be obtained for the above third feature vector and second feature vector, which is assumed to be expressed as G k1 .

[0117] The calculation process of the encoding vectors corresponding to other second text blocks contained in the image to be processed is consistent with the calculation process of the encoding vector corresponding to the second text block 1, and will not be repeated herein.

[0118] Furthermore, optionally, the second mean values ​​g of the encoding vectors corresponding to the plurality of second text blocks may be determined: k1 , with the second mean g k1 As the image feature encoding result corresponding to the image to be processed.

[0119] Finally, according to the first mean g t1 and the second mean g k1 Determine the difference between the first graph structural feature and the second graph structural feature.

[0120] The first mean g t1 and the second mean g k1 The smaller the difference between the first and second text blocks, the smaller the difference between the first and second text blocks in terms of overall layout information, semantic relevance of each text block, etc., that is, from the perspective of graph structure, the lower the difference between the first graph structure feature and the second graph structure feature, the lower the difference between the retrieval input image and the image to be processed corresponding to the two graph structures. Therefore, the first mean value g t1 and the second mean g k1 The smaller the difference between the first and second graph structure features, the smaller the difference between the first and second graph structure features, that is, the more similar the retrieval input image and the image to be processed are in terms of graph structure features. t1 and the second mean g k1 The larger the difference value between them, the greater the difference between the first graph structure feature and the second graph structure feature, that is, the less similar the retrieved input image and the image to be processed are in terms of graph structure features.

[0121] Optional, first mean g t1 and the second mean g k1 The difference between them can be measured by Euclidean distance. That is, the first mean g can be used t1 and the second mean g k1 The Euclidean distance between the retrieval input image and the image to be processed is used to represent the difference between the retrieval input image and the image to be processed. Then, the first mean g t1 and the second mean g k1 The calculation formula of the Euclidean distance s between is as follows:

[0122] s(g t1 , g k1 ) = ‖g t1 -g k1 ‖ 2

[0123] Among them, the first mean g t1 and the second mean g k1 The larger the Euclidean distance s is, the greater the difference between the first graph structure feature and the second graph structure feature is; the first mean g t1 and the second mean g k1 The smaller the Euclidean distance s is, the smaller the difference between the structural features of the first graph and the second graph is.

[0124] Based on this, if the Euclidean distance between the first mean and the second mean is less than the threshold, then it can be considered that the image to be processed and the retrieval input image have similarities in graph structure features, that is, it can be determined that the image to be processed matches the retrieval input image, and the image to be processed is output as the search result. Here, matching refers to the matching between the retrieval input image and the image to be processed belonging to the same category. Therefore, by traversing multiple images to be processed in combination with the retrieval input image, images to be processed belonging to the same category as the retrieval input image can be searched out.

[0125] In order to intuitively understand the execution process of the above image search method, Figure 8 The illustrated embodiment is used to exemplarily illustrate how to filter out images to be processed that match a certain retrieval input image from a certain retrieval target database.

[0126] exist Figure 8 In the illustrated embodiment, a practical scenario is assumed: pushing train ticket images that the user wants to view. In this scenario, a search target database is provided in the server, in which a large number of images are stored, including not only many train ticket images, but also images of other categories.

[0127] The user uploads a photographed picture a of a paper train ticket through a PC. The picture a can be used as a search input image. The picture a is annotated to obtain multiple first text blocks. The annotated picture a is transmitted to the server, and the server generates a first graph structure feature based on the multiple first text blocks. At the same time, the server selects a picture b from the search target database as the image to be processed, identifies the picture b to obtain the corresponding multiple second text blocks, and then generates a second graph structure feature based on the multiple second text blocks. The server calculates the difference between the picture a and the picture b based on the first graph structure feature and the second graph structure feature. If the difference is less than the threshold, it can be determined that the picture b is a train ticket image, and the picture b is output to the PC as a search result. Therefore, all pictures matching the picture a can be screened out from the search target database as the final search results.

[0128] Assuming that the above user is a financial staff of a company, the purpose of searching for train ticket images is to count the company's transportation expenses in a certain period of time. Therefore, based on the above search process, all train ticket images in the retrieval target database can be searched out. Then, the user can count the company's transportation expenses in a certain period of time (specifically the cost of traveling by train) based on the time and amount in the searched train ticket images.

[0129] In another optional embodiment, such an actual scenario is also assumed: assessing the risks of real estate loan business. In this scenario, a large number of images are stored in the bank's back-end database, including images of audit documents for various businesses, such as application forms, identity certificates, real estate certificates, and provident fund certificates. In this scenario, the above-mentioned retrieval input image can be a picture c of a real estate loan application form taken by a bank staff member, and the above-mentioned image to be processed can be a picture d selected from the bank's back-end database. Thus, by determining the difference between the graph structure features corresponding to each of the pictures c and d, it can be determined whether the picture d is a picture of a real estate loan application form. If so, the picture d is output as a search result. Based on the above search process, all the real estate loan application forms in the bank's back-end database can be searched out, so that according to the applicant's income, application amount, and repayment method in these searched real estate loan application forms, the handling of the bank's real estate loan business in a certain period of time (such as the applicant's income level and the total application amount) can be analyzed.

[0130] In another optional embodiment, a practical scenario is assumed: a merchant needs to take inventory of certain goods in the store. In this scenario, the merchant can take a picture of the product packaging when each product arrives to obtain the product image corresponding to the product, wherein the product description information needs to be photographed when the product packaging is photographed. In actual applications, the product description information may include the product name, type, origin, production license number, production date, ingredients, QR code or barcode, etc. Based on this, the corresponding relationship between various product images and the sales status of the corresponding products (including inventory and sales record details) can be stored in the management platform. It can be understood that every time a certain product is sold, the inventory of the product will be reduced by one. When the merchant wants to count the inventory of a certain commodity, he can take a photo of the commodity in real time to obtain an image containing the above commodity description information as the retrieval input image. All the above commodity images stored in the management platform constitute the retrieval target database. One image is selected from the retrieval target database as the image to be processed. By identifying the text blocks of the retrieval input image and the image to be processed, the corresponding graph structure features are generated based on the identified text blocks. Therefore, according to the difference between the graph structure features of the retrieval input image and the graph structure features of the image to be processed, it is determined whether the currently selected image to be processed matches the retrieval input image. If they match, it means that the currently selected image to be processed is the commodity image corresponding to the commodity that needs to be counted. At this time, the sales information such as the inventory corresponding to the image to be processed in the management platform can be displayed for the merchant to review.

[0131] The specific process of the image search method in the above embodiment can be understood by referring to the above text.

[0132] In summary, in the image search method provided by the embodiment of the present invention, since multiple images belonging to the same category will show certain similarities in graph structure features, the search for the image to be processed that is similar to the retrieval input image is mainly achieved by comparing the graph structure features corresponding to the retrieval input image and the image to be processed. Since the text content and the positional relationship between the text contents are considered simultaneously in the graph structure features, the accuracy of the search results is effectively guaranteed.

[0133] The previous article introduced the execution process of the image search method. The following briefly introduces the training process of the first graph convolutional neural network model and the second graph convolutional neural network model involved.

[0134] First, a large number of training samples are collected, and the training samples include multiple retrieval input images and multiple sample images that belong to the same category as the multiple retrieval input images. Next, the text content contained in each of the multiple retrieval input images is annotated. Here, in order to distinguish from the multiple first text blocks in the previous text, the annotated results of any retrieval input image are referred to as multiple third text blocks. It can be understood that the annotation method of multiple third text blocks is the same as the annotation method of multiple first text blocks in the previous text. The text content contained in each of the multiple sample images is identified. Here, in order to distinguish from the multiple second text blocks in the previous text, the recognition results of any sample image are referred to as multiple fourth text blocks. It can be understood that the recognition method of multiple fourth text blocks is the same as the recognition method of multiple second text blocks in the previous text. After that, the multiple retrieval input images and the multiple sample images are paired in pairs, and any annotated retrieval input image X and any recognized sample image Y are used as a pair of inputs. The third graph structure feature is generated according to the multiple third text blocks contained in the retrieval input image X, and the fourth graph structure feature is generated according to the multiple fourth text blocks contained in the sample image Y. Here, the method for obtaining the third graph structural feature is consistent with the method for obtaining the first graph structural feature, and the method for obtaining the fourth graph structural feature is consistent with the method for obtaining the second graph structural feature, and no further description is given. Thus, the third graph structural feature is input into the first graph convolutional neural network model, and the fourth graph structural feature is input into the second graph convolutional neural network model, so as to train the first graph convolutional neural network model and the second graph convolutional neural network model.

[0135] In the process of training the first graph convolutional neural network model and the second graph convolutional neural network model, the processing process of each third text block is consistent with the processing process of the first text block i in the previous text, and the processing process of each fourth text block is also consistent with the processing process of the second text block p in the previous text, which will not be repeated.

[0136] For any retrieval input image X and any sample image Y of the paired input, the final output is the prediction result of whether the any sample image Y matches the any retrieval input image X. The prediction result needs to be compared with the label result pre-set for this pair of inputs to determine the loss function of the model.

[0137] In this paper, the loss functions of the first graph convolutional neural network model and the second graph convolutional neural network model are as follows:

[0138]

[0139] Among them, L pair is the loss function, γ is the preset value, G src is the mean of the encoding vectors corresponding to the third text blocks output by the first graph convolutional neural network model, G tgtis the mean of the encoding vectors corresponding to the plurality of fourth text blocks output by the second graph convolutional neural network model, s(G src , G tgt ) is G src With G tgt If any retrieval input image and any sample image correspond to the same category, the value of t is set to 1; if any retrieval input image and any sample image correspond to different categories, the value of t is set to -1. Based on the loss function, the parameters of the first graph convolutional neural network model and the second graph convolutional neural network model are feedback-adjusted until the two models converge.

[0140] In summary, in the model training method provided by the embodiment of the present invention, by using multiple categories of training samples (retrieval input images and sample images), the model can have the ability to learn the correspondence between various categories of retrieval input images and their corresponding sample images in terms of graph structure features. In other words, through this model training method, a general model that can realize image matching processing can be finally obtained, and the image matching refers to the matching between the retrieval input image and its corresponding sample image of the same category. Based on this, even if a certain category of retrieval input images has not been used in the model training stage, if the actual application needs to process the task of searching for images that match the retrieval input image, the model can also be used to implement it. At this time, it is only necessary to define a retrieval input image of this category.

[0141] The image search device of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will appreciate that these image search devices can be configured using commercially available hardware components through the steps taught in this solution.

[0142] Fig. 9 A schematic diagram of the structure of an image search device provided by an embodiment of the present invention is shown in FIG. Fig. 9 As shown, the image search device includes: an acquisition module 11 and a search module 12.

[0143] The acquisition module 11 is used to acquire a search input image, wherein the search input image includes a plurality of first text blocks; and select an image to be processed from a search target database, wherein the image to be processed includes a plurality of second text blocks.

[0144] The search module 12 is used to generate a first graph structure feature according to the multiple first text blocks, and generate a second graph structure feature according to the multiple second text blocks; determine whether the first graph structure feature and the second graph structure feature meet a preset condition; and output the image to be processed as a search result.

[0145] Optionally, in the process of generating the first graph structure feature based on the multiple first text blocks, the search module 12 can be specifically used to: generate semantic vectors corresponding to each of the multiple first text blocks based on the text content contained in each of the multiple first text blocks; generate position relationship vectors corresponding to different first text blocks based on the positional relationship between the multiple first text blocks; generate the first graph structure feature composed of multiple groups of first feature vectors, wherein each group of first feature vectors is composed of a semantic vector corresponding to the first text block i, a semantic vector corresponding to the first text block j, and a position relationship vector between the first text block i and the first text block j, the first text block i is any one of the multiple first text blocks, and the first text block j is any one of the multiple first text blocks except the first text block i.

[0146] Optionally, in the process of generating semantic vectors corresponding to each of the multiple first text blocks based on the text content contained in each of the multiple first text blocks, the search module 12 can be specifically used to: encode the words contained in each of the multiple first text blocks to obtain corresponding word vectors; and calculate the mean of multiple word vectors corresponding to each first text block to obtain the semantic vector corresponding to each first text block.

[0147] Optionally, the corresponding position relationship vector between the first text block i and the first text block j includes at least one of the following ratios: the aspect ratio of the first text block i; the ratio of the width of the first text block j to the height of the first text block i; the ratio of the horizontal distance between the first text block j and the first text block i to the height of the first text block i; the ratio of the vertical distance between the first text block j and the first text block i to the height of the first text block i.

[0148] Optionally, in the process of generating the second graph structure feature according to the plurality of second text blocks, the search module 12 may be specifically configured to: generate the second graph structure feature according to the plurality of first text blocks and the plurality of second text blocks.

[0149] Optionally, in the process of generating the second graph structure feature according to the multiple first text blocks and the multiple second text blocks, the search module 12 can be specifically used to: generate semantic vectors corresponding to each of the multiple second text blocks according to the text content contained in each of the multiple second text blocks; generate position relationship vectors corresponding to different second text blocks according to the positional relationship between the multiple second text blocks; generate the second graph structure feature composed of multiple groups of second feature vectors and multiple groups of third feature vectors, wherein each group of second feature vectors is composed of a semantic vector corresponding to a second text block p, a semantic vector corresponding to a second text block q, and a position relationship vector between the second text block p and the second text block q, the second text block p is any one of the multiple second text blocks, and the second text block q is any one of the multiple second text blocks except the second text block p; each group of third feature vectors includes a semantic vector corresponding to a first text block i and a semantic vector corresponding to a second text block p, and the first text block i is any one of the multiple first text blocks.

[0150] Optionally, the preset condition includes that a difference between the first graph structural feature and the second graph structural feature is less than a set threshold.

[0151] Optionally, in the process of determining whether the first graph structure feature and the second graph structure feature meet preset conditions, the search module 12 can be specifically used to: input the first graph structure feature into the first graph convolutional neural network model to output the encoding vectors corresponding to the multiple first text blocks through the first graph convolutional neural network model; input the second graph structure feature into the second graph convolutional neural network model to output the encoding vectors corresponding to the multiple second text blocks through the second graph convolutional neural network model; determine the difference between the first graph structure feature and the second graph structure feature based on the encoding vectors corresponding to the multiple first text blocks and the encoding vectors corresponding to the multiple second text blocks; if the difference is less than a set threshold, the first graph structure feature and the second graph structure feature meet the preset conditions.

[0152] Optionally, in the process of determining the difference between the first graph structure feature and the second graph structure feature based on the encoding vectors corresponding to each of the multiple first text blocks and the encoding vectors corresponding to each of the multiple second text blocks, the search module 12 can be specifically used to: determine a first mean of the encoding vectors corresponding to each of the multiple first text blocks, and determine a second mean of the encoding vectors corresponding to each of the multiple second text blocks; determine the difference between the first graph structure feature and the second graph structure feature based on the first mean and the second mean.

[0153] Optionally, the device further includes a training module. The training module is used to obtain multiple retrieval input images and multiple sample images belonging to the same category as the multiple retrieval input images; annotate multiple third text blocks contained in each of the multiple retrieval input images; identify multiple fourth text blocks contained in each of the multiple sample images; take any retrieval input image and any sample image as a pair of inputs, generate a third graph structure feature according to the multiple third text blocks, generate a fourth graph structure feature according to the multiple third text blocks and the multiple fourth text blocks, and input the third graph structure feature and the fourth graph structure feature into the first graph convolutional neural network model and the second graph convolutional neural network model, respectively, to train the first graph convolutional neural network model and the second graph convolutional neural network model.

[0154] The loss functions of the first graph convolutional neural network model and the second graph convolutional neural network model are:

[0155]

[0156] Among them, L is the loss function, γ is the preset value, G src is the mean of the encoding vectors corresponding to the plurality of third text blocks output by the first graph convolutional neural network model, G tgt is the mean of the encoding vectors corresponding to the plurality of fourth text blocks output by the second graph convolutional neural network model, s(G src , G tgt ) is G src With G tgt distance; if any of the retrieval input images and any of the sample images correspond to the same category, then t=1; if any of the retrieval input images and any of the sample images correspond to different categories, then t=-1.

[0157] Fig. 9 The image search device shown can execute the methods provided in the aforementioned embodiments. For the parts not described in detail in this embodiment, reference can be made to the relevant descriptions of the aforementioned embodiments, which will not be repeated here.

[0158] In one possible design, the above Fig. 9 The structure of the image search device shown can be implemented as an electronic device. Fig.10 As shown, the electronic device may include: a processor 21 and a memory 22. The memory 22 stores executable codes, and when the executable codes are executed by the processor 21, the processor 21 can at least implement the image search method provided in the above-mentioned embodiment.

[0159] The structure of the electronic device may also include a communication interface 23 for communicating with other devices or a communication network.

[0160] In addition, an embodiment of the present invention provides a non-transitory machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of a wireless router, the processor executes the image search method provided in the aforementioned embodiments.

[0161] The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0162] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution can essentially or contribute to the part in the form of a computer product, and the present invention can be implemented in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image search method, characterized in that: include: Acquire a search input image, wherein the search input image includes a plurality of first text blocks, and the plurality of first text blocks correspond to fixed texts that are irrelevant to a user; Selecting an image to be processed from a search target database, wherein the image to be processed includes a plurality of second text blocks; Generating a first graph structure feature according to the plurality of first text blocks, and generating a second graph structure feature according to the plurality of second text blocks, wherein the graph structure feature includes semantic features corresponding to each of the plurality of text blocks, and positional relationship features between the plurality of text blocks; Determining that the first graph structural feature and the second graph structural feature meet a preset condition; The image to be processed is output as a search result, and the retrieved input image and the image to be processed belong to the same category.

2. The method according to claim 1, characterized in that The generating a first graph structure feature according to the plurality of first text blocks comprises: Generating semantic vectors corresponding to each of the plurality of first text blocks according to text contents respectively contained in the plurality of first text blocks; Generating corresponding position relationship vectors between different first text blocks according to the position relationship between the plurality of first text blocks; Generate the first graph structure feature composed of multiple groups of first feature vectors, wherein each group of first feature vectors is composed of a semantic vector corresponding to a first text block i, a semantic vector corresponding to a first text block j, and a position relationship vector between the first text block i and the first text block j, the first text block i is any one of the multiple first text blocks, and the first text block j is any one of the multiple first text blocks except the first text block i.

3. The method according to claim 2, characterized in that The step of generating semantic vectors corresponding to each of the plurality of first text blocks according to the text contents respectively contained in the plurality of first text blocks comprises: Encoding the words contained in each of the plurality of first text blocks to obtain a corresponding word vector; An average calculation is performed on multiple word vectors corresponding to each first text block to obtain a semantic vector corresponding to each first text block.

4. The method according to claim 2, characterized in that: The step of generating corresponding position relationship vectors between different first text blocks according to the position relationship between the plurality of first text blocks comprises: For the first text block i and the first text block j, determining the corresponding position relationship vector between the first text block i and the first text block j includes at least one of the following ratios: The aspect ratio of the first text block i; a ratio of the width of the first text block j to the height of the first text block i; The ratio of the horizontal distance between the first text block j and the first text block i to the height of the first text block i; The ratio of the vertical distance between the first text block j and the first text block i to the height of the first text block i.

5. The method according to claim 1, characterized in that The generating a second graph structure feature according to the plurality of second text blocks comprises: A second graph structure feature is generated according to the plurality of first text blocks and the plurality of second text blocks.

6. The method according to claim 5, characterized in that The generating a second graph structure feature according to the plurality of first text blocks and the plurality of second text blocks comprises: Generating semantic vectors corresponding to each of the plurality of second text blocks according to text contents respectively contained in the plurality of second text blocks; Generating corresponding position relationship vectors between different second text blocks according to the position relationship between the plurality of second text blocks; Generate the second graph structure feature consisting of multiple groups of second feature vectors and multiple groups of third feature vectors, wherein each group of second feature vectors consists of a semantic vector corresponding to a second text block p, a semantic vector corresponding to a second text block q, and a position relationship vector between the second text block p and the second text block q, the second text block p is any one of the multiple second text blocks, and the second text block q is any one of the multiple second text blocks except the second text block p; each group of third feature vectors includes a semantic vector corresponding to a first text block i and a semantic vector corresponding to the second text block p, and the first text block i is any one of the multiple first text blocks.

7. The method according to claim 1, characterized in that The preset condition includes that the difference between the first graph structural feature and the second graph structural feature is less than a set threshold.

8. The method according to claim 7, characterized in that The determining that the first graph structural feature and the second graph structural feature meet a preset condition includes: Inputting the first graph structure feature into a first graph convolutional neural network model, so as to output encoding vectors corresponding to each of the plurality of first text blocks through the first graph convolutional neural network model; Inputting the second graph structure feature into a second graph convolutional neural network model, so as to output encoding vectors corresponding to each of the plurality of second text blocks through the second graph convolutional neural network model; Determining the difference between the first graph structure feature and the second graph structure feature according to the encoding vectors corresponding to each of the plurality of first text blocks and the encoding vectors corresponding to each of the plurality of second text blocks; If the difference is less than a set threshold, the first graph structural feature and the second graph structural feature satisfy the preset condition.

9. The method according to claim 8, characterized in that The determining, according to the encoding vectors corresponding to each of the plurality of first text blocks and the encoding vectors corresponding to each of the plurality of second text blocks, the difference between the first graph structural feature and the second graph structural feature comprises: Determine a first mean value of the encoding vectors corresponding to each of the plurality of first text blocks, and determine a second mean value of the encoding vectors corresponding to each of the plurality of second text blocks; The difference between the first graph structural feature and the second graph structural feature is determined according to the first mean and the second mean.

10. The method according to claim 1, characterized in that The method further comprises: Acquire a plurality of search input images and a plurality of sample images that belong to the same category as the plurality of search input images; marking a plurality of third text blocks respectively contained in the plurality of search input images; identifying a plurality of fourth text blocks respectively contained in the plurality of sample images; Taking any retrieval input image and any sample image as a pair of inputs, generating a third graph structure feature according to the multiple third text blocks, generating a fourth graph structure feature according to the multiple fourth text blocks, and inputting the third graph structure feature and the fourth graph structure feature into the first graph convolutional neural network model and the second graph convolutional neural network model respectively, so as to train the first graph convolutional neural network model and the second graph convolutional neural network model.

11. The method according to claim 10, characterized in that The loss functions of the first graph convolutional neural network model and the second graph convolutional neural network model are: Among them, L is the loss function, γ is the preset value, G src is the mean of the encoding vectors corresponding to the plurality of third text blocks output by the first graph convolutional neural network model, G tgt is the mean of the encoding vectors corresponding to the plurality of fourth text blocks output by the second graph convolutional neural network model, s(G src , G tgt ) is G src With G tgt distance; if any of the retrieval input images and any of the sample images correspond to the same category, then t=1; if any of the retrieval input images and any of the sample images correspond to different categories, then t=-1.

12. An image search device, characterized in that: include: An acquisition module is used to acquire a search input image, wherein the search input image includes a plurality of first text blocks, and the plurality of first text blocks correspond to fixed texts irrelevant to the user; and select an image to be processed from a search target database, wherein the image to be processed includes a plurality of second text blocks; A search module is used to generate a first graph structure feature according to the multiple first text blocks, and to generate a second graph structure feature according to the multiple second text blocks, wherein the graph structure feature includes semantic features corresponding to each of the multiple text blocks, and positional relationship features between the multiple text blocks; determine whether the first graph structure feature and the second graph structure feature meet a preset condition; and output the image to be processed as a search result, wherein the retrieval input image and the image to be processed belong to the same category.

13. An electronic device, characterized in that: include: A memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the image search method as described in any one of claims 1 to 11.

Images