Topic image searching method, device and equipment and readable storage medium

Abstract

The invention provides a question image searching method, device and equipment and a readable storage medium, and the method comprises the steps: carrying out the segmentation processing of an input question image, recalling a primarily selected question image through a feature vector, preliminarily screening out possibly related images, determining a reference region and a non-reference region based on the structural layout of a question in a candidate question image, and carrying out the searching of the question image through the reference region and the non-reference region; the layout similarity is determined by using the relative distance between the region pairs when the candidate question image and the region pairs are mapped to the input question image, and the image similarity between the candidate question image and the input question image is determined by integrating the layout similarity of each region pair, so that a target question image is determined; image search is performed by analyzing the image content features of the image and the structural layout features in the image, character recognition is not depended on, topic images of different languages can be directly processed and matched, unified search of topics of different languages is achieved, the image search accuracy is improved, and the image search efficiency is improved. And the user can quickly acquire the target question image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a question image search method and device, equipment and a readable storage medium. BACKGROUND

[0002] In a question image search scenario, the search demand of users for various types of questions has increased rapidly, and it is crucial to quickly obtain accurate information. At present, due to the diversity and complexity of question types and question languages, the recognition accuracy of different languages and characters is greatly different when using traditional optical character recognition technology for question search, and the recognition effect is poor. When the question text is in handwritten or printed form and the text is not clear, the recognition difficulty is further increased, which greatly reduces the efficiency of question search based on optical character recognition technology for recognizing question content, and it is difficult to meet the demand of users for quickly and accurately obtaining question titles. SUMMARY

[0003] Therefore, in order to solve the above technical problems, the present application provides a question image search method, device, equipment and readable storage medium.

[0004] Specifically, the present application is realized by the following technical solutions:

[0005] According to a first aspect of the embodiments of the present application, a question image search method is provided, and the method comprises:

[0006] The input question image to be searched is divided to obtain N divided images, and based on the feature vector of the input question image and the feature vector corresponding to each divided image, relevant question images are recalled from the retrieval library respectively to obtain preliminary selected question images;

[0007] A plurality of images matching the input question image are preliminarily selected from the preliminary selected question images as candidate question images;

[0008] For each region pair in each candidate question image, the first relative distance of the two regions in the region pair in the candidate question image is determined, and the second relative distance of the two regions in the region pair in the candidate question image when mapped to the input question image is determined, and the layout similarity of the region pair is determined according to the first relative distance and the second relative distance; wherein any region pair includes a designated reference region and any non-reference region in the candidate question image; the reference region and the non-reference region are determined based on the structural layout of the question included in the candidate question image;

[0009] Based on the layout similarity of each region pair in the candidate question image, the image similarity between the candidate question image and the input question image is determined;

[0010] Based on the image similarity between each candidate topic image and the input topic image, a target topic image searched for the input topic image is determined from the plurality of candidate topic images.

[0011] Optionally, based on the feature vector of the input question image and the feature vectors corresponding to each segmentation graph, relevant question images are respectively recalled from a search library to obtain preliminary question images, including:

[0012] For each feature vector, a set of related topic images whose vector similarity with the feature vector meets the requirements is retrieved from the retrieval library as a related set;

[0013] For each recalled relevant topic image, a recall score of the relevant topic image is determined based on whether the relevant topic image exists in the relevant set corresponding to each feature vector; wherein the recall score is determined based on the weight score of the relevant topic image in each relevant set; for each relevant set, if the relevant topic image exists in the relevant set, the weight score of the relevant topic image in the relevant set is a first value; otherwise, the weight score of the relevant topic image in the relevant set is a second value;

[0014] Based on the recall score of each relevant topic image, multiple relevant topic images are screened out from the relevant set corresponding to each feature vector as preliminary topic images.

[0015] Optionally, the first value and the second value of the weight score of the relevant topic image in the relevant set are determined by:

[0016] Based on the importance of the relevant set, a value pair of weight scores of the relevant topic image in the relevant set is set for each relevant set, wherein the value pair of weight scores includes a first value and a second value;

[0017] The importance of the related set includes:

[0018] The importance of the related set based on the feature vector recall of the input question image is higher than the related set based on the feature vector recall corresponding to each cut graph; the importance of the related set based on the feature vector recall corresponding to each cut graph is positively correlated with the size of the cut graph.

[0019] Optionally, preliminarily screening out a plurality of images matching the input topic image from the preliminarily selected topic images as candidate topic images includes:

[0020] Divide the input question image and each preliminary question image into a grid image including a specified number of grids according to the same grid division method;

[0021] For each preliminary question image, traverse the multiple matched feature points successfully matched between the preliminary question image and the input question image, and detect whether the orientation of the grid where the matched feature point is located on the input question image is consistent with the orientation of the grid where the matched feature point is located on the preliminary question image. If yes, the matched feature point is determined as an effective feature point, and the number of effective feature points corresponding to the preliminary question image is obtained.

[0022] Based on the number of effective feature points corresponding to each preliminary question image, multiple preliminary question images matched with the input question image are determined as candidate question images.

[0023] Optionally, the feature point matching between the preliminary question image and the input question image is implemented by the following method:

[0024] The feature point extraction algorithm is used to extract feature points from the to-be-processed image, and the first M feature points with the largest feature point intensity are selected from the multiple feature points extracted by measuring the feature point intensity index as the to-be-matched feature points of the to-be-processed image. The feature point intensity index corresponds to the feature point extraction algorithm used. The to-be-processed image includes the input question image and each preliminary question image.

[0025] The to-be-matched feature points of each preliminary question image are matched with the to-be-matched feature points of the input question, and multiple matched feature points corresponding to the preliminary question image are obtained.

[0026] Optionally, before determining the second relative distance between the two regions in the region pair in the candidate question image when the two regions are mapped to the input question image, the following steps are included:

[0027] For multiple specified regions in the candidate question image, the position information of each specified region mapped to the input question image is determined. The multiple specified regions include at least one reference region and at least one non-reference region.

[0028] The determination of the second relative distance between the two regions in the region pair in the candidate question image when the two regions are mapped to the input question image includes:

[0029] Based on the position information of the two regions in the region pair mapped to the input question image, the relative distance between the two regions is determined.

[0030] Optionally, the determination of the position information of each specified region mapped to the input question image includes:

[0031] The position information of each specified region in the candidate question image is determined, the position information is represented by at least one coordinate point, and the association relationship between the coordinate point representing the position information and the feature point in the candidate question image is established.

[0032] Based on the feature points in the candidate question image and the feature points in the input question image, a matching feature point pair is determined through feature point matching;

[0033] Based on the matching feature point pair and the association relationship, a coordinate point representing the position information of each specified region on the candidate question image is mapped to a coordinate point on the input question image, and the position information of each specified region mapped to the input question image is obtained.

[0034] Optionally, the first relative distance / second relative distance is determined in the following manner:

[0035] The first coordinate difference and the second coordinate difference of the position information of the non-reference region relative to the position information of the reference region in the region pair are determined in a first direction and a second direction; the first direction represents the width direction of the image where the non-reference region is located; and the second direction represents the height direction of the image where the non-reference region is located;

[0036] The first coordinate difference and the second coordinate difference are normalized, and the corresponding relative distance of the region pair is determined based on the normalized first coordinate difference and second coordinate difference;

[0037] Wherein, when the image where the non-reference region is located is the candidate question image, the corresponding relative distance of the region pair is the first relative distance; and when the image where the non-reference region is located is the input question image, the corresponding relative distance of the region pair is the second relative distance.

[0038] According to a second aspect of the embodiments of the present application, a question image search device is provided, and the device comprises:

[0039] A feature vector recall module is configured to perform segmentation processing on an input question image to be searched to obtain N segmented images, and based on the feature vector of the input question image and the feature vectors corresponding to the segmented images, recall relevant question images from a retrieval library respectively to obtain preliminary selected question images.

[0040] A preliminary screening module is configured to preliminarily screen multiple images matched with the input question image from the preliminary selected question images as candidate question images.

[0041] The layout similarity screening module is configured to determine, for each region pair in each candidate question image, a first relative distance between two regions in the region pair in the candidate question image, and a second relative distance between the two regions in the region pair in the candidate question image when the two regions are mapped to the input question image, and determine a layout similarity of the region pair according to the first relative distance and the second relative distance; any region pair includes a designated reference region and any non-reference region in the candidate question image; the reference region and the non-reference region are determined based on a structural layout of a question included in the candidate question image; the image similarity between the candidate question image and the input question image is determined based on the layout similarities of the region pairs in the candidate question image; the title question image searched for the input question image is determined from the plurality of candidate question images based on the image similarity of each candidate question image and the input question image.

[0042] Optionally, the feature vector retrieval module at least includes:

[0043] The retrieval module is configured to retrieve, for each feature vector, a set of relevant question images from the search library as a relevant set, where the relevant question images have a vector similarity with the feature vector satisfying a requirement.

[0044] The retrieval statistics module is configured to determine, for each relevant question image, a retrieval score of the relevant question image according to whether the relevant question image exists in the relevant set corresponding to each feature vector; the retrieval score is determined based on a weight score of the relevant question image in each relevant set; for each relevant set, if the relevant question image exists in the relevant set, the weight score of the relevant question image in the relevant set is a first value, otherwise the weight score of the relevant question image in the relevant set is a second value.

[0045] The retrieval screening module is configured to screen, based on the retrieval score of each relevant question image, a plurality of relevant question images from the relevant set corresponding to each feature vector as the preliminary selected question image.

[0046] Optionally, the first value and the second value of the weight score of the relevant question image in the relevant set are determined by the following method:

[0047] Based on the importance of the relevant set, a value pair of the weight score of the relevant question image in the relevant set is set for each relevant set, the value pair including the first value and the second value; the importance of the relevant set includes: the importance of the relevant set based on the feature vector retrieval of the input question image is higher than the importance of the relevant set based on the feature vector retrieval of each segmentation graph; the importance of the relevant set based on the feature vector retrieval of each segmentation graph is positively correlated with the size of the segmentation graph.

[0048] Optionally, the preliminary screening module at least includes:

[0049] A grid division module is used to divide the input question image and each preliminary question image into grid images including a specified number of grids according to the same grid division method;

[0050] A feature point detection module is used to, for each preliminary question image, traverse a plurality of successfully matched feature points determined by feature point matching between the preliminary question image and the input question image, and detect whether the grid where the successfully matched feature point is located on the input question image is consistent with the grid where it is located on the preliminary question image. If so, the successfully matched feature point is determined to be a valid feature point, and the number of valid feature points corresponding to the preliminary question image is obtained;

[0051] The candidate question image screening module is used to determine multiple preliminary question images that match the input question image as candidate question images based on the number of valid feature points corresponding to each preliminary question image.

[0052] Optionally, the feature point matching between the preliminary question image and the input question image is achieved by:

[0053] Extracting feature points from the image to be processed using a feature point extraction algorithm, and selecting the top M feature points with the largest feature point intensities from the extracted multiple feature points using an index for measuring the strength of the feature points as the feature points to be matched for the image to be processed; wherein the index for measuring the strength of the feature points corresponds to the feature point extraction algorithm used; the image to be processed includes the input question image and each of the preliminarily selected question images;

[0054] The feature points to be matched of each preliminary question image are matched with the feature points to be matched of the input question to obtain a plurality of successfully matched feature points corresponding to the preliminary question image.

[0055] Optionally, the device further comprises:

[0056] a mapping module, configured to determine, for a plurality of designated areas in the candidate question image, location information of each designated area mapped to the input question image; the plurality of designated areas including at least one reference area and at least one non-reference area;

[0057] The layout similarity screening module, when used to determine the second relative distance between two regions in the region pair in the candidate question image when mapped to the input question image, includes: determining the relative distance between the two regions based on position information of the two regions in the region pair mapped to the input question image.

[0058] Optionally, the mapping module includes:

[0059] The position information determining and associating module is configured to determine position information of each specified region on the candidate question image, and establish an association between a coordinate point representing the position information and a feature point in the candidate question image.

[0060] The feature point pair obtaining module is configured to determine a matching feature point pair by feature point matching based on a feature point in the candidate question image and a feature point in the input question image.

[0061] The coordinate mapping module is configured to determine a coordinate point representing the position information of each specified region on the candidate question image that is mapped to a coordinate point on the input question image based on the matching feature point pair and the association, and obtain position information of each specified region mapped to the input question image.

[0062] Optionally, the first relative distance / second relative distance is determined by the following method:

[0063] The first coordinate difference and the second coordinate difference of the position information of the non-reference region relative to the position information of the reference region in the region pair are determined in a first direction and a second direction, respectively; the first direction represents a width direction of the image in which the non-reference region is located; and the second direction represents a height direction of the image in which the non-reference region is located.

[0064] The first coordinate difference and the second coordinate difference are normalized, and the corresponding relative distance of the region pair is determined based on the normalized first coordinate difference and the normalized second coordinate difference.

[0065] When the image in which the non-reference region is located is the candidate question image, the corresponding relative distance of the region pair is the first relative distance; and when the image in which the non-reference region is located is the input question image, the corresponding relative distance of the region pair is the second relative distance.

[0066] According to a third aspect of the embodiments of the present application, an electronic device is provided, which includes a memory and a processor; the memory is configured to store a computer program; and the processor is configured to execute the above question image search method by calling the computer program.

[0067] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program, and the program is executed by a processor to implement the above question image search method.

[0068] The technical scheme provided by the embodiments of the present application can have the following beneficial effects:

[0069] In the technical solutions provided in the application, the input question image is first divided and processed, and a feature vector is used to recall the initial question image, so that the images possibly related are preliminarily screened out, the data amount of subsequent processing is reduced, and the layout similarity is further calculated in the candidate question image to accurately match, so that the invalid search in a large number of irrelevant images is avoided, and the search efficiency is improved. The image search is performed by analyzing the image content features and the structure layout features of the image, does not depend on the character recognition, can directly process and match the question images in different languages, realizes the unified search of the question images in different languages, and enables the user to quickly obtain the question image.

[0070] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the application. In addition, any embodiment in the application does not need to achieve all the effects described above. BRIEF DESCRIPTION OF DRAWINGS

[0071] The accompanying drawings, which are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the application and, together with the specification, serve to explain the principles of the application.

[0072] Figure 1A is a question image search method flow diagram shown by an exemplary embodiment of the application;

[0073] Figure 1B is a diagram showing the division and processing of an input question image according to an exemplary embodiment of the application;

[0074] Figure 1C is a diagram showing a plurality of specified regions marked in a candidate question image according to an exemplary embodiment of the application;

[0075] Figure 2 is an initial question image determination flow diagram based on cumulative weight score according to an exemplary embodiment of the application;

[0076] Figure 3A is a step flow diagram for screening the candidate question image according to an exemplary embodiment of the application;

[0077] Figure 3B is a diagram showing the grid orientation of the effective feature points when an input question image and an initial question image are divided into grid images according to an exemplary embodiment of the application;

[0078] Figure 4 is a step flow diagram for calculating the relative distance of two regions in an image according to an exemplary embodiment of the application;

[0079] Figure 5is a flow chart of a subject image search method according to an example embodiment of the present application;

[0080] Figure 6 is a structural schematic diagram of a subject image search device according to an example embodiment of the present application;

[0081] Figure 7 is a hardware schematic diagram of an electronic device according to an example embodiment of the present application. DETAILED DESCRIPTION

[0082] The example embodiments will be described in detail herein with reference to the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following example embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims. It should be understood that although the terms first, second, third, etc. can be used herein to describe various information, these terms are not intended to denote a particular order of the information. These terms are used merely to distinguish one type of information from another.

[0083] In the subject image search scenario, the user's search demand for various types of subjects has surged, and it is crucial to quickly obtain accurate information. The traditional text search method mainly relies on optical character recognition technology, whose principle is to convert the text in the image into editable text, and then search and match based on the text. However, there are significant differences in the recognition accuracy of different languages, and existing optical character recognition technology is difficult to achieve efficient and accurate recognition. At the same time, the randomness of handwritten subject text, the unclearness of printed text, and other situations further increase the difficulty of text recognition.

[0084] In the subject image search scenario, the text search method based on OCR technology has large differences in recognition accuracy of different languages, poor recognition effect, and increased difficulty in recognizing unclear handwritten or printed text, resulting in a significant reduction in search efficiency based on OCR technology to recognize subject content, making it difficult for users to quickly and accurately obtain the subject, requiring a lot of time to filter effective information in the search results, and possibly failing to find the required subject due to recognition errors, affecting user experience and learning.

[0085] Therefore, the present application provides a subject image search method, which, for an input subject image to be searched, searches the image from the overall and local image content features, combines the image layout and image structure dimension information, and searches the image without considering language barriers, effectively solving the cross-language search problem, improving search accuracy, and enhancing user experience.

[0086] Referring to Figure 1A An exemplary step flow chart of a subject image search method can include at least the following steps:

[0087] S101, performing segmentation processing on an input subject image to be searched to obtain N segmented images, and recalling relevant subject images from a retrieval library based on a feature vector of the input subject image and a feature vector corresponding to each segmented image, to obtain preliminary subject images;

[0088] The input subject image to be searched represents a user input original image that needs to be searched, which can include a user real-time photographed subject image, such as a user directly photographing a subject using a mobile phone, tablet computer, digital camera, or the like, or an existing subject image file selected and uploaded from a photo library, and the like.

[0089] Segmentation processing is used to divide an image as a whole into multiple local images that constitute the image as a whole. Different segmentation strategies can be selected based on segmentation requirements, such as image segmentation according to dimensions such as size, content, structure, and the like. In this embodiment, in the process of segmenting the input subject image to obtain N segmented images, the input subject image can be segmented into multiple segmented images using one segmentation processing method or multiple segmentation processing methods, each segmentation processing method corresponding to a group of segmented images, and the number of segmented images corresponding to different segmentation processing methods can be the same or different.

[0090] Based on this, in this embodiment, when segmenting the input subject image to be searched to obtain N segmented images, the input subject image can be segmented into two sub-images, and the input subject image can be segmented into Q fixed-size sub-images (Q is greater than 2) as the N segmented images obtained by segmentation processing. When segmented into two sub-images, for example, the input subject image can be segmented into an upper half image and a lower half image according to one-half of the height of the input subject image, or the proportion of the two sub-images can be flexibly adjusted according to other segmentation ratios. Regarding segmentation of the input subject image into Q fixed-size sub-images, the input subject image can be segmented into equal-size sub-images after preprocessing such as scaling to a fixed size, image data standardization, and the like. For example, the input subject image can be scaled to a scaled image with a width of 640, the height of the scaled image can be processed to a multiple of 64, the width remains 640, the input subject image with a height and width of 64 multiples is segmented according to 64*64, and a plurality of segmented images with a size of 64*64 are obtained. For example, as shown in Figure 1B An exemplary schematic diagram of segmentation processing of an input subject image is shown. An input subject image P is segmented into an upper half image P_01 and a lower half image P_02, and the input subject image P is segmented into nine fixed-size segmented images P_11 to P_19.

[0091] The feature vector of the input question image and the feature vectors corresponding to each split image can be obtained by a pre-trained feature vector extraction model. The input question image and the split image can use the same feature vector extraction model or a separate feature vector extraction model for the split image. For example, the input question image can be input into the trained feature vector extraction model for forward reasoning, and a fixed dimension feature vector such as 1*576 is output. For the split image, each split image can be input into the trained split image feature vector extraction model, and a fixed dimension feature vector is output for each split image. For example, each split image outputs a 1*64 dimension feature vector. Other dimensions of the feature vector can also be used in actual application. The model reasoning of the split image is performed in a batch mode to save the model reasoning time. The feature vector extraction model used in this embodiment can be a MobileNetV3Small model, and other models can also be used in actual application, which is not limited in the present application.

[0092] The retrieval library is a database containing a large number of question images and their related feature vector information. For each question image in the library, the feature vector of the question image itself and the feature vectors of each split image obtained by various split processing of the question image are stored. In this embodiment, the split processing method used when the input question image is split is a subset of the various split processing methods used to obtain the split images of the question images in the retrieval library. Therefore, when a set of related question images is recalled from the retrieval library using the feature vector of each split image of the input question image, for each question image in the retrieval library, the feature vector of the split image in the question image that maintains the same position as the split image to which the feature vector belongs in the input question image can be calculated for similarity, and accurate recall can be achieved.

[0093] For the split image in the question image in the retrieval library that maintains the same position as the split image to which the feature vector belongs in the input question image, for example, the input question image is split into Q split images, and the corresponding question image in the retrieval library is also split into Q split images by the same split processing method. The positions (i.e. orientations) of the Q split images in the input question image and the corresponding question image in the retrieval library are one-to-one and consistent. For example, the input question image is split into four split images in the order from top to bottom and from left to right, which are split image 1, split image 2, split image 3 and split image 4. Figure 2 Figure 4 Therefore, when the feature vector of the split image 1 is used to recall from the retrieval library, the feature vector of the split image corresponding to the split image 1 in the four split images of each question image in the retrieval library is calculated for similarity with the feature vector of the split image 1 to achieve retrieval recall.​

[0094] In the embodiment, the same search library can be used to store the feature vectors of the question image itself and the feature vectors of each split image obtained by various split processing of the question image. In the storage process, different fields can be used to distinguish and control the feature vectors of different types (such as the feature vector of the whole question image and the feature vector of the split image under different split modes) so as to be accurately recognized and called subsequently. Alternatively, in actual application, the feature vectors of the question image itself and the feature vectors of each split image obtained by various split processing of the question image can be stored separately according to the size of the question library and the number of access users.

[0095] The relevant question image is recalled according to the input feature vector to find the question image with a similar vector in the search library. Based on this, the relevant question image is recalled from the search library based on the feature vector of the input question image and the feature vector of each split image, which is used as a query condition independently, and the similarity between the query condition (i.e. the feature vector) and the specified feature vector of each question image in the search library is calculated using a similarity measurement method such as cosine similarity, Euclidean distance, etc., and then a group of relevant question images is recalled based on the similarity. The specified feature vector of the question image refers to the feature vector of the image that maintains the same position as the image in the input question image to which the query condition belongs among the feature vectors corresponding to all images of the question image (including the question image itself and each split image obtained by various split processing).

[0096] For example, when the relevant question image is recalled from the search library based on the feature vector of the input question image, the similarity between the feature vector and the whole feature vector of each question image in the search library for representing the feature of the question image itself is calculated, and a group of relevant question images with a similarity meeting the requirement is recalled. When the relevant question image is recalled from the search library based on the feature vector of each split image corresponding to the input question image, the feature vector of each split image is calculated with the feature vector of each split image in the search library that belongs to the same split mode and maintains the same position as the split image, and a group of relevant question images with a similarity meeting the requirement is recalled. For the input question image and N split images, N+1 groups of relevant question images are recalled.

[0097] As to the requirement of similarity, different similarity thresholds can be set for different feature vectors to meet different feature vector recall requirements. For example, the feature vector based on the input question image itself contains all the information of the question, including text, graphics and layout, etc., which can provide more extensive context information, while the segmented image only contains part of the information and has less information, so when setting the similarity threshold, the following rules can be based on: the similarity threshold of the feature vector based on the input question image is lower than the similarity threshold of the feature vector based on the segmented image; the similarity threshold of the feature vector based on the segmented image is inversely proportional to the size of the segmented image, that is, the larger the segmented image, the lower the corresponding similarity threshold, and the smaller the segmented image, the higher the corresponding similarity threshold, so as to increase the relevance of the recall results when the smaller segmented image contains limited information segments and reduce noise interference. In this way, the similarity threshold can be flexibly adjusted according to the characteristics of different feature vectors, improving the accuracy of the overall recall, covering more potential related question images while ensuring accuracy.

[0098] The preliminary selected question image is not a single retrieval recall result, but is obtained by merging the retrieval recall results of each feature vector (i.e. a set of related question images). As to the merging process, all related question images recalled from different feature vectors can be collected to form a large image set, and since there may be repeated recall (i.e. the same question image is included in the recall results of multiple feature vectors), image deduplication processing is performed on the large image set, such as determining whether the image is repeated by the unique identifier of the image and removing the repeated items, and the question images in the image set after deduplication processing are taken as the preliminary selected question images. The preliminary selected question images obtained in this way are the union of all feature vector recall results and remove the repeated items.

[0099] Alternatively, in order to improve the retrieval efficiency, different weights can also be assigned to the recall results of each feature vector, and the weight is used to reflect the influence of different feature vectors on the recall results, and part of the related question images are selected from all the recall results as the preliminary selected question images based on the cumulative weight of each recalled related question image, such as selecting multiple question images whose cumulative weight meets the weight threshold, so as to reduce the number of intermediate images in the image retrieval process while ensuring accuracy, speeding up the image search, which will be described in subsequent embodiments.

[0100] S102, preliminarily selecting multiple images matching the input question image from the preliminary selected question image as candidate question images;

[0101] The step in this embodiment is used to quickly narrow down the range from a large number of primary question images, find the images most similar in content, structure or features to the input question image, and then perform more accurate matching analysis.

[0102] Based on this, the preliminary screening process can be realized by global feature matching, deep learning model matching, structural similarity matching, hash algorithm matching, etc. ① Global feature matching is a similarity detection based on the overall features of the image. For example, the feature vector of each primary question image for describing the overall content features of the primary question image can be matched with the feature vector of the input question image, and multiple images are quickly screened out as candidate question images based on the feature vector matching degree; ② Deep learning model matching refers to using a trained deep learning model (such as a convolutional neural network CNN) to extract features and classify images, and calculate the similarity between the input question image and the primary question image, and then perform preliminary screening based on the similarity; ③ Structural similarity matching is an index for measuring the structural similarity of two images, which considers the brightness, contrast and structural information of the image, and can more comprehensively evaluate the similarity between images; ④ Hash algorithm matching realizes image matching by converting the primary question image and the input question image into fixed-length hash values and comparing the similarity between the hash values. Common hash algorithms include average hash, perceptual hash and difference hash, etc.

[0103] In addition, the application also provides a preliminary screening method based on feature point matching, which extracts representative feature points from the primary question image and the input question image, evaluates the similarity between them based on the number and quality of the matched feature point pairs in the primary question image and the input question image, and performs fast screening based on the similarity. See the subsequent embodiments for details.

[0104] S103, for each region pair in each candidate question image, determining a first relative distance between the two regions in the region pair in the candidate question image, and determining a second relative distance when the two regions in the region pair in the candidate question image are mapped to the input question image, and determining the layout similarity of the region pair according to the first relative distance and the second relative distance; wherein any region pair includes a designated reference region and any non-reference region in the candidate question image; the reference region and the non-reference region are determined based on the structural layout of the question included in the candidate question image;

[0105] Based on the fact that the candidate question image includes one or more question information, the embodiment labels a plurality of specified regions on the candidate question image according to the structural layout of the question in the candidate question image, and takes at least one specified region as a reference region, and forms a plurality of region pairs [reference region, non-reference region] from any reference region and other regions in the plurality of specified regions except the reference region, and the reference region is used as a reference for the positional relationship of other regions. Among them, the region type of the plurality of specified regions labeled on the candidate question image can include at least one of the stem region of the question in the candidate question image and the answer region. That is, each region pair in the candidate question image can only consist of two stem regions, or each region pair can only consist of two answer regions, or the region pair in the candidate question image can consist of a stem region and an answer region.

[0106] Regarding the determination of the reference region, at least one of the specified region located at the top of the height direction of the candidate question image and the specified region located at the bottom of the height direction of the candidate question image along the height direction of the candidate question image can be taken as the reference region, so that the other specified regions in the candidate question image except the reference region form a plurality of region pairs with the reference region; or a region with a representative position can be selected as the reference region from the plurality of specified regions in the candidate question image, or a specified region closest to the center of the image in the plurality of specified regions in the candidate question image can be taken as the reference region, which is not limited in the present application.

[0107] For example, referring to Figure 1C An exemplary schematic diagram of a plurality of specified regions labeled in a candidate question image is shown. For a certain candidate question image, the specified regions R1-R3 of the stem region type and the specified regions T1-T2 of the answer region type are labeled, and the region pairs are set according to the region types. The region pairs can include at least one of the following two types: stem region type region pairs (R1, R2) and (R1, R3), in which R1 is taken as the reference region; and answer region type region pairs (T1, T2), in which T1 is taken as the reference region.

[0108] The relative distance of the two regions in the region pair is used to measure the relative position relationship of the non-reference region with respect to the reference region in the region pair. The relative distance can be a spatial distance such as a pixel distance, an Euclidean distance, etc., or other indicators that can measure the relative position between regions. Specifically, the first relative distance represents the relative distance of the two regions in the region pair in the candidate question image, and the second relative distance represents the relative distance in the input question image when the region pair is mapped to the input question image.

[0109] The difference between the second relative distance and the second relative distance can reflect the layout similarity of the region pair in the candidate question image and the input question image. The greater the difference between the second relative distance and the second relative distance, the lower the layout similarity of the region pair, and the two are negatively correlated. The calculation method of the layout similarity can be selected according to specific needs, such as using the difference degree or the similarity coefficient as the layout similarity, and the smaller the difference between the first relative distance and the second relative distance of the region pair, the greater the similarity coefficient, indicating that the layout similarity is higher.

[0110] S104, determining the image similarity between the candidate question image and the input question image based on the layout similarity of each region pair in the candidate question image;

[0111] The relative positions and arrangement modes of different region pairs in the candidate question image can reflect the question layout of the entire image, and a single region pair reflects the similarity between local layouts. Therefore, the embodiment fuses the layout similarities of each region pair in the candidate question image as the image similarity between the candidate question image and the input question image to reflect the overall layout structure similarity between the two images.

[0112] As to the acquisition method of the image similarity, the layout similarities of the region pairs can be weighted and summed, or averaged, or other statistical methods can be used to calculate to obtain the image similarity, which can be determined according to specific application scenarios and needs. In the embodiment, the layout similarities of all region pairs in the candidate question image can be directly summed, and the sum result is taken as the image similarity between the candidate question image and the input question image.

[0113] S105, determining the searched question title image for the input question image from the plurality of candidate question images based on the image similarity between each candidate question image and the input question image.

[0114] The target of the image search is to find the question image that is most matched (i.e. the highest similarity or the smallest difference) with the input question image. The image similarity is used to reflect the similarity degree of the candidate question image and the input question image in content, structure, layout, etc. When using the difference degree to represent the image similarity, the smaller the difference degree means the more similar the two images are; and when using the similarity coefficient to represent, the higher the similarity coefficient means the more similar the two images are.

[0115] Based on this, in the process of determining the question title image searched for the input question image from the plurality of candidate question images, if the difference degree is used to represent the image similarity, one or more candidate question images with the minimum difference degree are selected as the question title image; if the similarity coefficient is used to represent, one or more candidate question images with the highest similarity coefficient are selected as the question title image.

[0116] The question image search method provided in the application first performs segmentation processing on the input question image, and recalls relevant question images from the retrieval library as the preliminary selected question images by using the image feature vectors, to preliminarily narrow the search range; then, a plurality of candidate question images matching the input question image are screened from the preliminary selected question images; then, the concept of region pair is innovatively introduced, the reference region and the non-reference region are determined based on the structural layout of the question in the candidate question image, the layout similarity is determined by using the relative distance of the region pair when the candidate question image is mapped to the input question image, and the image similarity between the candidate question image and the input question image is determined by comprehensively determining the layout similarity of each region pair, so as to determine the question title image.

[0117] In the embodiments of the present disclosure, the question image search method does not depend on character recognition, and performs image search by analyzing the image content features and structural layout features of the image itself, thereby improving the search accuracy; moreover, the method first performs segmentation processing on the input question image and recalls the preliminary selected question images by using the feature vectors, to preliminarily screen the possible relevant images, thereby reducing the data amount of subsequent processing, and further accurately matching by calculating the layout similarity in the candidate question image, thereby avoiding invalid search in a large number of irrelevant images, thereby improving the search efficiency, enabling the user to quickly obtain the question title image. In addition, the method can directly process and match the question images in different languages, thereby realizing unified search of the question images in different languages.

[0118] In some embodiments, the process of recalling relevant question images based on the feature vector of the input question image and the feature vectors corresponding to each segmented image respectively from the retrieval library to obtain the preliminary selected question images is actually a process of merging the recall results based on each feature vector. When recalling relevant question images based on the feature vector of the input question image and the feature vectors of each segmented image respectively, for each feature vector, a set of relevant question images with a vector similarity meeting the requirements as a relevant set is retrieved from the retrieval library. The present embodiment provides a way of determining the preliminary selected question images from all the recall results based on the cumulative weight score of each recalled relevant question image on all relevant sets, that is, each relevant set is assigned a weight score in advance, and the weight score of the relevant question image on the relevant set is determined as the screening basis of the preliminary selected question image according to whether the relevant question image is included in the relevant set.

[0119] Referring to Figure 2 An exemplary flowchart of an initial question image determination procedure based on cumulative weight score is shown in the figure, and the initial question image can be obtained in the following manner:

[0120] In S201, for each relevant question image recalled, a recall score of the relevant question image is determined according to whether the relevant question image exists in the corresponding relevant set of each feature vector; wherein the recall score is determined based on the weight score of the relevant question image in each relevant set; for each relevant set, if the relevant question image exists in the relevant set, the weight score of the relevant question image in the relevant set is a first value, otherwise the weight score of the relevant question image in the relevant set is a second value.

[0121] The weight score in the relevant set as a quantitative index can directly reflect the degree of association between the relevant question image and each relevant set. Based on the embodiment, the feature vector corresponding to the image used in recalling each relevant set includes the input question image as a whole and each split image corresponding to the input question image. Since the information contained in images of different sizes is different, the importance of the relevant set recalled based on the feature vector of different images is different, and the weight score of the same recalled relevant question image in each relevant set can be set based on the importance of the relevant set. Regarding the first value and the second value of the weight score in the same relevant set, the second value is less than the first value and the second value of the weight score in all relevant sets can be uniformly set to 0.

[0122] That is, the first value and the second value of the weight score of the relevant question image in the relevant set are determined in the following manner: based on the importance of the relevant set, a value pair of the weight score of the relevant question image in each relevant set is set, the value pair of the weight score includes the first value and the second value; wherein the importance of the relevant set includes: the importance of the relevant set recalled based on the feature vector of the input question image is higher than the importance of the relevant set recalled based on the feature vector of each split image; the importance of the relevant set recalled based on the feature vector of each split image is positively correlated with the size of the split image, that is, the larger the split image, the higher the importance of the relevant set recalled based on the split image, for example, the importance of the relevant set recalled based on the feature vector of the upper half image and the lower half image of the input question image is higher than the importance of the relevant set recalled based on the feature vector of the fixed size Q split images of the input question image.

[0123] For example, the second value is set as 0, the first value of the weight of the relevant set K1 recalled based on the feature vector of the input question image is set as 0.5, the first value of the weight of the relevant sets K2_1 and K2_2 recalled based on the feature vectors of the upper half and the lower half of the input question image after being divided is set as 0.2, and the first value of the weight of the relevant sets K3_1-K3_Q recalled based on the feature vectors of the Q (Q=9) divided images of a fixed size of the input question image after being divided is set as 0.1, and the weight fraction decreases with the decrease of the image.

[0124] Therefore, for each recalled relevant question image, it is determined whether the relevant question image exists in each relevant set, and the weight fraction of the relevant question image on each relevant set is obtained, and then the weight fraction of the relevant question image on each relevant set is summed as the recall score of the relevant question image. For example, in the case that the first value of the weight of the relevant set K1 is set as 0.5, the first value of the weight of the relevant sets K2_1 and K2_2 is set as 0.2, and the first value of the weight of the relevant sets K3_1-K3_Q is set as 0.1, for a recalled relevant question image P0, if the image appears in the relevant sets K1, K2_1, K3_2, K3_5 and K3_Q, the recall score can be represented as the sum of the first value of the weight of the relevant sets K1, K2_1, K3_2, K3_5 and K3_Q, that is, 0.5+0.2+0.1+0.1+0.1=1.

[0125] In S202, a plurality of relevant question images are selected from each relevant set corresponding to each feature vector as the preliminary selected question images based on the recall score of each relevant question image.

[0126] Based on the fact that the recall score can reflect the matching degree between the relevant question image and the input question image, the higher the recall score is, the higher the matching degree between the relevant question image and the input question image is, and the better the quality of the relevant question image is, therefore, a plurality of relevant question images with the highest recall score can be selected from each relevant set corresponding to each feature vector as the preliminary selected question images, or the relevant question images with a recall score higher than a score threshold can be selected as the preliminary selected question images.

[0127] In the embodiments of the present disclosure, the weight fraction is allocated to each relevant set, and the weight fraction of each relevant question image on each relevant set is determined according to whether the relevant question image is included in the relevant set, and then the recall score is calculated. Since the feature vectors corresponding to the input question image as a whole and each divided image are used to recall relevant question images from the retrieval library respectively, and the selection based on the recall score comprehensively considers the overall features and local features of the question image, the relevant question images that better meet the query requirements can be selected, the recall result is optimized, and the accuracy and efficiency of the retrieval are improved.

[0128] In some embodiments, the step of preliminarily screening the multiple images matching the input image from the preliminary selected images as the candidate images can be realized by a fast screening method of effective feature point matching. Specifically, referring to Figure 3A An exemplary flow chart of a step of screening the candidate images is shown, which at least includes the following steps:

[0129] S301, dividing the input image and each preliminary selected image into a grid image including a specified number of cells according to the same grid division manner;

[0130] Grid division means that an image is divided into multiple small areas or units according to certain rules and structures. When dividing the input whole-page image and each preliminary selected image, the same grid division manner is adopted. For example, a 6*6 grid division is adopted for the input image, and a 6*6 grid division is also adopted for each preliminary selected image, so as to ensure the comparability of different images.

[0131] S302, for each preliminary selected image, traversing the multiple feature points matching successfully determined by the feature point matching between the preliminary selected image and the input image, detecting whether the orientation of the cell where the matching successful feature point is located on the input image and the cell where the matching successful feature point is located on the preliminary selected image remains consistent, if yes, determining that the matching successful feature point is an effective feature point, and obtaining the number of effective feature points corresponding to the preliminary selected image;

[0132] Feature point matching is used to find similar local feature points between two images, which can be the position of the image with obvious texture, edge or corner point. The multiple matching successful feature points represent the feature points with corresponding relationship between the input image and the preliminary selected image. In the feature point matching process, each feature point is accompanied by a descriptor (i.e. a numerical vector), which is used to represent the local image information around the point. By comparing the similarity between the descriptors of the feature points in the input image and the preliminary selected image (such as using Euclidean distance, Hamming distance or correlation coefficient as a measure), it can be determined whether they are the same physical point in different images, so as to determine the matching successful feature points.

[0133] Orientation means the relative position or direction of the feature point in the cell. For example, a feature point is located at the upper left corner of the cell. By comparing the orientation of the matching successful feature points in the input image and the preliminary selected image, it can be determined whether the feature points maintain similar positional relationship in the two images, so as to improve the accuracy of feature point matching.

[0134] The feature points based on successful matching are usually determined based on the similarity of descriptors. There is a problem of feature point mismatching, such as the actual local area features in different images are completely different but are mismatched due to similar descriptors. Therefore, this application performs a secondary test on each successfully matched feature point obtained by feature point matching, so as to select the multiple successfully matched feature points that meet the requirements of the same local area and similar descriptors as valid feature points, thereby reducing the occurrence of mismatching problems.

[0135] During the secondary detection process, for each successfully matched feature point, the first grid area where the feature point is located on the input question image and the second grid area where the feature point is located on the preliminary question image are determined. Based on the fact that the grid areas on the input question image and the preliminary question image are consistent and one-to-one corresponding, when the first grid area and the second grid area of ​​the feature point maintain the same position, the successfully matched feature point is determined to be a valid feature point.

[0136] For example, see Figure 3B A schematic diagram of the grid positions of valid feature points when an input question image and a preliminary question image are divided into grid images is shown as an example. The input question image and the preliminary question image are divided into grid images containing 81 grids respectively using a 9*9 grid division method. A successfully matched feature point X1 is located in the grid areas G1 and G7 of the input question image and the preliminary question image respectively, so the feature point is not a valid feature point. A feature point X2 is located in the grid areas G31 and G31 of the input question image and the preliminary question image respectively, so the feature point is a valid feature point.

[0137] For each preliminary question image, the above method can be used to determine the number of valid feature points included in the multiple feature points that are successfully matched between the preliminary question image and the input question image through feature point matching. The number of valid feature points is less than or equal to the number of the multiple feature points that are successfully matched.

[0138] S303 : Based on the number of valid feature points corresponding to each preliminary-selected topic image, determine a plurality of preliminary-selected topic images that match the input topic image as candidate topic images.

[0139] The effective feature points reflect the similarity and consistency of the local features between the input question image and the preliminary question image. The larger the number of effective feature points corresponding to the preliminary question image, the higher the similarity / matching degree between the preliminary question image and the input question image. Therefore, a threshold for the number of effective feature points can be set in advance, and the number of effective feature points of each preliminary question image can be compared with the threshold. If the number of effective feature points is greater than or equal to the threshold, the preliminary question image is determined as a candidate question image. The setting of the threshold can be adjusted according to actual needs and experience, or all the preliminary question images can be sorted from high to low according to the number of effective feature points, and multiple preliminary question images with the highest ranking can be selected as candidate question images. The specific number of selections can be determined according to actual needs. For example, the first N preliminary question images can be selected as candidate question images, where N is a pre-set value.

[0140] In the embodiment of the present disclosure, by performing a secondary detection on the successfully matched feature points determined based on feature point matching, and by checking whether the orientations of the successfully matched feature points in the corresponding grids of the input question image and the preliminary question image are consistent, these mismatched feature points can be effectively eliminated, thereby enhancing the reliability of the feature points. The number of valid feature points determined can more accurately reflect the similarity between the input question image and the preliminary question image, thereby achieving efficient and accurate selection of candidate question images that match the input question image, thereby improving search efficiency.

[0141] In some embodiments, for the feature point matching between the preliminary question image and the input question image described in the aforementioned embodiments, in order to improve the feature point matching efficiency, the M feature points with the largest feature point intensity in the image may be used for matching, which can be achieved through the following steps a1-a2:

[0142] a1. Extract feature points from the image to be processed using a feature point extraction algorithm, and select the top M feature points with the largest feature point strengths from the extracted multiple feature points using an index that measures the strength of the feature points, as the feature points to be matched for the image to be processed; the index that measures the strength of the feature points corresponds to the feature point extraction algorithm used; the image to be processed includes the input question image and each of the preliminarily selected question images;

[0143] The feature point extraction algorithm is used to automatically detect and locate representative feature points from an image. The extracted feature points can include, but are not limited to, points located at positions with rich texture, obvious edges, or corner points in the image. The feature point extraction algorithm can include, but is not limited to, SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), etc.

[0144] Different feature point extraction algorithms will set an index for measuring the strength of the feature point while extracting the feature point. In this embodiment, the first M feature points with the strongest feature point strength are selected as the to-be-matched feature points of the to-be-processed image according to the index of the feature points. For example, when the SURF algorithm is used to extract feature points, the algorithm itself will calculate the response value of each feature point in the process of detecting the feature points. The response value of the feature point is used to measure the feature point saliency and stability, and reflects the uniqueness of the feature point in the image and the ability to maintain in different image transformations. The higher the response value, the more significant and stable the feature point, i.e., the greater the feature point strength. The response value can be used as an index for measuring the strength of the feature point. Similarly, for other feature point extraction algorithms, such as ORB, SIFT, etc., the algorithm will also calculate a certain measure value of each feature point when extracting the feature points. For example, the ORB algorithm uses FAST corner detection combined with BRIEF descriptors. There is also an index similar to the saliency of the corner point in the corner detection process. The SIFT algorithm constructs a scale space and detects extreme points. The contrast of the extreme points can also be used as an index for measuring the strength of the feature point.

[0145] In this embodiment, the first M feature points with the strongest feature point strength in the input question image are obtained as the to-be-matched feature points of the input question image by the feature point extraction algorithm, and the first M feature points with the strongest feature point strength in each preselected question image are obtained as the to-be-matched feature points of the preselected question image by the same method.

[0146] a2, performing feature point matching between the to-be-matched feature points of each preselected question image and the to-be-matched feature points of the input question, to obtain a plurality of matched feature points corresponding to the preselected question image.

[0147] Feature point matching determines whether there is a corresponding relationship between two feature points in different images by comparing the feature descriptors of the feature points to be matched in two images. The feature descriptor is a mathematical representation of the image information in the local area around the feature point, which contains information such as the position, direction, scale, and gray scale distribution of the surrounding pixels of the feature point. Whether two feature points match can be determined by calculating the similarity between the feature descriptors of the two feature points.

[0148] In the feature point matching process, when the similarity of the descriptors of two feature points exceeds a set threshold, it can be considered that the two feature points are successfully matched. For each preliminary question image, a set of successfully matched feature points can be obtained after feature point matching with the input question image.

[0149] In the embodiments of the present disclosure, by using the first M feature points with the largest intensity in the preliminary question image and the input question image for matching, the number of feature point pairs that need to be calculated for similarity is greatly reduced, thereby reducing the computational complexity. The time required for the entire feature point matching process is also correspondingly shortened, thereby significantly improving the matching efficiency. At the same time, selecting the first M feature points with the largest intensity for matching avoids introducing noise points or less significant feature points, effectively reducing the influence of noise points and improving the quality of matching.

[0150] In some embodiments, before determining the second relative distance of the two regions in the region pair in the candidate question image when mapping to the input question image, the position information of each specified region in the candidate question image when mapping to the input question image can be determined; the plurality of specified regions include at least one reference region and at least one non-reference region; based on this, the determination of the second relative distance of the two regions in the region pair in the candidate question image when mapping to the input question image can be determined based on the position information of the two regions in the region pair when mapping to the input question image.

[0151] The position information can include at least one of the center coordinates of the region or the boundary coordinates of the region, for describing the specific position of the region in the input question image.

[0152] The position information of each specified region in the candidate question image when mapping to the input question image indicates the position coordinates and other information of the region in the input question image after each specified region in the candidate question image is projected onto the input question image by transformation. Each specified region in the candidate question image determines its position information on the input question image.

[0153] In determining the position information of each designated region on the input question image, a pre-established transformation model can be used. The position information of each designated region in the candidate question image is input into the transformation model, and the position information of the designated region output by the model is obtained. Alternatively, the position information of each designated region after mapping can also be determined by establishing the association between the position information of each designated region in the candidate question image and the feature points in the candidate question image, and by matching the feature points. That is:

[0154] The position information of each designated region in the candidate question image can be determined first. The position information can be represented by at least one coordinate point. The association between the coordinate point representing the position information and the feature points in the candidate question image is established. Next, based on the feature points in the candidate question image and the feature points in the input question image, the matching feature point pairs are determined by matching the feature points. Based on the matching feature point pairs and the association, the coordinate point representing the position information of each designated region in the candidate question image is mapped to the coordinate point on the input question image, and the position information of each designated region on the input question image is obtained.

[0155] The association between the coordinate point representing the position information and the feature points in the candidate question image can be determined by the nearest neighbor association or the region-based feature point association. For example, for each coordinate point representing the position information of the designated region, the distances between the coordinate point and all feature points in the candidate question image can be calculated, and the nearest feature point is associated with the coordinate point. Alternatively, the feature points within or near a certain range of the designated region can be counted, and these feature points are associated with the coordinate point representing the position information of the region.

[0156] In the process of determining the mapped coordinate point based on the matching feature point pairs and the association, the transformation relationship between the candidate question image and the input question image can be established according to the matching feature point pairs. Then, the coordinate point representing the position information of the designated region in the candidate question image can be transformed based on the transformation relationship, and the corresponding coordinate point in the input question image is obtained. According to the mapped coordinate point, the position information of each designated region on the input question image can be determined. For example, if the position information of the designated region is represented by the center point coordinate, the mapped center point coordinate represents the position information of the designated region on the input question image. If the position information of the designated region is represented by multiple coordinate points representing the region boundary, the position information of the designated region on the input question image can be determined after mapping the boundary coordinate points.

[0157] Based on the fact that the feature points are points with significant features and stability in the image, the mapping mode based on feature point matching can accurately determine the mapping position of the specified region by using the corresponding relationship of these stable points, reduce the mapping error caused by image deformation, noise and other factors, thereby improving the mapping accuracy of each specified region in the candidate question image to the input question image, and enhancing the accuracy of the determined position information.

[0158] In some embodiments, for the first relative distance / second relative distance described in the foregoing embodiments, "first" and "second" are only used to distinguish the same type of relative distance, and do not have a specific indicative meaning by themselves. The first relative distance and the second relative distance are determined in the same way, see Figure 4 The calculation steps of the relative distance of the two regions in the region pair on the image are shown in the flowchart, which can be determined in the following way. When the image where the non-reference region is located is the candidate question image, the corresponding relative distance of the region pair is the first relative distance. When the image where the non-reference region is located is the input question image, the corresponding relative distance of the region pair is the second relative distance:

[0159] S401, determine the first coordinate difference in the first direction and the second coordinate difference in the second direction of the position information of the non-reference region in the region pair relative to the position information of the reference region; the first direction represents the width direction of the image where the non-reference region is located; the second direction represents the height direction of the image where the non-reference region is located;

[0160] The first direction represents the width direction of the image where the non-reference region is located, which usually refers to the horizontal direction of the image, and in the coordinate system, it can correspond to the x-axis direction (depending on the definition of the coordinate system); correspondingly, the second direction is along the height direction of the image where the non-reference region is located, which usually refers to the vertical direction of the image, and in the coordinate system, it can correspond to the y-axis direction (depending on the definition of the coordinate system). The first direction and the second direction are perpendicular to each other.

[0161] The first coordinate difference is used to reflect the coordinate difference value of the position information of the non-reference region relative to the position information of the reference region in the first direction (width direction), which reflects the horizontal offset of the non-reference region relative to the reference region; the second coordinate difference is used to reflect the coordinate difference value of the position information of the non-reference region relative to the position information of the reference region in the second direction (height direction), which reflects the vertical offset of the non-reference region relative to the reference region.

[0162] The position information of the non-reference region and the position information of the reference region can represent the position of the region by using the center point coordinates, boundary coordinates, etc. of the region. For example, taking the center point coordinates of the region as an example, assuming that the center point coordinates of the reference region are (X_0, Y_0) and the center point coordinates of the non-reference region are (X_r, Y_r), then:

[0163] According to the definition of the first direction (the image width direction, assuming corresponding to the x-axis direction), the coordinate difference Δx of the position information of the non-reference region relative to the position information of the reference region in the first direction can be represented as Δx = X_r - X_0, which represents the offset of the non-reference region relative to the reference region in the horizontal direction. If Δx is positive, it means that the non-reference region is on the right side of the reference region. If Δx is negative, it means that the non-reference region is on the left side of the reference region.

[0164] Similarly, according to the definition of the second direction (the image height direction, assuming corresponding to the y-axis direction), the coordinate difference Δy of the position information of the non-reference region relative to the position information of the reference region in the second direction can be represented as Δy = Y_r - Y_0, which represents the offset of the non-reference region relative to the reference region in the vertical direction. If Δy is positive, it means that the non-reference region is below the reference region. If Δy is negative, it means that the non-reference region is above the reference region.

[0165] S402, normalize the first coordinate difference and the second coordinate difference, and determine the corresponding relative distance of the region pair based on the normalized first coordinate difference and second coordinate difference;

[0166] In this embodiment, the purpose of normalization is to eliminate the influence of the image size difference between the candidate question image and the input question image on the coordinate difference, so that the data of different features have comparability. Therefore, when normalizing the first coordinate difference, the first coordinate difference can be divided by the image length of the image in which the non-reference region is located in the first direction, and similarly, when normalizing the second coordinate difference, the second coordinate difference can be divided by the image length of the image in which the non-reference region is located in the second direction. Assuming that the width of the image in which the non-reference region is located is W and the height is H, then for the first coordinate difference Δx, the normalized value is Δx = Δx / W, and the value range of Δx is limited to [-1, 1] (assuming that the value range of Δx is [-W, W]), and similarly, for the second coordinate difference Δy, the normalized value is Δy = Δy / H, and the value range of Δy is limited to [-1, 1] (assuming that the value range of Δy is [-H, H]).

[0167] Based on the normalized first coordinate difference Δx and the second coordinate difference Δy, the relative distance D of the region pair can be calculated. For example, the Euclidean distance formula can be used: The measured is the distance of the normalized coordinate difference in two-dimensional space, which can comprehensively reflect the position offset degree of the non-reference region relative to the reference region in the horizontal and vertical directions. The greater the value of the relative distance D, the farther the position of the non-reference region relative to the reference region; the smaller the value of the relative distance D, the closer the position of the non-reference region relative to the reference region.

[0168] In the embodiments of the present disclosure, by simultaneously considering the coordinate difference in the first direction (the image width direction) and the second direction (the image height direction), the position change of the non-reference region relative to the reference region can be comprehensively reflected, and by calculating the coordinate difference in the two directions respectively and comprehensively calculating the relative distance, the relative position relationship between the region pairs can be more accurately described.

[0169] In order to enable those skilled in the art to better understand the subject image search method provided by the present application, the following will be described in a more specific process of searching the subject title image from the retrieval library for the input subject image.

[0170] As Figure 5 The flowchart of the exemplary subject image search method provided by the present application can include three search stages:

[0171] S501, in the first stage, TOP-K1 primary selected subject images are recalled and determined from the retrieval library by the feature vectors of the input subject image and the split images thereof;

[0172] S502, in the second stage, TOP-K2 candidate subject images are quickly screened out from the TOP-K1 primary selected subject images searched in the first stage by preliminary screening;

[0173] S503, in the third stage, the subject title image meeting the requirements is determined from the TOP-K2 candidate subject images searched in the second stage by using the layout similarity of the region pairs.

[0174] (1) The first stage

[0175] ① Feature vector acquisition: for the input subject image, the input subject image is split into an upper half image and a lower half image, and the input subject image is split into Q fixed size split images (Q is greater than 2), the feature vector V1 of the input subject image itself is acquired, the feature vectors V2_1 and V2_2 of the upper half image and the lower half image are acquired, and the feature vectors V3_1-V3_Q corresponding to the Q fixed size split images are acquired; wherein, when the input subject image is split into Q fixed size split images, the input subject image can be first scaled to a scaling image with a specified width and height in equal proportions, and then the scaling image is split.

[0176] ②Retrieval: input the feature vector of the upper half of the image into the vector search library of the upper half of the image, output TopN1 relevant question images; input the feature vector of the lower half of the image into the vector search library of the lower half of the image, output TopN2 relevant question images; input the feature vector of the whole image into the feature vector search library of the whole image, output TopN3 relevant question images; input the Q segmented images obtained after segmentation into the feature vector search library of the segmented images, and each segmented image outputs TopN4 relevant question images;

[0177] ③Determination of TOP-K1 preliminary selected question images: the recalled relevant question images of the whole image, the upper half of the image, the lower half of the image and all segmented images are summarized, the weight of each relevant question image appearing in the whole image is divided into 0.4, the weight of each relevant question image appearing in the lower half of the image is divided into 0.2, and the weight of each relevant question image appearing in the segmented image is divided into 0.1. The same relevant question image appearing in the whole image, the upper half of the image, the lower half of the image and the segmented image is accumulated, that is, if a relevant question image appears once in the whole image search, it adds 0.4, if it appears once in the upper half of the image or the lower half of the image, it adds 0.2, and if it appears once in the segmented image search, it adds 0.1. In actual application, the weight of the whole image search, the weight of the upper half of the image and the lower half of the image, and the weight of the segmented image search can be flexibly adjusted according to the application scene. After the weight of all recalled relevant question images of the whole image, the upper half of the image, the lower half of the image and all segmented images is accumulated, each relevant question image can be reordered according to the weight accumulation, and the TOP-K1 relevant question images with the highest weight accumulation are taken as the preliminary selected question images. K1 can be adjusted according to the application scene in actual application.

[0178] (2) Second stage

[0179] ①Feature point extraction: a feature point extraction algorithm is used to extract feature points from the input question image, and the extracted feature points are screened to select the top X feature points with the strongest features. The same method is used to select the top X feature points with the strongest features for each preliminary selected question image. The value of X can be adjusted according to the application scene in actual application. The feature point extraction algorithm used in this embodiment is SURF feature point extraction algorithm. Other feature point extraction algorithms such as ORB, SIFT, etc. can also be used in actual application, and neural network can also be used for feature point extraction.

[0180] ② Determine the number of valid feature points: Use a feature point matching algorithm such as the knn algorithm to match the current input question image with the TOP-K1 preliminary question images. Perform a secondary confirmation on the successfully matched feature points, and detect whether the grid orientation of the successfully matched feature points on the user image and the search result image is consistent. The successfully matched feature points that meet the grid orientation consistency are regarded as valid feature points. All successfully matched feature points of the preliminary question image are traversed to determine the number of valid feature points corresponding to the preliminary question image. In this embodiment, the input question image and the preliminary question image are divided into Z grids according to the same grid division method, and the grids on the two images correspond one to one.

[0181] ③ Preliminary screening: Sort the TOP-K1 preliminary selected topic images from large to small according to the number of valid feature points, and take the TOP-K2 preliminary selected topic images (as candidate topic images) as the output results of the preliminary screening. In actual application, the K2 value can be adjusted according to the application scenario, or a dynamic output method can be used. For example, when the number of valid feature points is greater than the threshold, it is used as the output result of the preliminary screening.

[0182] (3) The third stage

[0183] ① Area mapping: Traverse the TOP-K2 candidate question images, determine the question stem box and answer area box for each question for the current candidate question image, input the current candidate question image, the question stem box and answer area box, and the input question image into the trained model, and output the question stem box and answer area box in the current candidate question image projected to the corresponding position information in the input question image. In actual application, other methods can also be used to map the question stem box and answer area box in the current candidate question image to the input question image.

[0184] ② The relative distance of the region pair in the candidate question image: traverse the TOP-K2 candidate question images, select the top stem frame and the bottom stem frame from all the stem frames in the current candidate question image as two reference frames, and count the relative distances of other non-reference stem frames to each reference stem frame. Regarding the calculation of this relative distance, one or more boundary points of the accurate identification area frame range of each stem frame can be taken, and the coordinate differences of each boundary point of each non-reference stem frame in the horizontal and vertical directions of the image can be counted. The mean of the coordinate differences is calculated and normalized. The normalization method is to divide the mean of the horizontal coordinate differences by the width of the current candidate question image, and the mean of the vertical coordinate differences by the height of the current candidate question image. Based on the normalized mean of the horizontal and vertical coordinate differences, the relative distance of each non-reference stem frame in the current candidate question image relative to each reference stem frame is determined;

[0185] ③Region pair mapping to relative distance in input question image: for each region pair in the current candidate question image formed by each non-reference stem frame and each reference stem frame respectively, based on the position information of the region pair mapped in the previous step ①, the relative distance between the non-reference stem frame and the reference stem frame in the region pair when they are mapped to the input question image is determined by using the relative distance calculation method recorded in the previous step ②; each region pair includes one non-reference stem frame and one reference stem frame;

[0186] ④Layout similarity of region pair: for each region pair in the current candidate question image, based on the difference of the relative distances in the candidate question image and the input question image, which can be directly expressed as the difference value of the two relative distances; based on the difference of the relative distances of each region pair in the current candidate question image, the stem frame similarity between the current candidate question image and the input question image is determined, for example, the sum of the relative distance difference values of all region pairs in the current candidate question image is taken as the stem frame similarity distance between the current candidate question image and the input question image.

[0187] The same way as the previous steps ① to ④ is used to traverse the TOP-K2 candidate question images, and the answer area frame similarity between the current candidate question image and the input question image is determined for all the answer area frames of the questions in the current candidate question image;

[0188] ⑤Image similarity between candidate question image and input question image: based on the stem frame similarity distance and the answer area frame similarity between the current candidate question image and the input question image, the image similarity between the current candidate question image and the input question image is determined; for example, the sum of the stem frame similarity and the answer area frame similarity is taken as the image similarity;

[0189] ⑥Question title image acquisition: based on the image similarity between each candidate question image and the input question image, one or more candidate question images whose image similarity meets the requirements are taken as the search results for the input question image; for example, if the image similarity is represented by the layout difference value, the results can be sorted in ascending order of the layout difference value, and the top TOP-K4 results are taken as the searched question title images for the input question image.

[0190] Corresponding to the embodiments of the question image search method described above, referring to FIG. 1, the present application also provides embodiments of a question image search device, which comprises: Figure 6

[0191] ​The feature vector recall module 601 is configured to perform segmentation processing on the input question image to be searched to obtain N segmented images, and recall relevant question images from the retrieval library based on the feature vector of the input question image and the feature vector corresponding to each segmented image, to obtain preliminary selected question images.

[0192] The preliminary screening module 602 is configured to preliminarily screen the preliminary selected question images to obtain a plurality of images matching the input question image as candidate question images.

[0193] The layout similarity screening module 603 is configured to determine, for each region pair in each candidate question image, a first relative distance between two regions in the region pair in the candidate question image, and a second relative distance between the two regions in the region pair when the two regions in the region pair are mapped to the input question image, and determine a layout similarity of the region pair according to the first relative distance and the second relative distance; wherein any region pair includes a designated reference region and any non-reference region in the candidate question image; the reference region and the non-reference region are determined based on the structural layout of the question included in the candidate question image; the image similarity between the candidate question image and the input question image is determined based on the layout similarities of the region pairs in the candidate question image; and the title question image searched for the input question image is determined from the plurality of candidate question images based on the image similarity between each candidate question image and the input question image.

[0194] In some embodiments, the feature vector recall module at least includes:

[0195] The recall module is configured to, for each feature vector, retrieve a set of relevant question images from the retrieval library, wherein the relevant question images satisfy a requirement of vector similarity with the feature vector as a relevant set;

[0196] The recall statistics module is configured to, for each relevant question image, determine a recall score of the relevant question image according to whether the relevant question image exists in the relevant set corresponding to each feature vector; wherein the recall score is determined based on a weight score of the relevant question image in each relevant set; for each relevant set, if the relevant question image exists in the relevant set, the weight score of the relevant question image in the relevant set is a first value, otherwise the weight score of the relevant question image in the relevant set is a second value;

[0197] The recall screening module is configured to, based on the recall score of each relevant question image, screen a plurality of relevant question images from the relevant set corresponding to each feature vector as preliminary selected question images.

[0198] In some embodiments, the first value and the second value of the weight score of the relevant question image in the relevant set are determined in the following manner:

[0199] The weight value pair of the weight score of the relevant question image on the relevant set is set for each relevant set based on the importance of the relevant set, and the weight value pair includes a first value and a second value; the importance of the relevant set includes that the importance of the relevant set recalled based on the feature vector of the input question image is higher than the importance of the relevant set recalled based on the feature vector corresponding to each split image; the importance of the relevant set recalled based on the feature vector corresponding to each split image is positively correlated with the size of the split image.

[0200] In some embodiments, the preliminary screening module at least includes:

[0201] The grid division module is configured to divide the input question image and each preselected question image into grid images including a specified number of grids according to the same grid division manner.

[0202] The feature point detection module is configured to, for each preselected question image, traverse the plurality of matched feature points of the preselected question image and the input question image determined through feature point matching, detect whether the orientation of the grid where the matched feature point is located on the input question image and the grid where the matched feature point is located on the preselected question image remains consistent, and if so, determine that the matched feature point is a valid feature point, and obtain the number of valid feature points corresponding to the preselected question image.

[0203] The candidate question image screening module is configured to determine a plurality of preselected question images matched with the input question image as candidate question images based on the number of valid feature points corresponding to each preselected question image.

[0204] In some embodiments, the feature point matching of the preselected question image and the input question image is achieved by the following manner:

[0205] The feature point extraction algorithm is used to extract feature points from the to-be-processed image, and the first M feature points with the largest feature point intensity are selected from the plurality of feature points extracted by measuring the feature point intensity index as the to-be-matched feature points of the to-be-processed image; wherein the feature point intensity index corresponds to the feature point extraction algorithm used; the to-be-processed image includes the input question image and each preselected question image.

[0206] The to-be-matched feature points of each preselected question image are matched with the to-be-matched feature points of the input question image to obtain the plurality of matched feature points corresponding to the preselected question image.

[0207] In some embodiments, the device further includes:

[0208] The mapping module is configured to determine, for each of the specified regions in the candidate question image, position information of the specified region on the input question image.

[0209] The layout similarity screening module is configured to determine, when determining the second relative distance between the two regions in the region pair in the candidate question image, a relative distance between the two regions based on the position information of the two regions on the input question image.

[0210] In some embodiments, the mapping module comprises:

[0211] The position information determining and associating module is configured to determine position information of each of the specified regions in the candidate question image, and establish an association between a coordinate point representing the position information and a feature point in the candidate question image.

[0212] The feature point pair obtaining module is configured to determine a matching feature point pair by matching feature points in the candidate question image with feature points in the input question image.

[0213] The coordinate mapping module is configured to determine, based on the matching feature point pair and the association, a coordinate point representing the position information of each of the specified regions in the candidate question image, and map the coordinate point to a coordinate point on the input question image, to obtain the position information of each of the specified regions on the input question image.

[0214] In some embodiments, the first / second relative distance is determined by:

[0215] determining a first coordinate difference and a second coordinate difference between the position information of the non-reference region and the position information of the reference region in the region pair in a first direction and a second direction, respectively, the first direction representing a width direction of the image in which the non-reference region is located, and the second direction representing a height direction of the image in which the non-reference region is located;

[0216] normalizing the first coordinate difference and the second coordinate difference, and determining the relative distance corresponding to the region pair based on the normalized first coordinate difference and the second coordinate difference.

[0217] wherein, when the image in which the non-reference region is located is the candidate question image, the relative distance corresponding to the region pair is the first relative distance; and when the image in which the non-reference region is located is the input question image, the relative distance corresponding to the region pair is the second relative distance.

[0218] The implementation process of the functions and roles of each unit in the above apparatus is specifically described in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0219] The apparatus embodiments described above are merely illustrative, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the scheme of the present application according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0220] The electronic device provided in the embodiment of the present application has the advantages of the above-mentioned method. Figure 7 As shown in the structural schematic diagram of the electronic device 700, the electronic device 700 includes at least one processor 701, a memory 702 and a bus 703, and the at least one processor 701 is electrically connected with the memory 702; the memory 702 is configured to store at least one computer executable instruction, and the processor 701 is configured to execute the at least one computer executable instruction, so as to execute the steps of any one of the question image search methods provided in any one of the embodiments or any one of the optional implementation manners of the present application.

[0221] Further, the processor 701 can be an FPGA (Field-Programmable Gate Array, field programmable gate array) or other devices with logic processing capability, such as an MCU (Microcontroller Unit, microcontroller unit) or a CPU (Central Process Unit, central processing unit).

[0222] The embodiment of the present application further provides another readable storage medium storing a computer program, which is used to implement the steps of any one of the question image search methods provided in any one of the embodiments or any one of the optional implementation manners of the present application when executed by a processor.

[0223] The readable storage medium provided by the embodiments of the present application includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the readable storage medium includes any medium that stores or transmits information in a form readable by an apparatus (for example, a computer).

[0224] Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures need not be performed in the particular order shown, or sequentially, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0225] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for searching a topic image, characterized in that: The method comprises: The input question image to be searched is segmented to obtain N segmented graphs, and based on the feature vector of the input question image and the feature vectors corresponding to each segmented graph, relevant question images are respectively recalled from the search library to obtain preliminary selected question images; Preliminarily screening out a plurality of images matching the input topic image from the preliminarily selected topic images as candidate topic images; For each region pair in each candidate question image, determining a first relative distance between two regions in the region pair in the candidate question image, and determining a second relative distance between the two regions in the region pair when mapped to the input question image, and determining a layout similarity of the region pair based on the first relative distance and the second relative distance; wherein any region pair includes a designated reference region and any non-reference region in the candidate question image; the reference region and the non-reference region are determined based on the structural layout of the question included in the candidate question image; Determining the image similarity between the candidate topic image and the input topic image based on the layout similarity of each region pair in the candidate topic image; Based on the image similarity between each candidate topic image and the input topic image, a target topic image searched for the input topic image is determined from the plurality of candidate topic images.

2. The method according to claim 1, characterized in that Recalling relevant question images from the search library based on the feature vector of the input question image and the feature vectors corresponding to each segmentation graph to obtain preliminary question images, including: For each feature vector, a set of related topic images whose vector similarity with the feature vector meets the requirements is retrieved from the retrieval library as a related set; For each recalled relevant topic image, a recall score of the relevant topic image is determined based on whether the relevant topic image exists in the relevant set corresponding to each feature vector; wherein the recall score is determined based on the weight score of the relevant topic image in each relevant set; for each relevant set, if the relevant topic image exists in the relevant set, the weight score of the relevant topic image in the relevant set is a first value; otherwise, the weight score of the relevant topic image in the relevant set is a second value; Based on the recall score of each relevant topic image, multiple relevant topic images are screened out from the relevant set corresponding to each feature vector as preliminary topic images.

3. The method according to claim 2, characterized in that The first value and the second value of the weight score of the relevant topic image on the relevant set are determined by the following method: Based on the importance of the relevant set, a value pair of weight scores of the relevant topic image in the relevant set is set for each relevant set, wherein the value pair of weight scores includes a first value and a second value; The importance of the related set includes: The importance of the related set based on the feature vector of the input question image is higher than the related set based on the feature vector corresponding to each segmentation graph; The importance of the related set recalled based on the feature vector corresponding to each segmentation graph is positively correlated with the size of the segmentation graph.

4. The method according to claim 1, wherein Preliminarily screening out multiple images matching the input topic image from the preliminarily selected topic images as candidate topic images, including: Divide the input question image and each preliminary question image into a grid image including a specified number of grids according to the same grid division method; For each preliminary question image, traverse multiple successfully matched feature points determined by feature point matching between the preliminary question image and the input question image, and check whether the grid where the successfully matched feature point is located on the input question image is consistent with the grid where it is located on the preliminary question image. If so, determine that the successfully matched feature point is a valid feature point, and obtain the number of valid feature points corresponding to the preliminary question image; Based on the number of valid feature points corresponding to each preliminary-selected topic image, a plurality of preliminary-selected topic images matching the input topic image are determined as candidate topic images.

5. The method according to claim 4, characterized in that The feature point matching between the preliminary question image and the input question image is achieved in the following way: Extracting feature points from the image to be processed using a feature point extraction algorithm, and selecting the top M feature points with the largest feature point intensities from the extracted multiple feature points using an index for measuring the strength of the feature points as the feature points to be matched for the image to be processed; wherein the index for measuring the strength of the feature points corresponds to the feature point extraction algorithm used; the image to be processed includes the input question image and each of the preliminarily selected question images; The feature points to be matched of each preliminary question image are matched with the feature points to be matched of the input question to obtain a plurality of successfully matched feature points corresponding to the preliminary question image.

6. The method according to claim 1, characterized in that Before determining a second relative distance between two regions in the region pair in the candidate question image when mapped to the input question image, the method includes: For a plurality of designated areas in the candidate question image, determining position information of each designated area mapped to the input question image; the plurality of designated areas include at least one reference area and at least one non-reference area; The determining of a second relative distance between two regions in the region pair in the candidate question image when mapped to the input question image includes: Based on the position information of the two regions in the region pair mapped onto the input question image, the relative distance between the two regions is determined.

7. The method according to claim 6, characterized in that Determine the location information of each designated area mapped to the input question image, including: Determine the position information of each designated area on the candidate question image, wherein the position information is represented by at least one coordinate point, and establish an association relationship between the coordinate point representing the position information and a feature point in the candidate question image; Based on the feature points in the candidate question image and the feature points in the input question image, a matching feature point pair is determined by feature point matching; Based on the matching feature point pairs and the association relationship, the coordinate points representing the position information of each designated area on the candidate question image are mapped to the coordinate points on the input question image to obtain the position information of each designated area mapped to the input question image.

8. The method according to claim 1, characterized in that The first relative distance / the second relative distance is determined by: Determining a first coordinate difference in a first direction and a second coordinate difference in a second direction between position information of the non-reference region in the region pair and position information of the reference region; the first direction represents a width direction along the image where the non-reference region is located; and the second direction represents a height direction along the image where the non-reference region is located; Normalizing the first coordinate difference and the second coordinate difference, and determining a relative distance corresponding to the region pair based on the normalized first coordinate difference and the second coordinate difference; When the image where the non-reference area is located is the candidate question image, the relative distance corresponding to the area pair is the first relative distance; when the image where the non-reference area is located is the input question image, the relative distance corresponding to the area pair is the second relative distance.

9. A title image search device, characterized in that: The device comprises: A feature vector recall module is used to segment the input question image to be searched to obtain N segmentation images, and based on the feature vector of the input question image and the feature vectors corresponding to each segmentation image, recall relevant question images from the search library to obtain preliminary question images; A preliminary screening module, configured to preliminarily screen out a plurality of images matching the input topic image from the preliminarily selected topic images as candidate topic images; A layout similarity screening module is used to determine, for each region pair in each candidate question image, a first relative distance between two regions in the region pair in the candidate question image, and to determine a second relative distance between the two regions in the region pair in the candidate question image when mapped to the input question image, and to determine the layout similarity of the region pair based on the first relative distance and the second relative distance; wherein any region pair includes a designated reference region and any non-reference region in the candidate question image; the reference region and the non-reference region are determined based on the structural layout of the question included in the candidate question image; based on the layout similarity of each region pair in the candidate question image, the image similarity between the candidate question image and the input question image is determined; based on the image similarity between each candidate question image and the input question image, a target question image searched for the input question image is determined from the multiple candidate question images.

10. An electronic device, characterized in that: include: Memory, processor; The memory is used to store computer programs; The processor is configured to call the computer program to implement the method according to any one of claims 1 to 8.

11. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images