Question search methods, devices and electronic equipment

Abstract

This disclosure relates to a question search method, apparatus, and electronic device. By repeatedly confirming the existence of defective questions in the search results during the question search process and performing separate searches on the illustrations of the question to be searched, this disclosure reduces the probability of search failure when the target question is a defective question, improves the recall rate of defective questions, and thus effectively improves the accuracy of the question search method.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus and electronic device for searching questions. Background Technology

[0002] Image-based question search refers to users taking pictures of questions using smartphones or other devices and uploading them to a Q&A system. The system then automatically returns the stems, answers, and explanations of several most similar questions for the user's reference. With the continuous development of online education, image-based question search has become a common method for users to solve difficult problems.

[0003] Q&A systems typically identify the text in user-uploaded question images, then search for questions with high similarity to the text in the uploaded images and provide feedback. This method works well for questions with a lot of text, but for questions with less text and illustrations, the results of image-based question searching are often unsatisfactory.

[0004] To improve this situation, the current common solution is to introduce image search technology to find questions that are highly similar to the illustrations in the uploaded question images. However, some special questions still cannot be found after using image search, and the answers and explanations are not consistent with the uploaded questions. The accuracy of question search is poor, which affects the user experience. Summary of the Invention

[0005] To address the aforementioned technical issues, this disclosure provides a question search method, apparatus, and electronic device to improve the accuracy of question searches and ensure a better user experience.

[0006] In a first aspect, embodiments of this disclosure provide a question search method, including:

[0007] Obtain the original image corresponding to the target question;

[0008] Based on the feature vector of the original image, a search is performed in the original image question search library to obtain the first question search result corresponding to the target question. The first question search result includes multiple first candidate question images.

[0009] If the first candidate question image with the highest similarity to the original image among the plurality of first candidate question images is not a defective question image, and there is at least one defective question image in the first question search results, the original image is processed to obtain an illustration image, wherein the defective question image is an image of a search question in the historical search record whose search result evaluation index value is less than a first preset threshold.

[0010] Extract the feature vector of the illustration image and search it in the illustration title search library to obtain the second title search result corresponding to the target title. The second title search result includes multiple second candidate title images.

[0011] If the second candidate question image with the highest similarity to the illustration image in the second question search results is identified as a defective question image, the second question search results will be returned to the user.

[0012] In some embodiments, the step of extracting the feature vector of the illustration image and searching in the illustration title search library to obtain the second title search result corresponding to the target title includes:

[0013] The illustration image is input into a pre-trained illustration feature extraction model to obtain the feature vector of the illustration image;

[0014] Calculate the similarity between the feature vector of the illustration image and each index in the illustration question search library, wherein the illustration question search library includes multiple non-defective question images and a single original image index and a single illustration index corresponding to each non-defective question image, as well as multiple defective question images and a single original image index and N illustration indices corresponding to each defective question image, where N is an integer greater than 0;

[0015] The question image corresponding to the index whose similarity to the feature vector of the illustration image is greater than a second preset threshold is used as the second question search result corresponding to the target question.

[0016] In some embodiments, the multiple defective question images in the illustration question search library, and the single original image index and N illustration indexes corresponding to each defective question image, are obtained through the following steps:

[0017] Each defective question image in the plurality of defective question images is extracted to obtain a defective question illustration image corresponding to each defective question.

[0018] For each defective title illustration image, N-1 image processing methods are performed to obtain defective title augmented illustration images, which are N-1 processed illustration images corresponding to each defective title image.

[0019] Extract the feature vectors corresponding to the augmented illustrations of the defective title image, and use the feature vectors of the augmented illustrations of the defective title image and the feature vectors of the illustrations of the defective title image as the N illustration indices corresponding to the defective title image.

[0020] In some embodiments, after obtaining the second question search result corresponding to the target question, the method further includes:

[0021] If the second candidate question image with the highest similarity to the illustration image in the second question search results is a non-defect question image, and there is at least one defective question image in the second question search results, a first feature vector is determined for each second candidate question image, wherein the first feature vector is a partial feature vector of the feature vector corresponding to the second candidate question image;

[0022] The similarity between the illustration image and each second candidate title image is calculated based on the first feature vector;

[0023] Based on the similarity between the illustration image and each second candidate question image, the multiple second candidate question images are reordered to obtain the sorted third question search results;

[0024] If the second candidate question image with the highest similarity to the illustration image in the third question search results is identified as a defective question image, the third question search results are returned to the user.

[0025] In some embodiments, determining the first feature vector corresponding to each second candidate question image includes:

[0026] For each second candidate question image, extract the multi-dimensional feature vector corresponding to the second candidate question image;

[0027] The target feature vector with the largest eigenvalue or the first X eigenvalues ​​arranged in descending order among the multidimensional feature vectors is taken as the first feature vector, where X is an integer greater than 0.

[0028] In some embodiments, calculating the similarity between the illustration image and each second candidate title image based on the first feature vector includes:

[0029] Determine the second feature vector corresponding to the first feature vector from the multidimensional feature vector corresponding to the illustration image;

[0030] Calculate the similarity between the first feature vector and the second feature vector.

[0031] In some embodiments, the image processing of the original image to obtain the illustration image includes:

[0032] The original image is preprocessed to obtain a preprocessed image;

[0033] The preprocessed image is input into the inset detection model for inset detection to obtain the inset image to be processed, wherein the inset image to be processed is a target image cropped from the original image, or the inset image to be processed is the original image;

[0034] The illustration image is obtained based on the illustration image to be processed.

[0035] In some embodiments, obtaining the illustration image based on the illustration image to be processed includes:

[0036] The illustration image to be processed is converted into a grayscale image to obtain the first sub-image;

[0037] The illustration image to be processed is subjected to image segmentation processing to eliminate interference information in the illustration image to be processed, and the segmented illustration image is obtained.

[0038] The segmented inset image is converted into a grayscale image to obtain the second sub-image;

[0039] The first sub-image and the second sub-image are merged to obtain the inset image.

[0040] Secondly, embodiments of this disclosure provide a question search device, comprising:

[0041] The acquisition module is used to acquire the original image corresponding to the target question;

[0042] The first search module is used to search in the original image question search library according to the feature vector of the original image to obtain the first question search result corresponding to the target question. The first question search result includes multiple first candidate question images.

[0043] The processing module is used to respond to the fact that the first candidate question image with the highest similarity to the original image among the plurality of first candidate question images is not a defective question image, and there is at least one defective question image in the first question search results, and to perform image processing on the original image to obtain an illustration image, wherein the defective question image is an image of a question to be searched whose search result evaluation index value is less than a first preset threshold in the historical search record.

[0044] The second search module is used to extract the feature vector of the illustration image and search in the illustration title search library to obtain the second title search result corresponding to the target title. The second title search result includes multiple second candidate title images.

[0045] The feedback module is used to respond to the second candidate question image with the highest similarity to the illustration image in the second question search results being a defective question image, and to provide feedback on the second question search results to the user.

[0046] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0047] Memory;

[0048] Processor; and

[0049] Computer programs;

[0050] The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in the first aspect.

[0051] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method described in the first aspect.

[0052] Fifthly, embodiments of this disclosure also provide a computer program product, which includes a computer program or instructions that, when executed by a processor, implement the question search method described above.

[0053] The question search method, apparatus, and electronic device provided in this disclosure improve the accuracy of the question search method by repeatedly confirming the existence of defective questions in the question search results during the question search process and searching separately for the illustration images of the question to be searched. This reduces the probability of search failure when the target question is a defective question and improves the recall rate of defective questions. Attached Figure Description

[0054] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0055] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0056] Figure 1 A flowchart of the question search method provided in this embodiment of the disclosure;

[0057] Figure 2 A schematic diagram illustrating an application scenario provided by an embodiment of this disclosure;

[0058] Figure 3 A flowchart of a question search method provided in this embodiment of the present disclosure;

[0059] Figure 4A flowchart of an illustration image acquisition method provided in this disclosure embodiment;

[0060] Figure 5 A flowchart of a question search method provided in another embodiment of this disclosure;

[0061] Figure 6 A schematic diagram of the structure of the question search device provided in the embodiments of this disclosure;

[0062] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0063] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0064] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0065] This disclosure provides a method for searching for questions, which will be described below with reference to specific embodiments.

[0066] Figure 1 A flowchart illustrating the question search method provided in this embodiment of the disclosure. This method can be applied to... Figure 2 The application scenario shown includes a question-and-answer platform 21 and a smart device 22. Specifically, the smart device 22 can be a terminal, such as a smartphone, PDA, tablet, wearable device with a display screen, desktop computer, laptop computer, all-in-one computer, smart home device, etc. It is understood that the question search method provided in this embodiment can also be applied to other scenarios.

[0067] The following is combined Figure 2 The application scenarios shown are for Figure 1 The problem search method shown is introduced below, and the specific steps of the method are as follows:

[0068] S101. Obtain the original image corresponding to the target question.

[0069] The original image corresponding to the target question can be taken by the user using a smart device and uploaded to the question-and-answer platform. The original image includes both the text and illustrations of the target question.

[0070] S102. Based on the feature vector of the original image, a search is performed in the original image question search library to obtain the first question search result corresponding to the target question. The first question search result includes multiple first candidate question images.

[0071] Specifically, the original image is input into a pre-trained image feature extraction model, which outputs a feature vector of the original image. Based on this feature vector, a search is performed in the original image question search library. The feature vector of the original image is compared with the feature vectors of each question in the library. Multiple candidate question images with high similarity to the feature vector of the original image are selected from each question in the library to form the first question search result.

[0072] In some embodiments, inputting the original image into a pre-trained image feature extraction model and having the image feature extraction model output a feature vector of the original image includes: inputting the original image into the pre-trained image feature extraction model for forward inference and outputting a multi-dimensional feature vector of fixed dimensions, such as a 576-dimensional feature vector. The image feature extraction model can be a MobileNetV3 model, a ResNet18 model, an EfficientNet model, or any other model capable of implementing the above functions; this disclosure does not limit the specific model.

[0073] In some embodiments, the feature vector of the original image is matched and compared with the feature vectors of each question in the original image question search library. The questions in the original image question search library are sorted in descending order of similarity between the feature vector of the original image and the feature vectors of each question in the original image question search library to obtain a sorting result. The top-ranked question images in the sorting result are selected as the first candidate question images.

[0074] In some embodiments, when there is no defective question image in the first question search results, that is, when all the first question search results are non-defective questions, it indicates that the target question is unlikely to be a defective question, and the first question search results can be directly fed back to the user; or, when the first candidate question with the highest similarity to the target question in the first question search results is a defective question, it indicates that the target question has been matched with the corresponding defective question, and the first question search results can still be directly fed back to the user.

[0075] S103. In response to the fact that the first candidate question image with the highest similarity to the original image among the plurality of first candidate question images is not a defective question image, and there is at least one defective question image in the first question search results, the original image is processed to obtain an illustration image, wherein the defective question image is an image of a question to be searched whose search result evaluation index value is less than a first preset threshold in the historical search record.

[0076] A bad case is a question that performs poorly in search history. Specifically, each search in the history has a corresponding search result evaluation metric, which assesses the degree of match between the searched question and the search result. A higher degree of match results in a higher metric; conversely, a lower degree of match results in a lower metric. When a question's metric falls below a first preset threshold, the search performance for that question is considered poor.

[0077] For defective questions, the problem might be due to poor image quality uploaded by the user or unusual illustrations in the question, causing a poor match between the question and its corresponding correct question and answer. Therefore, if at least one defective question image exists in the first search result, and the first candidate question with the highest similarity to the illustration image in the first search result is not a defective question, then it is considered that although the similarity between the defective question and the original image in the first search result is not the highest, it is still highly likely to be the correct search result for the target question. In this case, further processing based on the first search result is needed to determine whether the defective question is indeed the correct search result for the target question.

[0078] As mentioned above, the target question is the question the user wants to search for. The user captures an image of the target question using their smart device, obtains the original image of the target question, and uploads it to the question-and-answer platform. The platform processes the original image of the target question to obtain the corresponding illustration image.

[0079] In some embodiments, the target question includes a question stem and a question illustration, where the illustration includes the question illustration portion of the target question.

[0080] In some embodiments, the target question includes only the question stem text, in which case the above-described illustration includes the question stem text.

[0081] S104. Extract the feature vector of the illustration image and search it in the illustration title search library to obtain the second title search result corresponding to the target title. The second title search result includes multiple second candidate title images.

[0082] The illustrated question search library is built based on the illustrated portions of the question images in the question bank. That is, the images of each question in the question bank are pre-processed to obtain the illustrated images of each question in the question bank, and then the illustrated question search library is built based on the illustrated images of each question in the question bank.

[0083] The system searches the illustration image against an illustration question search database to obtain the second question search result corresponding to the target question. Specifically, the illustration image is output to a pre-trained image feature extraction model, which outputs the feature vector of the illustration image. The feature vector of the illustration image is then used to search the illustration question search database. The feature vector of the illustration image is compared with the feature vectors of each question in the database. Multiple candidate second question images with high similarity to the feature vector of the illustration image are selected from each question in the database to form the second question search result.

[0084] In some embodiments, inputting the illustration image into a pre-trained illustration feature extraction model and having the illustration feature extraction model output a feature vector of the illustration image includes: inputting the illustration image into the pre-trained illustration feature extraction model for forward inference and outputting a multi-dimensional feature vector of fixed dimensions, such as a 576-dimensional feature vector. The image feature extraction model can be a MobileNetV3 model, a ResNet18 model, an EfficientNet model, or any other model capable of implementing the above functions; this disclosure does not limit the specific model.

[0085] In some embodiments, the feature vector of the illustration image is matched and compared with the feature vectors of each question in the illustration question search library. The questions in the illustration question search library are sorted in descending order of similarity between the feature vector of the illustration image and the feature vectors of each question in the illustration question search library to obtain a sorting result. The top-ranked question images in the sorting result are selected as second candidate question images.

[0086] S105. In response to the second candidate question image with the highest similarity to the illustration image in the second question search results being a defective question image, the second question search results are fed back to the user.

[0087] If the second candidate question image with the highest similarity to the illustration image in the search results for the second question is a defective question image, it indicates that the defective question may have been interfered with during the search process with the original image. Processing the original image to obtain the illustration image eliminates some interfering factors, thereby improving the recall rate of the defective question.

[0088] This embodiment of the disclosure obtains the original image corresponding to the target question; searches the original image question search library based on the feature vector of the original image to obtain a first question search result corresponding to the target question, the first question search result including multiple first candidate question images; in response to the first candidate question image with the highest similarity to the original image among the multiple first candidate question images not being a defective question image, and at least one defective question image existing in the first question search result, image processing is performed on the original image to obtain an illustration image, the defective question image being an image of the question to be searched whose search result evaluation index value is less than a first preset threshold in the historical search record; the feature vector of the illustration image is extracted and searched in the illustration question search library to obtain a second question search result corresponding to the target question, the second question search result including multiple second candidate question images; in response to the second question search result having the second candidate question image with the highest similarity to the illustration image being a defective question image, the second question search result is fed back to the user. By repeatedly confirming the existence of defective questions in the question search results during the question search process, and performing a separate search for the illustration image of the question to be searched, the probability of search failure when the target question is a defective question is reduced, the recall rate of defective questions is improved, thereby effectively improving the accuracy of the question search method.

[0089] In addition, this embodiment of the present disclosure, before searching for the illustration image corresponding to the target question in the illustration question search library, first searches for the original image corresponding to the target question in the original image question search library. If there is a defective question image in the search results of the original image question search library, the original image is processed and then reconfirmed, which reduces the amount of computation in the question search process and improves the efficiency of the question search method.

[0090] In some embodiments, when there is no defective question image in the second question search results, that is, when all the second question search results are non-defective questions, it indicates that the target question is unlikely to be a defective question, and the second question search results can be directly fed back to the user; or, when the second candidate question with the highest similarity to the target question in the second question search results is a defective question, it indicates that the target question has been matched with the corresponding defective question, and the second question search results can still be directly fed back to the user.

[0091] In some embodiments, the original image search library includes multiple non-defective question images and a single original image index corresponding to each non-defective question image, and multiple defective question images and M original image indices corresponding to each defective question image, where M is an integer greater than 0.

[0092] Specifically, the single original image index corresponding to a non-defective question image refers to a feature vector corresponding to the original image of that non-defective question image; the M original image indices corresponding to a defective question image refer to the M feature vectors corresponding to the original image of that defective question image.

[0093] Optionally, the M original image indices corresponding to each defective question image in the original image search library are obtained through the following steps: For each defective question image among the plurality of defective question images, perform M-1 types of image processing to obtain an augmented original image of the defective question. The augmented original image of the defective question includes M-1 processed original images of the defective question corresponding to each defective question image; extract the feature vectors corresponding to the augmented original images of the defective question respectively, and use the feature vectors of the augmented original images of the defective question corresponding to each defective question image, as well as the feature vectors of the original images of the defective question corresponding to each defective question image, as the M inset indices corresponding to each defective question image.

[0094] That is, each defective question image in the original image search library has M original image indices, including a feature vector of the original defective question image corresponding to the defective question image, and M-1 feature vectors of the M-1 processed original defective question images corresponding to the augmented original defective question image.

[0095] The feature vector of the original image is matched and compared with the multidimensional feature vectors of each question in the original image question search library. The question image corresponding to the index with a similarity greater than a first preset threshold with the feature vector of the original image is taken as the first question search result corresponding to the target question. Alternatively, the questions in the illustration question search library are sorted in descending order of similarity between the feature vector of the illustration image and the feature vectors of each question in the illustration question search library. The sorted results are obtained, and the top few question images are selected from the candidate results as the first candidate question images.

[0096] In some embodiments, extracting the feature vector of the illustration image and searching it in an illustration question search library to obtain a second question search result corresponding to the target question includes: inputting the illustration image into a pre-trained illustration feature extraction model to obtain the feature vector of the illustration image; calculating the similarity between the feature vector of the illustration image and each index in the illustration question search library, wherein the illustration question search library includes multiple non-defective question images and a single original image index and a single illustration index corresponding to each non-defective question image, as well as multiple defective question images and a single original image index and N illustration indices corresponding to each defective question image, where N is an integer greater than 0; and taking the question image corresponding to the index whose similarity to the feature vector of the illustration image is greater than a second preset threshold as the second question search result corresponding to the target question.

[0097] The original image index is the feature vector of the original image, and the inset index is the feature vector of the inset image.

[0098] Specifically, the single original image index corresponding to a non-defective question image refers to a feature vector of the original image corresponding to that non-defective question image; the single illustration index corresponding to a non-defective question image refers to a feature vector of the illustration image corresponding to that non-defective question image; the single original image index corresponding to a defective question image refers to a feature vector of the original image corresponding to that defective question image; and the N illustration indices corresponding to a defective question image refer to the N feature vectors of the illustration images corresponding to that defective question image, where N is an integer greater than 0.

[0099] Optionally, the single original image index and N illustration indexes corresponding to each defective question image in the illustration question search library are obtained through the following steps: Illustration extraction is performed on each of the multiple defective question images to obtain a defective question illustration image corresponding to each defective question; for each illustration image corresponding to a defective question, N-1 image processing methods are performed to obtain an augmented illustration image for the defective question, wherein the augmented illustration image is one of the N-1 processed illustration images corresponding to each defective question image; feature vectors corresponding to the augmented illustration images are extracted, and the feature vectors of the augmented illustration images corresponding to each defective question image and the corresponding illustration images are used as the N illustration indices corresponding to each defective question image.

[0100] For each defective question, the illustration image is processed, including but not limited to: blurring, random cropping, adding backlighting, adding handwriting, changing the background, etc., or any other image processing method, or an image processing method that imitates the user's actual shooting scene. N-1 methods are selected to process the illustration image corresponding to the defective question to obtain the defective question augmented illustration image. Each defective question augmented illustration image includes N-1 processed illustration images.

[0101] That is, for a defective title image, its corresponding N inset indices refer to the N feature vectors of the inset image corresponding to the defective title image, including the single feature vector corresponding to the inset image of the defective title itself, and the N-1 feature vectors corresponding to the augmented inset image of the defective title.

[0102] The feature vector of the illustration image is matched and compared with the feature vectors of each question in the illustration question search library. The question image corresponding to the index with a similarity greater than a second preset threshold with the feature vector of the illustration image is taken as the second question search result corresponding to the target question. Alternatively, the questions in the illustration question search library are sorted in descending order of similarity between the feature vector of the illustration image and the feature vectors of each question in the illustration question search library. The sorted results are obtained, and the top-ranked question images are selected as the second candidate question images.

[0103] This embodiment of the disclosure increases the likelihood of defective questions being searched by establishing multiple indexes in the original image question search library and the illustration question search library for defective questions, thereby improving the recall rate of defective questions and further enhancing the accuracy of the question search method.

[0104] Figure 3 This is a flowchart illustrating a question search method provided in an embodiment of this disclosure. Figure 3 As shown, the method includes the following steps:

[0105] S301. Obtain the search result for the second question corresponding to the target question.

[0106] Specifically, S301 can be achieved through the steps described in S101 to S104, which will not be repeated here.

[0107] S302. In response to the second candidate question image with the highest similarity to the illustration image in the second question search results being a non-defect question image, and the existence of at least one defective question image in the second question search results, a first feature vector corresponding to each second candidate question image is determined, wherein the first feature vector is a partial feature vector of the feature vector corresponding to the second candidate question image.

[0108] If the second candidate question image with the highest similarity to the illustration image in the second question search results is a non-defective question image, and there is at least one defective question image in the second question search results, further detection of the second question search results is needed to determine whether the defective question is the target question corresponding to the question to be searched.

[0109] Specifically, for each second candidate question image, the multi-dimensional feature vector corresponding to the second candidate question image is extracted; the target feature vector with the largest feature value or the first X feature values ​​arranged from largest to smallest in the multi-dimensional feature vector is taken as the first feature vector, where X is an integer greater than 0.

[0110] Each candidate question image corresponds to a feature vector with multiple dimensions. Each feature vector corresponds to a feature value. The larger the feature value, the higher the importance of the corresponding feature vector to the feature representation of the candidate question image; conversely, the smaller the feature value, the lower the importance of the corresponding feature vector to the feature representation of the candidate question image. Therefore, determining the first feature vector corresponding to each candidate question image is actually equivalent to selecting the feature vector with higher importance to the feature representation of the candidate question image from the multi-dimensional feature vectors of each candidate question image.

[0111] S303. Calculate the similarity between the illustration image and each second candidate title image based on the first feature vector.

[0112] Specifically, a second feature vector corresponding to the first feature vector is determined from the multidimensional feature vector corresponding to the illustration image; the similarity between the first feature vector and the second feature vector is calculated.

[0113] From the multidimensional feature vectors corresponding to the illustrated image, a second feature vector corresponding to the first feature vector is determined. The dimension corresponding to the first feature vector is determined, and a second feature vector with the same dimension as the first feature vector is selected from the multidimensional feature vectors of the illustrated image. It can be understood that the first and second feature vectors have the same dimension and correspond one-to-one.

[0114] In some embodiments, the dimension of the first feature vector and / or the second feature vector can be determined based on the dimension of the multidimensional feature vector of the first candidate question. For example, feature vectors with a predetermined proportion of dimensions from the multidimensional feature vector of the first candidate question can be selected as the first feature vector. Taking a 576-dimensional multidimensional feature vector as an example, 90% of the feature vectors are selected as the first feature vector. Specifically, each feature value in the 576-dimensional multidimensional feature vector is sorted in descending order, and the feature vectors corresponding to the first 90% of the feature values ​​are selected as the first feature vector. The corresponding second feature vector is then determined based on the first feature vector.

[0115] S304. Based on the similarity between the illustration image and each second candidate question image, the multiple second candidate question images are reordered to obtain the sorted third question search results.

[0116] In the above steps, a first feature vector and a corresponding second feature vector with high feature representation importance for the second candidate question image are determined. The similarity between each set of first and second feature vectors is calculated. The multiple second candidate question images are sorted according to the similarity between the first and second feature vectors of each second candidate question image to obtain the sorted third question search results.

[0117] The purpose of this step is to remove weak feature values ​​from the second candidate question image and the illustration image, that is, to remove feature vectors that may interfere with the feature representation of the second candidate question image and the illustration image. Based on the similarity between the first feature vector and the second feature vector, the similarity between the illustration image and each second candidate question image is determined. The multiple second candidate question images in the second question search results are reordered to obtain the sorted third question search results.

[0118] S305. In response to the fact that the second candidate question image with the highest similarity to the illustration image in the third question search results is a defective question image, the third question search results are fed back to the user.

[0119] If the second candidate question image with the highest similarity to the illustration image in the third question search results after reordering is a defective question image, it indicates that the defective question was interfered with by feature vectors with small feature values ​​during the matching process before reordering. After cleaning the feature vectors and reordering, it rose to the position with the highest similarity to the illustration question. At this time, the third question search results after reordering are fed back to the user.

[0120] This embodiment of the disclosure improves the recall rate of defective questions by cleaning and reordering the feature vectors of the second search results, thereby effectively improving the accuracy of the question search method.

[0121] In some embodiments, the step of image processing the original image to obtain an illustration image includes: preprocessing the original image to obtain a preprocessed image; inputting the preprocessed image into an illustration detection model for illustration detection to obtain an illustration image to be processed, wherein the illustration image to be processed is a target image cropped from the original image, or the illustration image to be processed is the original image; and obtaining the illustration image based on the illustration image to be processed.

[0122] Figure 4 A flowchart of an illustration image acquisition method provided in this disclosure embodiment is shown below. Figure 4 As shown, processing the original image to obtain the inset image may include the following steps:

[0123] S401. Preprocess the original image to obtain a preprocessed image.

[0124] Preprocessing the original image involves actions such as scaling it to a fixed size or standardizing the image data. Standardizing the image data involves normalizing the channel values ​​of the original image, specifically normalizing channel values ​​from the 0-255 range to the 0-1 range, thereby reducing the computational load in subsequent image processing.

[0125] S402. Input the preprocessed image into the illustration detection model for illustration detection to obtain the illustration image to be processed, wherein the illustration image to be processed is a target image cropped from the original image, or the illustration image to be processed is the original image.

[0126] The original image is input into the illustration detection model for illustration detection. The trained illustration detection model performs forward inference based on the original image and finally outputs the position coordinate information of the illustration in the original image. Based on the position coordinate information of the illustration, the original image is further cropped to obtain the illustration image to be processed corresponding to the original image.

[0127] In some embodiments, the illustration detection model can be any model capable of performing the above functions, such as the YOLOv5 object detection model, the SSD model, or the R-CenterNet model. This disclosure does not limit the specific model.

[0128] In some embodiments, if the detection result output by the inset detection model is empty, the original image is used as the inset image to be processed, and subsequent processing steps are continued.

[0129] S403. Convert the illustration image to be processed into a grayscale image to obtain the first sub-image.

[0130] A grayscale image is an image that uses different gray levels to represent the proportions of "red, green, and blue" in an image. Converting the illustration image to grayscale yields a first sub-image, which carries complete detail information data from the illustration.

[0131] S404. Perform image segmentation processing on the illustration image to be processed to eliminate interference information in the illustration image to be processed, and obtain the segmented illustration image.

[0132] Image segmentation of the illustration image to be processed essentially involves using edge detection to extract the main subject information and eliminate interference information. Image edge detection significantly reduces the amount of data in the image and removes unimportant background information while preserving important structural properties.

[0133] In some embodiments, image segmentation of the illustration image to be processed can be achieved using the Sobel operator. The Sobel operator is a discrete differential operator used for edge detection. It is understood that other segmentation algorithms or segmentation models can also be used for this step, and the embodiments disclosed herein are not limited thereto.

[0134] In some embodiments, the segmented inset image is a single-channel image.

[0135] S405. Convert the segmented inset image into a grayscale image to obtain the second sub-image.

[0136] The second sub-image is a single-channel grayscale image after segmentation.

[0137] S406. Merge the first sub-image and the second sub-image to obtain the illustration image.

[0138] Based on the above steps, the first sub-image carries complete detailed information data of the illustration to be processed, and the second sub-image carries the main structural features of the illustration to be processed. The first sub-image and the second sub-image are merged to obtain a two-channel image, which reduces one channel compared to the original image (three-channel image), filtering out most of the background information. At the same time, the complete detailed information data in the first sub-image can avoid the information in the second sub-image being missing compared to the illustration image to be processed, ensuring that the feature information in the final illustration image is rich.

[0139] The embodiments disclosed herein ensure good search results based on illustrated images while reducing computational load and further improving the accuracy of the question search method.

[0140] Figure 5 A flowchart of a question search method provided in another embodiment of this disclosure is shown below. Figure 5 As shown, the method includes the following steps:

[0141] S501. Obtain the original image corresponding to the target question.

[0142] S502. Based on the feature vector of the original image, a search is performed in the original image question search library to obtain the first question search result corresponding to the target question. The first question search result includes multiple first candidate question images.

[0143] S503. Determine whether the search result for the first question meets the first preset condition. If yes, execute S505; otherwise, execute S504.

[0144] The first preset condition is that the first candidate question image with the highest similarity to the original image among the plurality of first candidate question images is a non-defect question image, and there is at least one defective question image in the first question search results.

[0145] S504. Return the search results for the first question to the user.

[0146] S505. Perform image processing on the original image to obtain the inset image.

[0147] S506. Extract the feature vector of the illustration image and search it in the illustration title search library to obtain the second title search result corresponding to the target title. The second title search result includes multiple second candidate title images.

[0148] S507. Determine whether the search result for the second question meets the second preset condition. If yes, proceed to S509; otherwise, proceed to S508.

[0149] The second preset condition is that the second candidate question image with the highest similarity to the illustration image in the second question search results is a non-defect question image, and there is at least one defective question image in the first question search results.

[0150] S508. Feedback the search results for the second question to the user.

[0151] S509. Determine the first feature vector corresponding to each second candidate topic image.

[0152] S510. Calculate the similarity between the illustration image and each second candidate title image based on the first feature vector.

[0153] S511. Based on the similarity between the illustration image and each second candidate question image, the multiple second candidate question images are reordered to obtain the sorted third question search results.

[0154] S512. Determine whether the search result for the third question after sorting meets the third preset condition. If yes, execute S513; if no, execute S508.

[0155] The third preset condition is that the third candidate question image ranked first in the search results of the third question after sorting is a defective question image.

[0156] S513. Feedback the search results for the third question after sorting to the user.

[0157] This embodiment of the disclosure effectively improves the recall rate of defective question searches by performing three verifications on the question search results, while ensuring the operational efficiency of the question search method. Furthermore, by first searching within the original image question search library using the original image, most target questions can be processed within the original image question search library; only some potentially defective questions need to be searched in the illustration question search library, thus ensuring the efficiency of the question search method. Further, after obtaining the search results in the illustration question search library, a second feature vector extraction and re-sorting are performed to understand the confidence level of the search results in the illustration question search library, thereby determining the final search results fed back to the user.

[0158] In addition, in the original image question search library and the illustration question search library of this disclosure, multiple indexes are established for defective questions to ensure the recall rate of defective questions as much as possible and effectively improve the accuracy of question search. Figure 6 This is a schematic diagram of the structure of the question search device provided in this embodiment. The question search device can be a question-and-answer platform as described in the above embodiment, or it can be a component or part of that question-and-answer platform. The question search device provided in this embodiment can execute the processing flow provided in the question search method embodiment, such as... Figure 6 As shown, the question search device 60 includes: an acquisition module 61, a first search module 62, a processing module 63, a second search module 64, and a feedback module 65. The acquisition module 61 acquires the original image corresponding to the target question. The first search module 62 searches the original image in a question search library based on its feature vector to obtain a first question search result corresponding to the target question. The first question search result includes multiple first candidate question images. The processing module 63 processes the original image to obtain an illustration image when the first candidate question image with the highest similarity to the original image is not a defective question image, and at least one defective question image exists in the first question search result. The defective question image is an image of a question to be searched whose search result evaluation index value is less than a first preset threshold in historical search records. The second search module 64 extracts the feature vector of the illustration image and searches the illustration question search library to obtain a second question search result corresponding to the target question. The second question search result includes multiple second candidate question images. The feedback module 65 returns the second question search result to the user when the second candidate question image with the highest similarity to the illustration image in the second question search result is a defective question image.

[0159] Optionally, the second search module 64 includes a first acquisition unit 641, a first calculation unit 642, and a first determination unit 643. The first acquisition unit 641 is used to input the illustration image into a pre-trained illustration feature extraction model to obtain the feature vector of the illustration image. The first calculation unit 642 is used to calculate the similarity between the feature vector of the illustration image and each index in the illustration question search library, wherein the illustration question search library includes multiple non-defective question images and a single original image index and a single illustration index corresponding to each non-defective question image, as well as multiple defective question images and a single original image index and N illustration indexes corresponding to each defective question image, where N is an integer greater than 0. The first determination unit 643 is used to take the question image corresponding to the index whose similarity to the feature vector of the illustration image is greater than a second preset threshold as the second question search result corresponding to the target question.

[0160] Optionally, the second search module 64 includes a second acquisition unit 644, used to acquire multiple defective question images in the illustration question search library and a single original image index and N illustration indexes corresponding to each defective question image; specifically, illustration extraction is performed on each of the multiple defective question images to obtain a defective question illustration image corresponding to each defective question; for each defective question illustration image, N-1 image processing methods are performed to obtain a defective question augmented illustration image, wherein the defective question augmented illustration image is N-1 processed illustration images corresponding to each defective question image; feature vectors corresponding to the defective question augmented illustration images are extracted, and the feature vectors of the defective question augmented illustration images corresponding to each defective question image and the defective question illustration images corresponding to the defective question images are used as the N illustration indexes corresponding to the defective question images.

[0161] Optionally, the question search device 60 further includes a determining module 66, a calculating module 67, and a sorting module 68; wherein the determining module 66 is used to determine a first feature vector corresponding to each second candidate question image in the second question search results if the second candidate question image with the highest similarity to the illustration image is a non-defective question image and at least one defective question image exists in the second question search results; wherein the first feature vector is a partial feature vector of the feature vector corresponding to the second candidate question image; the calculating module 67 is used to calculate the similarity between the illustration image and each second candidate question image based on the first feature vector; the sorting module 68 is used to re-sort the multiple second candidate question images based on the similarity between the illustration image and each second candidate question image to obtain a sorted third question search result; the feedback module 65 is also used to provide feedback to the user in response to the second candidate question image with the highest similarity to the illustration image in the third question search results being a defective question image.

[0162] Optionally, the determining module 66 includes an extraction unit 661 and a second determining unit 662; the extraction unit 661 is used to extract the multi-dimensional feature vector corresponding to each second candidate question image; the second determining unit 662 is used to take the target feature vector with the largest feature value or the feature values ​​arranged in descending order X from the multi-dimensional feature vector as the first feature vector, where X is an integer greater than 0.

[0163] Optionally, the calculation module 67 includes a third determining unit 671 and a second calculation unit 672; the third determining unit 671 is used to determine a second feature vector corresponding to the first feature vector from the multidimensional feature vector corresponding to the illustration image; the second calculation unit 672 is used to calculate the similarity between the first feature vector and the second feature vector.

[0164] Optionally, the processing module 63 includes a preprocessing unit 631, a detection unit 632, and a third acquisition unit 633; ​​the preprocessing unit 631 is used to preprocess the original image to obtain a preprocessed image; the detection unit 632 is used to input the preprocessed image into an illustration detection model for illustration detection to obtain an illustration image to be processed, wherein the illustration image to be processed is a target image cropped from the original image, or the illustration image to be processed is the original image; the third acquisition unit 633 is used to obtain the illustration image based on the illustration image to be processed.

[0165] Optionally, the third acquisition unit 633 is specifically used to convert the illustration image to be processed into a grayscale image to obtain a first sub-image; to perform image segmentation processing on the illustration image to be processed to eliminate interference information in the illustration image to be processed to obtain a segmented illustration image; to convert the segmented illustration image into a grayscale image to obtain a second sub-image; and to merge the first sub-image and the second sub-image to obtain the illustration image.

[0166] Figure 6 The question search device in the illustrated embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0167] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. The electronic device may be a device running a question-and-answer platform as described in the above embodiments. The electronic device provided in this embodiment of the disclosure can execute the processing flow provided in the question search method embodiments, such as… Figure 7 As shown, the electronic device 70 includes: a memory 71, a processor 72, a computer program, and a communication interface 73; wherein the computer program is stored in the memory 71 and configured to be executed by the processor 72 using the question search method described above.

[0168] In addition, this disclosure also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the question search method described in the above embodiments.

[0169] Furthermore, this disclosure also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, implement the question search method described above.

[0170] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0171] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A title search method characterized by comprising: The method comprises: obtaining an original image corresponding to a target title; searching in an original title search library according to a feature vector of the original image to obtain a first title search result corresponding to the target title, the first title search result comprising a plurality of first candidate title images; in response to a first candidate title image with the highest similarity to the original image in the plurality of first candidate title images not being a defective title image and at least one defective title image existing in the first title search result, performing image processing on the original image to obtain an illustration image, the defective title image being an image of a to-be-searched title in a historical search record with a search result evaluation index value less than a first preset threshold; extracting a feature vector of the illustration image and searching in an illustration title search library to obtain a second title search result corresponding to the target title, the second title search result comprising a plurality of second candidate title images; in response to a second candidate title image with the highest similarity to the illustration image in the second title search result being a defective title image, feeding back the second title search result to a user.

2. The method of claim 1, wherein, The step of extracting the feature vector of the illustration image and searching in the illustration title search library to obtain the second title search result corresponding to the target title comprises: inputting the illustration image into a pre-trained illustration feature extraction model to obtain the feature vector of the illustration image; calculating the similarity between the feature vector of the illustration image and each index in the illustration title search library, wherein the illustration title search library comprises a plurality of non-defective title images, a single original image index and a single illustration index corresponding to each non-defective title image, and a plurality of defective title images and a single original image index and N illustration indexes corresponding to each defective title image, N being an integer greater than 0; taking a title image corresponding to an index with a similarity greater than a second preset threshold to the feature vector of the illustration image as the second title search result corresponding to the target title.

3. The method of claim 2, wherein, The plurality of defective title images and the single original image index and N illustration indexes corresponding to each defective title image in the illustration title search library are obtained by the following steps: extracting an illustration image corresponding to each of the plurality of defective title images to obtain a defective title image corresponding to each defective title; performing N-1 image processing methods on each defective title image to obtain a defective title augmented illustration image, the defective title augmented illustration image being N-1 processed illustration images corresponding to each defective title image; extracting feature vectors corresponding to the defective title augmented illustration images, and taking the feature vector of each defective title augmented illustration image corresponding to each defective title image and the feature vector of the defective title image corresponding to the defective title image as N illustration indexes corresponding to the defective title image.

4. The method of claim 1, wherein, After the step of obtaining the second title search result corresponding to the target title, the method further comprises: In response to the second candidate question image with the highest similarity to the illustration image in the second question search result being a non-defect question image and at least one defect question image existing in the second question search result, a first feature vector corresponding to each second candidate question image is determined, wherein the first feature vector is part of a feature vector corresponding to the second candidate question image; Similarity of the illustration image to each second candidate question image is calculated according to the first feature vector; In response to the second candidate question image with the highest similarity to the illustration image in the third question search result being a defect question image, the third question search result is fed back to the user. The determination of the first feature vector corresponding to each second candidate question image includes:

5. The method of claim 4, wherein, For each second candidate question image, a multi-dimensional feature vector corresponding to the second candidate question image is extracted; A target feature vector with the largest feature value or the top X feature values from large to small in the multi-dimensional feature vector is arranged as the first feature vector, and X is an integer greater than 0. The calculation of the similarity of the illustration image to each second candidate question image according to the first feature vector includes:

6. The method of claim 5, wherein, A second feature vector corresponding to the first feature vector is determined from the multi-dimensional feature vector corresponding to the illustration image; The similarity of the first feature vector to the second feature vector is calculated. The image processing of the original image to obtain the illustration image includes:

7. The method of claim 1, wherein, The original image is preprocessed to obtain a preprocessed image; The preprocessed image is input into an illustration detection model for illustration detection to obtain a to-be-processed illustration image, wherein the to-be-processed illustration image is a target image cut from the original image, or the to-be-processed illustration image is the original image; The illustration image is obtained based on the to-be-processed illustration image. The obtaining of the illustration image based on the to-be-processed illustration image includes:

8. The method of claim 7, wherein, The to-be-processed illustration image is converted into a grayscale image to obtain a first sub-image; The to-be-processed illustration image is subjected to image segmentation processing to eliminate interference information in the to-be-processed illustration image to obtain a segmented illustration image; The segmented illustration image is converted into a grayscale image to obtain a second sub-image; The first sub-image and the second sub-image are merged to obtain the illustration image. It includes:

9. A title search apparatus characterized by comprising: An acquisition module is configured to acquire an original image corresponding to a target question; A first search module is configured to search in an original question search library according to a feature vector of the original image to obtain a first question search result corresponding to the target question, the first question search result including a plurality of first candidate question images; ​ The processing module is configured to, in response to the first candidate question image with the highest similarity to the original question image not being a defective question image and at least one defective question image existing in the first question search result, perform image processing on the original question image to obtain an illustration image, the defective question image being an image of a to-be-searched question in a historical search record, a search result evaluation index value of which is less than a first preset threshold value. The second search module is configured to extract a feature vector of the illustration image, and perform search in an illustration question search library to obtain a second question search result corresponding to the question title, the second question search result including a plurality of second candidate question images. The feedback module is configured to, in response to the second candidate question image with the highest similarity to the illustration image being a defective question image, feed back the second question search result to a user.

10. An electronic device, comprising: comprise: a memory; a processor; and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1-8.

Images