Search method and device of geometry problem, electronic equipment and storage medium

Abstract

The present disclosure provides a geometry question search method and device, electronic equipment and storage medium, the method comprising: performing text recognition and geometry detection on a to-be-identified question image respectively to obtain text content and geometry contained in the question image; determining a feature vector corresponding to the geometry contained in the question image based on the geometry; performing text search based on the text content to obtain a text search result, and performing vector search based on the feature vector to obtain a vector search result; and determining a question title matching the question image according to the text search result and the vector search result. In this scheme, when searching for a question for a geometry question, not only the characteristics of the geometry are considered, but also the text contained in the question, which can improve the matching degree between the determined question title and the user-uploaded question and improve the accuracy of the returned question title.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a method, apparatus, electronic device, and storage medium for searching geometric problems. Background Technology

[0002] In the field of intelligent education, scenarios involving the recognition of user-uploaded question images are frequently encountered. For example, in a question-and-answer system, the system recognizes the question image uploaded by the user and returns several questions most similar to the uploaded question, along with their corresponding answers and explanations, for the user's reference.

[0003] Currently, for geometry problems, the common approach is to search the problem bank by image to find a matching problem that matches the user's uploaded geometry problem and return it. However, this search method suffers from low accuracy. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, the present disclosure provides a method, apparatus, electronic device and storage medium for searching geometric problems.

[0005] According to one aspect of this disclosure, a method for searching geometric problems is provided, comprising:

[0006] Text recognition and geometric figure detection are performed on the question image to be identified to obtain the text content and geometric figures contained in the question image;

[0007] Based on the geometric figures, determine the feature vectors corresponding to the geometric figures contained in the problem image;

[0008] A text search is performed based on the text content to obtain text search results, and a vector search is performed based on the feature vector to obtain vector search results. The text search results and the vector search results each include at least one title.

[0009] Based on the text search results and the vector search results, a target question that matches the question image is determined.

[0010] According to another aspect of this disclosure, a search device for geometric problems is provided, comprising:

[0011] The recognition module is used to perform text recognition and geometric detection on the question image to be recognized, so as to obtain the text content and geometric figures contained in the question image;

[0012] The first determining module is used to determine the feature vector corresponding to the geometric shape contained in the question image based on the geometric shape;

[0013] The search module is configured to perform a text search based on the text content to obtain a text search result, and perform a vector search based on the feature vector to obtain a vector search result, wherein the text search result and the vector search result each include at least one title.

[0014] The second determination module is configured to determine a title matching the title image according to the text search result and the vector search result.

[0015] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0016] a processor; and

[0017] a memory storing a program,

[0018] The program includes instructions that, when executed by the processor, cause the processor to perform the search method for a geometry title according to the preceding aspect.

[0019] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform the search method for a geometry title according to the preceding aspect.

[0020] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the search method for a geometry title according to the preceding aspect.

[0021] One or more technical solutions provided in the embodiments of the present disclosure are as follows: text recognition and geometry detection are performed on a title image to be recognized to obtain text content contained in the title image and a geometry, and a feature vector corresponding to the geometry contained in the title image is determined based on the geometry. Then, a text search is performed based on the text content to obtain a text search result, and a vector search is performed based on the feature vector to obtain a vector search result. Furthermore, a title matching the title image is determined according to the text search result and the vector search result. By using the solutions of the present disclosure, the text content corresponding to the title image and the feature vector of the geometry in the title are obtained, and a text search is performed based on the text content and a vector search is performed based on the feature vector. A title matching the title image is finally determined according to the search results of the two parts. Thus, when a title search is performed on a geometry title, not only the features of the geometry are considered, but also the text contained in the title is considered. The matching degree between the determined title and the title uploaded by a user can be improved, and the accuracy of the returned title can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0022] More details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0023] Figure 1 A flow chart of a search method of a geometry problem according to an exemplary embodiment of the present disclosure is shown;

[0024] Figure 2 A flow chart of a search method of a geometry problem according to another exemplary embodiment of the present disclosure is shown;

[0025] Figure 3 A flow chart of a search method of a geometry problem according to yet another exemplary embodiment of the present disclosure is shown;

[0026] Figure 4 A schematic block diagram of a search apparatus of a geometry problem according to an exemplary embodiment of the present disclosure is shown;

[0027] Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0028] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein, but rather, the embodiments are provided so as to more thoroughly and completely understand the present disclosure. It is understood that the drawings and embodiments of the present disclosure are for exemplary purposes only and are not intended to limit the scope of protection of the present disclosure.

[0029] It should be understood that the various steps in the method embodiments of the present disclosure can be performed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit performing the steps shown. The scope of the present disclosure is not limited in this respect.

[0030] The term "comprising" and variations thereof as used herein are open-ended, that is, "comprising but not limited to." The term "based on" is "based, at least in part, on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments." Related terms are defined as follows. It should be noted that the concepts mentioned in the present disclosure are merely used to distinguish different apparatuses, modules or units, and are not intended to limit the order or interdependence of the functions performed by these apparatuses, modules or units.

[0031] It should be noted that the modification of "one", "multiple" mentioned in the present disclosure is illustrative but not restrictive, and those skilled in the art should understand that unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0032] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not used to limit the scope of the messages or information.

[0033] The search method, device, electronic equipment and storage medium for geometry problems provided by the present disclosure are described below with reference to the accompanying drawings.

[0034] Figure 1 A flowchart of a search method for geometry problems according to an example embodiment of the present disclosure is shown, which can be executed by a search device for geometry problems provided by an embodiment of the present disclosure, wherein the device can be implemented by software and / or hardware, and can be generally integrated in an electronic equipment, including a mobile phone, a tablet computer, a wearable device, etc.

[0035] As Figure 1 shown, the search method for geometry problems can include the following steps:

[0036] Step 101, text recognition and geometry detection are performed on the to-be-recognized question image respectively, to obtain the text content and geometry contained in the question image.

[0037] Among them, the question involved in the to-be-recognized question image is a geometry question, the question image contains text and geometry, and the user (for example, a student) can input the question image by shooting and uploading, screenshot, etc.

[0038] In the embodiments of the present disclosure, for the to-be-recognized question image, text recognition and geometry detection can be performed on it respectively, through text recognition on the question image, the text content contained in the question image can be obtained, and through geometry detection on the question image, the geometry contained in the question image can be obtained. Among them, the text content includes the stem information and problem information in the geometry question involved in the question image, and the geometry is an image region containing geometry cut from the question image.

[0039] Exemplarily, when text recognition is performed on the to-be-identified question image, an OCR recognition technology can be used to perform text recognition on the to-be-identified question image. When the OCR recognition is performed on the to-be-identified question image, it can be divided into two parts of text detection and text line recognition. In the text detection part, the to-be-identified question image can be input into a pre-trained text detection model for forward reasoning, and the text detection model outputs the position coordinate information of each text line in the question image. For example, a DBNet text detection model can be used to detect the position coordinate information of the text line in the question image, and this model performs well in both detection efficiency and detection effect in the field of text detection. It can be understood that other text detection models can also be used in actual application, such as EAST model, PSENet model, etc. Then, in the text line recognition part, the corresponding line text image can be cut from the question image according to the position coordinate information of the text line output by the text detection part, and the cut line text image is subjected to text line correction. The corrected text line data is preprocessed, such as scaling to a standard size or blank filling, image data standardization, etc. Then, the preprocessed text line data is input into a pre-trained text recognition model for forward reasoning, and the text recognition model outputs the text information corresponding to the line text image, that is, the text content contained in the question image is obtained. In the embodiment of the present disclosure, a Convolutional Recurrent Neural Network (CRNN) model can be used as a text recognition model, and other text recognition models can also be used in actual application, such as a model introducing Attention, a CRNN+CTC model, etc.

[0040] Exemplarily, when geometric image detection is performed on the to-be-identified question image, the question image can be input into a pre-trained graphic detection model for forward reasoning, and the graphic detection model performs geometric figure detection on the question image, outputs the position coordinate information of the geometric figure in the question image, and further cuts the image region where the geometric figure is located from the question image as the geometric figure according to the obtained position coordinate information of the geometric figure. Wherein, a YOLOv5 target detection model can be used as a graphic detection model, and this model has very high detection efficiency in the field of target detection, and the detection effect is also good. It can be understood that other target detection models can also be used in actual application, such as SSD target detection model, R-CenterNet model, etc. Optionally, when the position coordinate information output by the graphic detection model is empty, the to-be-identified question image can be determined as a geometric figure.

[0041] In an optional embodiment of the present disclosure, before text recognition and geometric figure detection are performed on the to-be-identified question image, the to-be-identified question image can also be preprocessed, which can include but is not limited to scaling the image to a fixed size, image data standardization, etc.

[0042] In step 102, a feature vector corresponding to the geometric figure contained in the question image is determined based on the geometric figure.

[0043] In the embodiments of the present disclosure, after the geometric figure contained in the question image is obtained, the feature vector corresponding to the geometric figure contained in the question image can be determined based on the geometric figure.

[0044] In the determination of the feature vector corresponding to the geometric figure in the question image, a commonly used feature vector acquisition method of an image can be used to acquire the feature vector corresponding to the geometric figure in the geometric figure.

[0045] Exemplarily, a plurality of geometric figures can be collected in advance as training samples, a feature extraction model is trained, and then the geometric figure is input into the trained feature extraction model, the feature extraction model is used to extract features of the geometric figure, and the feature vector corresponding to the geometric figure is obtained.

[0046] In step 103, text search is performed based on the text content to obtain a text search result, and vector search is performed based on the feature vector to obtain a vector search result.

[0047] In the embodiments of the present disclosure, after the text content contained in the question image is obtained, text search can be performed based on the text content to obtain a text search result. The text search result can include one question or a plurality of questions, and the present disclosure does not limit the number of questions contained in the obtained text search result.

[0048] Exemplarily, the text content can be queried in a pre-indexed text search library based on the text content. When queried, the similarity between the text content and each index in the text search library is calculated, and the search results corresponding to each index are sorted in descending order of similarity, and the top N search results are returned as the text search result, where N is a positive integer. The text search library can use an ES (Elasticsearch) library. In the embodiments of the present disclosure, when the text search library is indexed, accurate question text can be used for indexing, and the question text used for indexing is the result of artificial verification to ensure the accuracy of the index. The reason why the OCR text recognition content is not used for indexing is that when the OCR recognized text content has errors, the recognition error can affect the search accuracy, so the accurate text content verified by artificial verification is used for indexing.

[0049] In the embodiments of the present disclosure, after obtaining the feature vector corresponding to the geometric figure in the question image, vector search is performed based on the feature vector to obtain a vector search result. The vector search result can include one question or multiple questions, and the number of questions included in the obtained vector search result is not limited by the present disclosure.

[0050] Exemplarily, the obtained feature vector can be used to query a pre-indexed vector search library. When querying, the similarity between the feature vector and each index in the vector search library is calculated, and the search results corresponding to each index are sorted in descending order of similarity. The top N search results are returned as the vector search result, where N is a positive integer. Each index in the vector search library is represented in the form of a vector. The feature vector of each question in the vector search library can be used as the index of the question. The vector search library can use an ES library or other vector search libraries such as Milvus.

[0051] It can be understood that the number of search results included in the text search result and the vector search result can be the same or different in the embodiments of the present disclosure, and the number of search results included in the text search result and the vector search result is not limited by the present disclosure.

[0052] In step 104, the title of the question image is determined based on the text search result and the vector search result.

[0053] The number of determined question titles can be one or multiple. The determined question title can include only the question itself, or can also include the answer and analysis corresponding to the question, the problem solving approach, etc.

[0054] In the embodiments of the present disclosure, after obtaining the text search result and the vector search result, the title of the question image that matches the question image can be determined based on the text search result and the vector search result. Then, the title of the question image can be returned for user reference.

[0055] Exemplarily, the same question can be selected from the text search result and the vector search result as the title of the question image that matches the question image. For example, if the question identification corresponding to a question appears in both the text search result and the vector search result, the question can be determined as the title of the question image that matches the question image to be identified.

[0056] Exemplarily, one or more questions with the highest similarity can be selected from the text search result and the vector search result as the title of the question image that matches the question image to be identified. It can be understood that the similarity here refers to the similarity calculated when searching the text search library and the vector search library to obtain the text search result and the vector search result.

[0057] The search method for a geometric figure question in the embodiments of the present disclosure can obtain the text content and the geometric figure contained in the question image by respectively performing text recognition and geometric figure detection on the question image to be recognized, determine the feature vector corresponding to the geometric figure contained in the question image based on the geometric figure, then perform text search based on the text content to obtain a text search result, perform vector search based on the feature vector to obtain a vector search result, and finally determine the question title matching the question image according to the text search result and the vector search result. By using the scheme of the present disclosure, the text content corresponding to the question image and the feature vector of the geometric figure in the question are obtained, and the text search is performed based on the text content and the vector search is performed based on the feature vector, and finally the question title matching the question image is determined according to the search results of the two parts. Therefore, when searching for a question for a geometric figure question, not only the features of the geometric figure are considered, but also the text contained in the question is considered, which can improve the matching degree between the determined question title and the question uploaded by the user and improve the accuracy of the returned question title.

[0058] In an optional embodiment of the present disclosure, as shown in Figure 2 based on the embodiment shown in Figure 1 Step 102 can include the following sub-steps:

[0059] Step 201: performing segmentation on the geometric figure based on a preset segmentation algorithm to obtain a segmentation image corresponding to the geometric figure.

[0060] The preset segmentation algorithm can be a Sobel operator. In actual applications, other segmentation algorithms such as a Roberts operator and a Prewitt operator can also be used.

[0061] In the embodiments of the present disclosure, when the feature vector corresponding to the geometric figure is obtained based on the geometric figure, the geometric figure can be segmented by using a preset segmentation algorithm to obtain a segmentation image corresponding to the geometric figure.

[0062] It can be understood that the segmentation image is obtained by segmenting the geometric figure in the geometric figure based on the segmentation algorithm, and the segmentation image contains the geometric figure in the question image to be recognized.

[0063] Step 202: inputting the geometric figure and the segmentation image into a pre-trained feature extraction model to obtain a first feature vector corresponding to the geometric figure and a second feature vector corresponding to the segmentation image.

[0064] The feature extraction model is pre-trained and can be MobileNetV3 model. In actual application, other models can also be used as the feature extraction model, and the selection of the feature extraction model is not limited in the present disclosure.

[0065] In the embodiments of the present disclosure, a plurality of images containing geometric figures can be collected, and the collected images can be segmented by using a segmentation operator to obtain corresponding segmented images. Then, the original images and the corresponding segmented images can be put into the same batch for simultaneous training. The feature extraction model includes an output head that can output a feature vector corresponding to each image in a batch, and the dimensions of the feature vectors are the same. It can be understood that the same batch contains several images, and the output head of the feature extraction model outputs several feature vectors with the same dimension. Therefore, in the embodiments of the present disclosure, the original images and the corresponding segmented images are put into the same batch, and the output head of the feature extraction model outputs the feature vector corresponding to the original image and the feature vector corresponding to the segmented image. By putting the original images and the corresponding segmented images into the same batch for simultaneous training, the feature vector output by the original image through the feature extraction model and the feature vector output by the segmented image data are made to be consistent or as close as possible. In addition, after the image is segmented, a large amount of useless background information in the image is set to 0, reducing the interference of the background information. In order to better extract the features of the geometric figures when training the feature extraction model, the present disclosure increases the segmented image corresponding to the original image as another input of the model on the basis of the original image as the input of the original model. Thus, the feature extraction model can learn to ignore the background information, so that the feature extraction model can extract more effective features during training, improving the generalization ability of the model. Therefore, when using the trained feature extraction model to extract the feature vector of the image, a more accurate feature vector of the geometric figure can be extracted.

[0066] Further, when it is necessary to extract the feature vector of the geometric figure, the obtained geometric figure and the segmented image obtained by segmenting the geometric figure can be input into the trained feature extraction model, and the feature extraction model can extract the features of the geometric figure and the segmented image to output the feature vector corresponding to the geometric figure (referred to as the first feature vector for convenience of description and distinction) and the feature vector corresponding to the segmented image (referred to as the second feature vector for convenience of description and distinction). The dimensions of the first feature vector and the second feature vector are the same.

[0067] Optionally, before the geometric figure and the segmented image obtained by segmenting the geometric figure are input into the feature extraction model to obtain the feature vector, the geometric figure and the segmented image obtained by segmenting the geometric figure can be preprocessed respectively. The preprocessing can include but is not limited to scaling the image data to a fixed size, normalizing the image data, etc.

[0068] As is known, a geometric figure is usually a figure composed of some lines, and in a conventional feature extraction model, a convolution sum is usually used to extract features in an image, and the convolution sum has poor line feature extraction capability, which leads to too much useless information being paid attention to by the extracted image features, and this phenomenon can be seen from the poor line-shaped object segmentation capability of a segmentation network. A conventional segmentation network often fails to segment thin objects such as hair, backpack straps, and ropes, which shows that the conventional convolution sum operation has poor line feature extraction capability, or too much attention is paid to the information of data around the line. The performance on a geometric figure question is that the feature vector extracted by the conventional convolution has poor recall capability for the original question.

[0069] In view of the poor feature extraction capability of a conventional convolution sum for a geometric figure, in an optional embodiment of the present disclosure, an improved scheme is provided, that is, at least one convolution layer of a feature extraction model is improved from a conventional convolution sum (i.e., multiplication plus bias) to a variance convolution (i.e., variance plus bias) on the basis of the original feature extraction model, wherein the calculation process of the variance convolution includes: determining an expected value of the output of the previous layer of neurons based on the output of the previous layer of neurons, and then calculating a convolution result of the variance convolution based on the output of the previous layer of neurons and the corresponding expected value. Since variance is more capable of extracting boundary features in an image than sum, compared with a conventional feature extraction model, the improved feature extraction model of the present disclosure is more capable of extracting the difference of data, amplifying the distribution difference of data, and is more beneficial to subsequent network layers for effective feature extraction, whether it is a geometric figure or a segmentation image, so as to further improve the feature extraction capability of the feature extraction model and enhance the generalization capability of the model.

[0070] In the embodiments of the present disclosure, for at least one convolution layer in the feature extraction model that is improved to a variance convolution, the output thereof is adjusted from the conventional Y=Sum(WX)+b to:

[0071]

[0072] The above formula is a calculation formula of the convolution result of variance convolution. Wherein, Y represents the output of the current convolution layer using variance convolution, X represents the output of the last layer of neurons of the current convolution layer in the feature extraction model, W is the convolution kernel parameter corresponding to X, represents the connection weight between the last layer of neurons of the current convolution layer and the neurons of the current convolution layer, E(X) represents the expected value of the output X of the last layer of neurons; n represents the convolution kernel window area of the current convolution layer, for example, for a 3*3 convolution, n is 9; b represents the bias coefficient of the current convolution layer.

[0073] In view of the fact that the calculation amount of variance is much larger than that of weighted summation, in order to improve the feature extraction capability of the feature extraction model while reducing the calculation amount in the feature extraction process as much as possible, in an optional embodiment of the present disclosure, the first layer of convolution layers of the feature extraction model can be adjusted from the conventional weighted summation mode to the variance convolution mode, and the variance feature map can be obtained after the first layer of convolution layers, and then subsequent processing is performed on the basis of the variance feature map. Therefore, the calculation amount of the whole model is not increased too much, and each layer after the first layer of convolution layers in the model enjoys the strong feature benefit brought by the variance feature map, and the feature extraction operation of each subsequent layer network is based on the variance feature map output by the first layer of convolution layers. After this improvement, the line feature extraction capability of the feature extraction model is greatly improved, and therefore the feature extraction model improved by the embodiment of the present disclosure can extract the line features of the geometric figure, so that the feature vector extracted from the geometric figure question is more robust, and the recall capability of the original question is greatly improved when searching for the question.

[0074] It can be understood that in actual application, the number of layers of variance convolution and the positions of the corresponding layers in the feature extraction model can be flexibly adjusted according to the computing power of the electronic device running the feature extraction model, and the present disclosure does not limit this.

[0075] In step 203, a feature vector corresponding to a geometric figure contained in the question image is determined based on the first feature vector and the second feature vector.

[0076] In the embodiment of the present disclosure, after obtaining the first feature vector corresponding to the geometric figure and the second feature vector corresponding to the segmented image, the feature vector corresponding to the geometric figure contained in the question image can be determined according to the first feature vector and the second feature vector.

[0077] Different calculation methods can be used to determine the feature vector corresponding to the geometric figure contained in the question image according to the first feature vector and the second feature vector.

[0078] As a possible implementation manner, the first feature vector and the second feature vector can be spliced to obtain the feature vector corresponding to the geometric figure contained in the question image. For example, the first feature vector can be spliced after the second feature vector, or the second feature vector can be spliced after the first feature vector to obtain the feature vector corresponding to the geometric figure. When splicing the two vectors, splicing can be performed in the row direction, or splicing can be performed in the column direction.

[0079] In the embodiments of the present disclosure, the geometric figure and the segmentation image obtained by segmenting the geometric figure are input into the feature extraction model, the feature extraction model outputs two feature vectors of the same dimension, and then the two feature vectors are spliced together to form a feature vector. Thus, when performing vector search, the two feature vectors are searched once, and when searching by calculating the vector distance, the total difference between the two feature vectors and the index in the vector search library is compared. Thus, when the feature vector distance between the geometric figure and the question image in the vector search library is far, if the feature vector distance between the segmentation image and the question image in the vector search library is relatively close, the question image in the vector search library can also be recalled. When the feature vector distance between the geometric figure and the question image in the vector search library is relatively close, even if the feature vector distance between the segmentation image and the question image in the vector search library is relatively far, the question image in the vector search library can also be recalled. When the feature vector distance between the geometric figure and the question image in the vector search library is relatively close, and the feature vector distance between the segmentation image and the question image in the vector search library is also relatively close, the question image in the vector search library can also be recalled. Those questions that cannot be recalled are only in the case that the feature vector distances between the geometric figure and the question image in the vector search library and between the segmentation image and the question image in the vector search library are both relatively far. However, since the original image and the segmentation image are optimized in the direction closest to the feature vector of the question image during training of the feature extraction model, the probability of such a case theoretically is the lowest. Therefore, the present disclosure obtains the feature vector corresponding to the geometric figure by splicing, so that similar feature vectors are more similar, and dissimilar feature vectors are farther apart, which is beneficial to recalling the original question and improving the recall rate of the original question when the geometric figure is searched by the feature vector.

[0080] As a possible implementation manner, the first feature vector and the second feature vector can be summed at the corresponding positions to obtain the feature vector corresponding to the geometric figure contained in the question image. For example, the feature value of the first position in the first feature vector is a1, the feature value of the first position in the second feature vector is b1, and the feature value of the first position in the obtained feature vector corresponding to the geometric figure is (a1+b1).

[0081] As a possible implementation manner, the first feature vector and the second feature vector can take maximum values at corresponding positions to obtain feature values at the corresponding positions in the feature vector of the geometric figure, thereby obtaining the feature vector corresponding to the geometric figure contained in the question image. For example, the feature value at the first position in the first feature vector is a1, and the feature value at the first position in the second feature vector is b1, where a1 is greater than b1, and the feature value at the first position in the feature vector corresponding to the geometric figure obtained is a1.

[0082] The search method of the geometric figure question of the embodiments of the present disclosure obtains the feature vector corresponding to the geometric figure contained in the question image by segmenting the geometric figure based on a preset segmentation algorithm to obtain a segmented image corresponding to the geometric figure, inputting the geometric figure and the segmented image into a pre-trained feature extraction model to obtain a first feature vector corresponding to the geometric figure and a second feature vector corresponding to the segmented image, and then determining the feature vector corresponding to the geometric figure contained in the question image based on the first feature vector and the second feature vector. Thus, by increasing the segmented image obtained by segmentation as the input of the feature extraction model together with the geometric figure to obtain the corresponding feature vector, and then obtaining the feature vector corresponding to the geometric figure according to the two feature vectors output by the model, more effective geometric figure features can be obtained, which is conducive to improving the recall rate of the original question during vector search and improving the accuracy of vector search.

[0083] In an optional embodiment of the present disclosure, as shown in Figure 3 on the basis of the foregoing embodiments, step 104 can include the following sub-steps:

[0084] Step 301: obtaining the number of characters of the characters contained in the text content.

[0085] Step 302: in the case where the number of characters is less than a first preset value, determining the vector search result as a question title matching the question image.

[0086] The first preset value can be set according to actual conditions, for example, the first preset value can be set as 20.

[0087] Generally, the number of characters affects the accuracy of the text search result. The more characters contained in the title, the higher the matching degree of the matched text search result and the original title when the text search is performed according to the recognized text content. Conversely, the less characters contained in the title, the lower the matching degree of the matched text search result and the original title, and the lower the reliability of the text search result. Therefore, in the embodiment of the present disclosure, after obtaining the text search result and the vector search result, when determining the title subject matched with the title image, the number of characters contained in the text content can be obtained first, and the obtained number of characters is compared with the first preset value. If the number of characters contained in the text content is less than the first preset value, it indicates that the text contained in the title is less, and the reliability of the text search result is lower. In this case, all the titles contained in the vector search result can be determined as the title subject matched with the title image.

[0088] Further, in an optional embodiment of the present disclosure, when the number of characters contained in the text content is not less than the first preset value, it can be further judged whether the number of characters contained in the text content is greater than a second preset value, and corresponding manners are selected to determine the title subject matched with the title image according to the two cases that the number of characters is greater than the second preset value and is not greater than the second preset value. The second preset value is greater than the first preset value. Thus, as shown in FIG. 1, step 104 can further include the following sub-steps: Figure 3

[0089] Step 303, in the case that the number of characters is not less than the first preset value and the number of characters is not greater than a second preset value, merging the text search result and the vector search result to obtain a search result set, wherein when the text search result and the vector search result are merged, the similarity of the same title is added.

[0090] The second preset value can be set according to actual conditions, for example, the second preset value can be set to 80.

[0091] In the embodiment of the present disclosure, when the number of characters contained in the text content is not less than the first preset value, and the number of characters is not greater than the second preset value, the text search result and the vector search result can be merged to obtain a search result set, wherein when the two search results are merged, the similarity of the same title is added.

[0092] ​That is, when merging the text search result and the vector search result, for one or more topics appearing in both the text search result and the vector search result, the similarity of the same topic in the text search result and the similarity in the vector search result are added to obtain the similarity of the topic in the search result set, and for those topics appearing only in the text search result or the vector search result, the original similarity is retained as the similarity of the topic in the search result set.

[0093] For example, assuming that the text search result includes topic 1 and topic 2, the similarity of topic 1 is S1, and the similarity of topic 2 is S2, the vector search result includes topic 2 and topic 3, the similarity of topic 2 is S4, and the similarity of topic 3 is S3, when merging the text search result and the vector search result, the similarities S2 and S4 corresponding to topic 2 are added to obtain a new similarity corresponding to topic 2, the similarities of topic 1 and topic 3 are retained to obtain the search result set, wherein the search result set includes topic 1, topic 2 and topic 3, and the similarities corresponding to each of the topics are S1, S2+S4 and S3 in turn.

[0094] In step 304, the topics in the search result set are sorted in descending order of similarity, and a preset number of topics in the front of the sorting are obtained as the topic titles matching the topic image.

[0095] In the embodiments of the present disclosure, after obtaining the search result set, the topics in the search result set can be sorted in descending order of similarity according to the similarity corresponding to each topic in the search result set, and a preset number of topics in the front of the sorting are obtained as the topic titles matching the topic image.

[0096] The value of the preset number can be set according to actual needs, and if the number of all topics included in the search result set is less than the preset number, all topics included in the search result set are determined as the topic titles matching the topic image to be recognized.

[0097] In step 305, if the number of characters is greater than the second preset value, the text search result is determined as the topic title matching the topic image.

[0098] In the embodiments of the present disclosure, if the number of characters included in the text content is greater than the second preset value, it indicates that the text included in the topic is more, and the reliability of the text search result is higher. In this case, all topics included in the text search result can be determined as the topic title matching the topic image.

[0099] The search method for the geometry question of the embodiment of the present disclosure, by acquiring the character number of the character contained in the text content, when the character number is less than a first preset value, the vector search result is determined as the target question title matching the question image; when the character number is not less than the first preset value and the character number is not greater than a second preset value, the text search result and the vector search result are merged to obtain a search result set, wherein when the text search result and the vector search result are merged, the similarity of the same question is added, and then the questions in the search result set are sorted in order of high to low similarity, and a preset number of questions in the front are acquired as the target question title matching the question image; when the character number is greater than the second preset value, the text search result is determined as the target question title matching the question image. Thus, when the question contains less text, the vector search result is used as the reference, when the question contains more text, the text search result is used as the reference, and when the number of text characters contained in the question is in the preset range, the question appearing in both the text search result and the vector search result is used as the reference, which is beneficial to improve the accuracy of the returned target question title and improve the matching degree of the target question title and the question image.

[0100] The present disclosure also provides a search device for geometry questions.

[0101] Figure 4 A schematic block diagram of a search device for geometry questions according to an exemplary embodiment of the present disclosure is shown, as shown in the figure, the search device for geometry questions 40 includes an identification module 410, a first determination module 420, a search module 430 and a second determination module 440. Figure 4 The identification module 410 is configured to perform text recognition and geometry detection on the question image to be identified respectively to obtain the text content and the geometry contained in the question image.

[0102] The identification module 410 is configured to perform text recognition and geometry detection on the question image to be identified respectively to obtain the text content and the geometry contained in the question image.

[0103] The first determination module 420 is configured to determine the feature vector corresponding to the geometry contained in the question image based on the geometry.

[0104] The search module 430 is configured to perform text search based on the text content to obtain a text search result, and perform vector search based on the feature vector to obtain a vector search result, wherein the text search result and the vector search result each include at least one question.

[0105] The second determination module 440 is configured to determine the target question title matching the question image according to the text search result and the vector search result.

[0106] Optionally, the first determination module 420 includes:

[0107] a segmentation unit, configured to segment the geometric figure based on a preset segmentation algorithm to obtain a segmentation image corresponding to the geometric figure;

[0108] a feature extraction unit, configured to input the geometric figure and the segmentation image into a pre-trained feature extraction model to obtain a first feature vector corresponding to the geometric figure and a second feature vector corresponding to the segmentation image;

[0109] a determination unit, configured to determine a feature vector corresponding to the geometric figure contained in the question image based on the first feature vector and the second feature vector.

[0110] Optionally, at least one convolution layer of the feature extraction model adopts a variance convolution, and a calculation process of the variance convolution includes:

[0111] determining an expected value of an output of a previous layer of neurons based on the output of the previous layer of neurons;

[0112] calculating a convolution result of the variance convolution based on the output of the previous layer of neurons and the expected value.

[0113] Optionally, the convolution result of the variance convolution is expressed by a formula as

[0114] wherein Y represents an output of a current convolution layer adopting the variance convolution, X represents an output of a previous layer of neurons of the current convolution layer in the feature extraction model, W represents a connection weight between the previous layer of neurons of the current convolution layer and a neuron of the current convolution layer, E(X) represents an expected value of the output X of the previous layer of neurons, n represents a convolution kernel window area of the current convolution layer, and b represents a bias coefficient of the current convolution layer.

[0115] Optionally, the determination unit is further configured to:

[0116] splice the first feature vector and the second feature vector to obtain the feature vector corresponding to the geometric figure contained in the question image.

[0117] Optionally, the second determination module 440 is further configured to:

[0118] obtain a character number of characters contained in the text content;

[0119] in a case where the character number is less than a first preset value, determine the vector search result as a question title matching the question image.

[0120] Optionally, the second determination module 440 is further configured to:

[0121] In a case where the number of characters is not less than the first preset value and the number of characters is not greater than a second preset value, the text search result and the vector search result are combined to obtain a search result set, wherein, when the text search result and the vector search result are combined, the similarities of the same questions are added.

[0122] The questions in the search result set are sorted according to the order from high to low of the similarities, and a preset number of questions in front of the sorting are obtained as the question titles matched with the question image.

[0123] Optionally, the second determining module 440 is further configured to:

[0124] In a case where the number of characters is greater than the second preset value, the text search result is determined as the question title matched with the question image.

[0125] The search device for the geometric figure question provided in the embodiments of the present disclosure can execute any search method for the geometric figure question applicable to the electronic device provided in the embodiments of the present disclosure, and has the function modules and beneficial effects corresponding to the execution method. The contents not described in detail in the device embodiments of the present disclosure can be referred to the description in any method embodiment of the present disclosure.

[0126] The exemplary embodiments of the present disclosure further provide an electronic device, including at least one processor, and a memory connected with the at least one processor in communication. The memory stores a computer program capable of being executed by the at least one processor, and the computer program is used to make the electronic device execute the search method for the geometric figure question according to the embodiments of the present disclosure when being executed by the at least one processor.

[0127] The exemplary embodiments of the present disclosure further provide a non-transitory computer readable storage medium storing a computer program, wherein the computer program is used to make the computer execute the search method for the geometric figure question according to the embodiments of the present disclosure when being executed by the processor of the computer.

[0128] The exemplary embodiments of the present disclosure further provide a computer program product including a computer program, wherein the computer program is used to make the computer execute the search method for the geometric figure question according to the embodiments of the present disclosure when being executed by the processor of the computer.

[0129] Reference Figure 5The present invention describes a structural block diagram of an electronic device 1100 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0130] like Figure 5 As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. The RAM 1103 may also store various programs and data required for the operation of the device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0131] Multiple components in electronic device 1100 are connected to I / O interface 1105, including: input unit 1106, output unit 1107, storage unit 1108, and communication unit 1109. Input unit 1106 can be any type of device capable of inputting information to electronic device 1100. Input unit 1106 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 1107 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1108 may include, but is not limited to, disk and optical disk. Communication unit 1109 allows electronic device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0132] The computing unit 1101 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs various methods and processes described above. For example, in some embodiments, the search method for geometry problems can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 1100 via the ROM 1102 and / or the communication unit 1109. In some embodiments, the computing unit 1101 can be configured to perform the search method for geometry problems by any other suitable means, such as by means of firmware.

[0133] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a function / operation specified in the flowchart and / or block diagram. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, and partially on a remote machine or entirely on a remote machine or server.

[0134] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0135] As used in this disclosure, the terms "machine-readable medium" and "computer- readable medium" refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0136] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0137] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0138] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

Claims

1. A method for searching geometric figure problems, wherein, The method includes: Text recognition and geometric figure detection are performed on the question image to be identified to obtain the text content and geometric figures contained in the question image; Based on the geometric figure, the feature vectors corresponding to the geometric figures contained in the question image are determined. The feature vectors are obtained by concatenating a first feature vector obtained by feature extraction of the geometric figure and a second feature vector obtained by feature extraction of the segmented image. The segmented image is obtained by segmenting the geometric figures in the geometric figure. A text search is performed based on the text content to obtain text search results, and a vector search is performed based on the feature vector to obtain vector search results. The text search results and the vector search results each include at least one title. Based on the text search results and the vector search results, a target question that matches the question image is determined.

2. The method for searching geometric problems as described in claim 1, wherein, The step of determining the feature vector corresponding to the geometric shape contained in the question image based on the geometric shape includes: The geometric figure is segmented based on a preset segmentation algorithm to obtain a segmented image corresponding to the geometric figure; The geometric figure and the segmented image are input into a pre-trained feature extraction model to obtain a first feature vector corresponding to the geometric figure and a second feature vector corresponding to the segmented image. Based on the first feature vector and the second feature vector, the feature vectors corresponding to the geometric figures contained in the question image are determined.

3. The method for searching geometric problems as described in claim 2, wherein, At least one convolutional layer of the feature extraction model employs variance convolution, and the calculation process of the variance convolution includes: Based on the output of the previous layer of neurons, determine the expected value of the output of the previous layer of neurons; Based on the output of the previous layer neuron and the expected value, the convolution result of the variance convolution is calculated.

4. The method for searching geometric problems as described in claim 3, wherein, The convolution result of the variance convolution is expressed by the formula: Where Y represents the output of the current convolutional layer using the variance convolution, X represents the output of the neuron in the previous layer of the current convolutional layer in the feature extraction model, W represents the connection weight between the neuron in the previous layer of the current convolutional layer and the neuron in the current convolutional layer, E(X) represents the expected value of the output X of the neuron in the previous layer, n represents the area of ​​the convolutional kernel window of the current convolutional layer, and b represents the bias coefficient of the current convolutional layer.

5. The method for searching geometric problems as described in any one of claims 1-4, wherein, The step of determining the target question matching the question image based on the text search results and the vector search results includes: Obtain the number of characters contained in the text content; If the number of characters is less than a first preset value, the vector search result is determined as the target question that matches the question image.

6. The method for searching geometric problems as described in claim 5, wherein, The step of determining the target question matching the question image based on the text search results and the vector search results further includes: When the number of characters is not less than the first preset value and the number of characters is not greater than the second preset value, the text search results and the vector search results are merged to obtain a search result set, wherein the similarity of the same questions is added when merging the text search results and the vector search results; The questions in the search results set are sorted in descending order of similarity, and a preset number of questions at the top of the list are selected as target questions that match the question image.

7. The method for searching geometric problems as described in claim 6, wherein, The step of determining the target question matching the question image based on the text search results and the vector search results further includes: If the number of characters is greater than the second preset value, the text search result is determined as the target question that matches the question image.

8. A search device for geometric problems, wherein, The device includes: The recognition module is used to perform text recognition and geometric detection on the question image to be recognized, so as to obtain the text content and geometric figures contained in the question image; The first determining module is used to determine the feature vector corresponding to the geometric shape contained in the question image based on the geometric shape. The feature vector is obtained by concatenating a first feature vector obtained by feature extraction of the geometric shape and a second feature vector obtained by feature extraction of the segmented image. The segmented image is obtained by segmenting the geometric shape in the geometric shape. The search module is used to perform a text search based on the text content to obtain a text search result, and to perform a vector search based on the feature vector to obtain a vector search result, wherein the text search result and the vector search result each include at least one title; The second determining module is used to determine the target question that matches the question image based on the text search results and the vector search results.

9. An electronic device, comprising: processor; as well as Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the search method for geometric problems according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the search method for geometric problems according to any one of claims 1-7.

Images