Question extraction method and device for fingertip question checking, equipment and medium

Abstract

The invention provides a question extraction method and device for fingertip question checking, equipment and a medium. Under the condition that an input image is acquired, a classification result of the input image is determined to determine whether the input image includes a question, double-image convolution processing is performed based on the input image and a first image obtained by performing feature enhancement processing on the input image, and a first feature image is obtained based on the first feature image obtained by the double-image convolution processing. Determining the position pointed by the finger of the user on the input image, and performing double-image convolution processing on the basis of the input image and a second image obtained by performing feature enhancement processing on the input image, so as to determine the position of each question in the input image on the basis of a second feature image obtained by performing double-image convolution processing, therefore, based on the classification result of the input image, the position pointed by the finger of the user on the input image and the position of each question in the input image, extraction of the question image of the finger of the user is realized, and the extraction accuracy of the question image of the finger of the user is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and medium for extracting questions for fingertip query. Background Art

[0002] With the advent of the information age, traditional teaching models can no longer meet people's needs for knowledge acquisition. New teaching models such as online learning and distance learning are gradually emerging, and various intelligent learning tools and intelligent learning platforms are also becoming increasingly popular.

[0003] As an intelligent learning tool, fingertip question checking can quickly scan and identify questions through applications on mobile phones, tablets or learning machines to obtain answers and related learning resources, making the learning process more efficient. With the fingertip question checking tool, users only need to place their fingers on the questions on the teaching aids or test papers, and the fingertip question checking tool will automatically analyze the questions on the user's finger and then answer questions based on the questions on the user's finger. In the process of using the fingertip question checking tool to answer questions, it is very important to correctly extract and identify the questions. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present application provides a question extraction method, device, equipment and medium for fingertip question checking.

[0005] According to a first aspect of an embodiment of the present application, a method for extracting questions for fingertip question checking is provided, the method comprising:

[0006] In response to acquiring an input image, determining a classification result of the input image, the classification result being used to indicate whether the input image includes a title;

[0007] Performing dual-image convolution processing based on the input image and a first image obtained by feature enhancement processing of the input image, so as to determine the position of the user's finger on the input image based on the first feature map obtained by the dual-image convolution processing;

[0008] Performing dual-image convolution processing based on the input image and a second image obtained by feature enhancement processing of the input image, so as to determine the position of each question in the input image based on the second feature map obtained by the dual-image convolution processing;

[0009] Based on the classification result of the input image, the position pointed by the user's finger on the input image, and the position of each question in the input image, a question image of the user's finger is extracted from the input image.

[0010] According to a second aspect of an embodiment of the present application, a question extraction device for fingertip question checking is provided, the device comprising:

[0011] a determination module, configured to determine a classification result of the input image in response to acquiring the input image, wherein the classification result is used to indicate whether the input image includes a title;

[0012] The determination module is further used to perform dual-image convolution processing based on the input image and a first image obtained by feature enhancement processing of the input image, so as to determine the position of the user's finger on the input image based on the first feature map obtained by the dual-image convolution processing;

[0013] The determination module is further configured to perform dual-image convolution processing based on the input image and a second image obtained by performing feature enhancement processing on the input image, so as to determine the position of each question in the input image based on the second feature map obtained by the dual-image convolution processing;

[0014] The question extraction module is used to extract the question image of the user's finger from the input image based on the classification result of the input image, the position of the user's finger on the input image, and the position of each question in the input image.

[0015] According to a third aspect of an embodiment of the present application, a terminal device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the operations performed by the question extraction method for fingertip question checking as described in the first aspect above are implemented.

[0016] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a program is stored, and the program is executed by a processor to perform the operations performed by the question extraction method for fingertip question checking as described in the first aspect above.

[0017] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the operations performed by the question extraction method for fingertip question checking as described in the first aspect above.

[0018] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0019] By determining the classification result of the input image when an input image is acquired to determine whether the input image includes a question, performing dual-image convolution processing based on the input image and a first image obtained by feature enhancement processing of the input image to determine the position of the user's finger on the input image based on the first feature map obtained by the dual-image convolution processing, and performing dual-image convolution processing based on the input image and a second image obtained by feature enhancement processing of the input image to determine the position of each question in the input image based on the second feature map obtained by the dual-image convolution processing, so as to improve the detection accuracy of the position pointed by the user's finger on the input image and the position of each question in the input image by using dual-image convolution, and obtain a more accurate detection result, thereby realizing the extraction of the question image of the user's finger based on the classification result of the input image, the position pointed by the user's finger on the input image, and the position of each question in the input image, and improving the extraction accuracy of the question image of the user's finger.

[0020] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in the specification and constitute a part of this application, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0022] Figure 1 It is a scenario diagram of a method for extracting questions for fingertip question checking according to an exemplary embodiment of the present application.

[0023] Figure 2 It is a flowchart of a method for extracting questions for fingertip question checking according to an exemplary embodiment of the present application.

[0024] Figure 3 It is a flowchart of a method for extracting questions for fingertip question checking according to an exemplary embodiment of the present application.

[0025] Figure 4 It is a block diagram of a question extraction device for fingertip question checking according to an exemplary embodiment of the present application.

[0026] Figure 5 It is a schematic diagram of the structure of a terminal device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as described in detail in the present application.

[0028] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items.

[0029] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0030] The present application provides a question extraction method for checking questions with fingertips. When an image of a user's finger on various educational materials such as teaching aids or test papers is collected, the collected image can be used as an input image. Through the question extraction method for checking questions with user's fingertips provided by the present application, the question image of the user's finger can be extracted from the input image, thereby answering questions for the user based on the extracted question image.

[0031] The question extraction method for fingertip question checking provided in the present application can be applied to a terminal device, and the terminal device may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a dedicated learning machine, etc. The present application does not limit the device type of the terminal device.

[0032] In some embodiments, the terminal device can realize image acquisition through a built-in or external camera to obtain an input image, thereby realizing the extraction of the question image of the user's finger based on the acquired input image. For example, the terminal device is a smart phone, and the smart phone can be connected to a dedicated camera. The dedicated camera needs to be set at a higher position to ensure that the complete image of the user's finger on various educational materials such as teaching aids or test papers can be acquired through the dedicated camera, so that the dedicated camera sends the acquired image to the terminal device, so that the terminal device can use the received image as an input image to realize the extraction of the question image of the user's finger. For another example, the terminal device is a dedicated learning machine, and the dedicated learning machine can be built-in with a camera, and the camera can acquire the complete image of the user's finger on various educational materials such as teaching aids or test papers. When the terminal device acquires the image of the user's finger on various educational materials such as teaching aids or test papers through the built-in camera, the acquired image can be used as the input image to realize the extraction of the question image of the user's finger.

[0033] Take the terminal device as a dedicated learning machine as an example, see Figure 1 , Figure 1 is a schematic diagram of a scenario of a method for extracting questions for fingertip query according to an exemplary embodiment of the present application. Figure 1 As shown, the learning machine 100 may be built with a camera 101, so that the camera 101 can be used to collect images of the user's finger on various educational materials such as teaching aids or test papers. The learning machine 100 also inherits a display screen 102, so that the image collected by the camera 101 can be displayed in real time through the display screen 102, so that the user can adjust the placement of the educational materials in time according to the display of the display screen 102, so as to ensure that the camera 101 can collect a relatively complete image. The learning machine 100 may also include a speaker 103, so that after obtaining the question-answering information corresponding to the question image of the user's finger, the obtained question-answering information can be played through the speaker 103, but it is not limited to this, and the obtained question-answering information can also be displayed through the screen 102.

[0034] The above is only an exemplary description of the application scenario of the present application and does not constitute a limitation on the application scenario of the present application. In more possible implementation methods, the present application can be applied to other scenarios involving fingertip answering.

[0035] The above is a relevant introduction to the application scenarios of the present application. Next, in combination with the embodiments of the present application, the question extraction method for fingertip question checking provided by the present application is described in detail.

[0036] like Figure 2 As shown, Figure 2This is a flowchart of a method for extracting questions for fingertip question checking according to an exemplary embodiment of the present application, the method comprising the following steps:

[0037] Step 201: In response to acquiring an input image, determining a classification result of the input image, where the classification result is used to indicate whether the input image includes a title.

[0038] The input image may be an image captured by a camera. The acquisition method and the acquisition process may refer to the introduction of the above-mentioned relevant scenes, which will not be described in detail here.

[0039] In some embodiments, when an input image is acquired, a neural network-based image classification method can be used to determine whether the input image includes a title, so as to determine the classification result of the input image, but this is not limited to this.

[0040] Step 202: Perform dual-image convolution processing based on the input image and the first image obtained by feature enhancement processing of the input image, so as to determine the position of the user's finger on the input image based on the first feature map obtained by the dual-image convolution processing.

[0041] In some embodiments, dual-image convolution processing may be performed based on the input image and the first image to obtain a first feature map, and then reasoning may be performed based on the first feature map to determine the position of the user's finger on the input image.

[0042] It should be noted that dual-image convolution processing is a processing method that performs convolution operations on two images or two feature maps at the same time. By adopting dual-image convolution processing, the features in the two images can be better extracted to improve the extraction effect of image features, so that the feature map obtained by fusion of the two images can better represent the information required for reasoning.

[0043] Step 203: Perform dual-image convolution processing based on the input image and the second image obtained by feature enhancement processing of the input image, so as to determine the position of each question in the input image based on the second feature map obtained by the dual-image convolution processing.

[0044] In some embodiments, dual-image convolution processing can be performed based on the input image and the second image to obtain a second feature map, and then reasoning is performed based on the second feature map to determine the position of each question in the input image.

[0045] It should be noted that the execution order of the above-mentioned step 201, step 202 and step 203 is not limited. In some embodiments, step 201, step 202 and step 203 can be executed simultaneously. In more possible implementations, step 201, step 202 and step 203 can be executed sequentially in a certain order. This application does not limit the specific execution order.

[0046] Step 204: extract the question image of the user's finger from the input image based on the classification result of the input image, the position of the user's finger on the input image, and the position of each question in the input image.

[0047] By adopting a dual-graph convolution method, the user's finger position and the question position in the input image are determined, so as to improve the detection accuracy of the position pointed by the user's finger on the input image and the position of each question in the input image, and obtain a more accurate detection result, thereby realizing the extraction of the question image of the user's finger based on the classification result of the input image, the position pointed by the user's finger on the input image and the position of each question in the input image, and improving the extraction accuracy of the question image of the user's finger.

[0048] After introducing the basic implementation process of the present application, various non-limiting implementation methods of the present application are specifically introduced below.

[0049] In some embodiments, in order to further improve the accuracy of detecting the position pointed by the user's finger on the input image and the position of each question in the input image, before step 201, the following steps may be further included:

[0050] Step 200: Perform blur detection on the input image to determine whether the input image is blurred.

[0051] In some embodiments, for the above step 200, the input image can be feature enhanced to obtain a third image, and then a dual-image convolution operation can be performed based on the input image and the third image obtained by the feature enhancement process of the input image, so as to determine whether the input image is blurred based on the third feature map obtained by the dual-image convolution operation.

[0052] In a possible implementation, when the input image is subjected to feature enhancement processing to obtain the third image, it can be implemented in the following manner:

[0053] The third Laplace operator is used to perform feature enhancement processing on the input image to obtain a third image. In other words, the third Laplace operator can be used to segment the input image to obtain a feature-enhanced third image.

[0054] Optionally, when using the third Laplacian operator to perform feature enhancement processing on the input image, the input image can be first converted into a grayscale image, and then the third Laplacian operator is used to convolve with the grayscale image obtained by converting the input image to obtain a third convolution feature map, and then the third convolution feature map is superimposed on the input image pixel by pixel to obtain a third image with feature enhancement.

[0055] By using the Laplace operator to achieve image feature enhancement, we can fully utilize the advantage of the Laplace operator in being able to better extract image edge features to improve the image feature enhancement effect.

[0056] In some embodiments, when the third convolution feature map is superimposed pixel by pixel with the input image, it can be implemented according to the following formula (1):

[0057] I3(x,y)=I(x,y)+k3×I3 ’ (x,y) (1)

[0058] Where I3(x,y) represents the pixel value of the pixel located at the xth row and yth column in the third image, I(x,y) represents the pixel value of the pixel located at the xth row and yth column in the input image, and I3 ’ (x, y) represents the pixel value of the pixel located at the xth row and yth column in the third convolution feature map, and k3 represents the third enhancement coefficient.

[0059] Optionally, the third enhancement coefficient may be an empirical value obtained through experiments, or the third enhancement coefficient may be a value based on the contrast, brightness, etc. of the input image, or the third enhancement coefficient may be a value set by the user, etc., which is not limited in the present application.

[0060] Optionally, the grayscale image obtained by converting the input image may be stored, so that the stored grayscale image can be reused when needed in a subsequent processing flow without the need to perform grayscale conversion again.

[0061] Optionally, after acquiring the third image in the above manner, dual-image convolution processing can be directly performed based on the input image and the third image, or the input image and the third image can be further processed to perform dual-image convolution processing based on the further processed input image and the third image.

[0062] In a possible implementation, the input image and the third image may be preprocessed respectively, so that the preprocessed input image and the preprocessed third image are used as two inputs for dual-image convolution processing.

[0063] Optionally, the preprocessing may include image normalization, scaling the image to a fixed size, etc., which is not limited in the present application.

[0064] It should be noted that, regardless of whether the input image and the third image have been further processed, a dual-image convolution operation based on the input image and the third image can be implemented through the following steps to determine whether the input image is blurred based on the third feature map obtained through the dual-image convolution operation:

[0065] Step 2001: Perform convolution processing based on the input image and the third image respectively to obtain a convolution feature map corresponding to the input image and a convolution feature map corresponding to the third image.

[0066] In a possible implementation, convolution processing may be performed on the input image and the third image based on the fifth convolution kernel and the sixth convolution kernel, respectively.

[0067] Optionally, the fifth convolution kernel is the same as the sixth convolution kernel, that is, a shared convolution method can be used to implement dual-image convolution processing based on the input image and the third image; or, the fifth convolution kernel is different from the sixth convolution kernel, that is, a non-shared convolution method can be used to implement dual-image convolution processing based on the input image and the third image.

[0068] It should be noted that by adopting a shared convolution method to perform dual-image convolution processing, the processing of two images can be realized based on more parameters, thereby improving the image processing effect. By adopting an unshared convolution method to perform dual-image convolution processing, the number of parameters required for the processing process can be reduced, thereby improving the image processing efficiency. Therefore, in actual applications, it is possible to choose whether to adopt a shared convolution method or an unshared convolution method to implement dual-image convolution processing based on the input image and the third image according to the needs of the scene, and this application does not limit this.

[0069] By performing convolution on the input image and the third image respectively, feature extraction can be performed on the dual images of the input image and the third image, so that more accurate features can be extracted to better judge whether the image is blurred.

[0070] Step 2002: Perform feature fusion processing based on the convolution feature map corresponding to the input image and the convolution feature map corresponding to the third image to obtain a third feature map.

[0071] Optionally, when the fifth convolution kernel and the sixth convolution kernel are the same, that is, when dual-image convolution processing is performed in a shared convolution manner, feature fusion processing of the convolution feature map corresponding to the input image and the convolution feature map corresponding to the third image can be implemented according to the following formula (2):

[0072] Y3=f(w3(x0+x3)+b3) (2)

[0073] Among them, Y3 represents the third feature map, f is the activation function, w3 is the parameter of the fifth convolution kernel and the sixth convolution kernel, x0 represents the convolution feature map corresponding to the input image, x3 represents the convolution feature map corresponding to the third feature map, and b3 represents the bias value obtained after training.

[0074] Optionally, for the case where the fifth convolution kernel and the sixth convolution kernel are different, that is, for the case where the dual-image convolution processing is performed in a non-shared convolution manner, the feature fusion processing of the convolution feature map corresponding to the input image and the convolution feature map corresponding to the third image can be implemented according to the following formula (3):

[0075] Y3=f(w 3a *x0+w 3b *x3+b3) (3)

[0076] Among them, Y3 represents the third feature map, f is the activation function, and w 3a is the parameter of the fifth convolution kernel, w 3b are the parameters of the sixth convolution kernel, x0 represents the convolution feature map corresponding to the input image, x3 represents the convolution feature map corresponding to the third feature, and b3 represents the bias value obtained after training.

[0077] Step 2003: Determine whether the input image is blurred based on the third feature map.

[0078] In some possible implementations, reasoning may be performed based on the third feature map to determine whether the input image is blurred.

[0079] In some embodiments, the image blur judgment process described in steps 2001 to 2003 above can be implemented through a blur judgment model, that is, the input image and the third image can be input into the blur judgment model to execute steps 2001 to 2003 through the blur judgment model, thereby realizing the judgment of whether the input image is blurred.

[0080] Optionally, the fuzzy judgment model may be a MobileNet Version 3 (MobileNetV3) model, but is not limited thereto.

[0081] In some embodiments, if it is determined that the input image is blurred, a new input image is reacquired to re-perform blur judgment based on the new input image, wherein the method of reacquiring the new input image can be referred to the above introduction and will not be repeated here.

[0082] In some embodiments, if the input image is not blurred, the subsequent processing flow may continue to be executed, that is, the classification result of the input image may be determined through step 201 .

[0083] In some embodiments, for step 201, when determining the classification result of the input image, the input image can be input into an image classification model to output a judgment result of whether the input image includes a question through the image classification model, that is, output the classification result of the input image.

[0084] Optionally, the image classification model may be an 18-layer residual network (Residual Network 18, ResNet18) model, but is not limited thereto.

[0085] The above embodiment is described by taking image classification directly based on an input image as an example. In more possible implementations, the input image may be preprocessed to perform image classification based on the preprocessed input image.

[0086] Optionally, the preprocessing may include image normalization, scaling the image to a fixed size, etc., which is not limited in the present application.

[0087] It should be noted that the classification results of the input image can be two types: the input image includes the title (or the input image is a title image) and the input image does not include the title (or the input image is not a title image).

[0088] In some embodiments, for step 202, feature enhancement processing can be performed on the input image to obtain a first image, and then dual-image convolution processing can be performed based on the input image and the first image obtained by the input image after feature enhancement processing, so as to determine the position of the user's finger on the input image based on the first feature map obtained by the dual-image convolution processing.

[0089] In a possible implementation, when performing feature enhancement processing on the input image to obtain the first image, it can be implemented in the following manner:

[0090] The first Laplacian operator is used to perform feature enhancement processing on the input image to obtain the first image. In other words, the first Laplacian operator can be used to segment the input image to obtain the first image after feature enhancement.

[0091] Optionally, when using the first Laplacian operator to perform feature enhancement processing on the input image, the input image can be first converted into a grayscale image, and then the first Laplacian operator is used to convolve with the grayscale image obtained by converting the input image to obtain a first convolution feature map, and then the first convolution feature map is superimposed with the input image pixel by pixel to obtain a first image with feature enhancement.

[0092] It should be noted that the above embodiment is described by taking the example of converting the grayscale of the input image to obtain the corresponding grayscale image. In more possible implementations, the grayscale image corresponding to the input image has been stored, so the stored grayscale image can also be reused.

[0093] By using the Laplace operator to achieve image feature enhancement, we can fully utilize the advantage of the Laplace operator in being able to better extract image edge features to improve the image feature enhancement effect.

[0094] In some embodiments, when the first convolution feature map is superimposed pixel by pixel with the input image, it can be implemented according to the following formula (4):

[0095] I1(x,y)=I(x,y)+k1×I1 ’ (x,y) (4)

[0096] Where I1(x,y) represents the pixel value of the pixel located at the xth row and yth column in the first image, I(x,y) represents the pixel value of the pixel located at the xth row and yth column in the input image, and I1 ’ (x, y) represents the pixel value of the pixel located at the xth row and yth column in the first convolution feature map, and k1 represents the first enhancement coefficient.

[0097] Optionally, the first enhancement coefficient may be an empirical value obtained through experiments, or the first enhancement coefficient may be a value based on the contrast, brightness, etc. of the input image, or the first enhancement coefficient may be a value set by the user, etc., which is not limited in the present application.

[0098] Optionally, after acquiring the first image in the above manner, dual-image convolution processing can be performed directly based on the input image and the first image, or the input image and the first image can be further processed to perform dual-image convolution processing based on the further processed input image and the first image.

[0099] In a possible implementation, the input image and the first image may be preprocessed respectively, so that the preprocessed input image and the preprocessed first image are used as two inputs for dual-image convolution processing.

[0100] Optionally, the preprocessing may include image normalization, scaling the image to a fixed size, etc., which is not limited in the present application.

[0101] It should be noted that, regardless of whether the input image and the first image have been further processed, when performing dual-image convolution processing based on the input image and the first image obtained by feature enhancement processing of the input image, and determining the position of the user's finger on the input image based on the first feature map obtained by the dual-image convolution processing, it can be achieved by the following steps:

[0102] Step 2021: Perform convolution processing based on the input image and the first image respectively to obtain a convolution feature map corresponding to the input image and a convolution feature map corresponding to the first image.

[0103] In a possible implementation, convolution processing may be performed on the input image and the first image based on the first convolution kernel and the second convolution kernel, respectively.

[0104] Optionally, the first convolution kernel is the same as the second convolution kernel, that is, a shared convolution method can be used to implement dual-image convolution processing based on the input image and the first image; or, the first convolution kernel is different from the second convolution kernel, that is, a non-shared convolution method can be used to implement dual-image convolution processing based on the input image and the first image.

[0105] Optionally, you can choose whether to use a shared convolution method or a non-shared convolution method to implement dual-image convolution processing based on the input image and the first image according to scene requirements, and this application does not limit this.

[0106] By performing convolution on the input image and the first image respectively, feature extraction is achieved on the dual images of the input image and the first image, so that more accurate features can be extracted to better determine the position where the user's finger is pointing on the input image.

[0107] Step 2022: Perform feature fusion processing based on the convolution feature map corresponding to the input image and the convolution feature map corresponding to the first image to obtain a first feature map.

[0108] Optionally, in the case where the first convolution kernel and the second convolution kernel are the same, that is, in the case where the dual-image convolution processing is performed in a shared convolution manner, the feature fusion processing of the convolution feature map corresponding to the input image and the convolution feature map corresponding to the first image can be implemented according to the following formula (5):

[0109] Y1=f(w1(x0+x1)+b1) (5)

[0110] Among them, Y1 represents the first feature map, f is the activation function, w1 is the parameter of the first convolution kernel and the second convolution kernel, x0 represents the convolution feature map corresponding to the input image, x1 represents the convolution feature map corresponding to the first feature map, and b1 represents the bias value obtained after training.

[0111] Optionally, for the case where the first convolution kernel and the second convolution kernel are different, that is, for the case where the dual-image convolution processing is performed in a non-shared convolution manner, the feature fusion processing of the convolution feature map corresponding to the input image and the convolution feature map corresponding to the first image can be implemented according to the following formula (6):

[0112] Y1=f(w 1a *x0+w 1b *x1+b1) (6)

[0113] Among them, Y1 represents the first feature map, f is the activation function, and w 1a is the parameter of the first convolution kernel, w 1bis the parameter of the second convolution kernel, x0 represents the convolution feature map corresponding to the input image, x1 represents the convolution feature map corresponding to the first feature map, and b1 represents the bias value obtained after training.

[0114] Step 2023: Based on the first feature map, determine the position of the user's finger on the input image.

[0115] In some possible implementations, reasoning may be performed based on the first feature map to determine the position where the user's finger is pointing on the input image.

[0116] In some embodiments, the finger detection process described in steps 2021 to 2023 above can be implemented through a finger detection model, that is, the input image and the first image can be input into the finger detection model to execute steps 2021 to 2023 through the finger detection model, thereby determining the position where the user's finger is pointing on the input image.

[0117] Optionally, the finger detection model may be a You Only Look Once Version 5 (YOLOv5) model, but is not limited thereto.

[0118] It should be noted that the specific position of the user's finger on the input image can be determined through the above steps 2021 to 2023. Optionally, the specific position can be achieved by marking a box selection area in the input image, or the specific position can be identified by position coordinates, etc., but is not limited to this. In addition, through the above steps 2021 to 2023, it can also be determined whether the position pointed by the user's finger on the input image is located in a blank area or a non-blank area (that is, the question area), so that when performing finger question detection later, the two types of position information can be combined to realize the judgment of the finger question, thereby improving the efficiency and accuracy of finger question detection.

[0119] In some embodiments, for step 203, feature enhancement processing can be performed on the input image to obtain a second image, thereby performing dual-image convolution processing based on the input image and the second image obtained by the input image after feature enhancement processing, so as to determine the position of each question in the input image based on the second feature map obtained by the dual-image convolution processing.

[0120] In a possible implementation, when performing feature enhancement processing on the input image to obtain the second image, it can be implemented in the following manner:

[0121] The second Laplace operator is used to perform feature enhancement processing on the input image to obtain the second image. In other words, the second Laplace operator can be used to segment the input image to obtain the second image with enhanced features.

[0122] Optionally, when using the second Laplacian operator to perform feature enhancement processing on the input image, the input image can be first converted into a grayscale image, and then the second Laplacian operator is used to convolve with the grayscale image obtained by converting the input image to obtain a second convolution feature map, and then the second convolution feature map is superimposed pixel by pixel with the input image to obtain a second image with feature enhancement.

[0123] It should be noted that the above embodiment is described by taking the example of converting the grayscale of the input image to obtain the corresponding grayscale image. In more possible implementations, the grayscale image corresponding to the input image has been stored, so the stored grayscale image can also be reused.

[0124] By using the Laplace operator to achieve image feature enhancement, we can fully utilize the advantage of the Laplace operator in being able to better extract image edge features to improve the image feature enhancement effect.

[0125] In some embodiments, when the second convolution feature map is superimposed pixel by pixel with the input image, it can be implemented according to the following formula (7):

[0126] I2(x,y)=I(x,y)+k2×I2 ’ (x,y) (7)

[0127] Where I2(x,y) represents the pixel value of the pixel located at the xth row and yth column in the second image, I(x,y) represents the pixel value of the pixel located at the xth row and yth column in the input image, and I2 ’ (x, y) represents the pixel value of the pixel located at the xth row and yth column in the second convolution feature map, and k2 represents the second enhancement coefficient.

[0128] Optionally, the second enhancement coefficient may be an empirical value obtained through experiments, or the second enhancement coefficient may be a value based on the contrast, brightness, etc. of the input image, or the second enhancement coefficient may be a value set by the user, etc., which is not limited in the present application.

[0129] Optionally, after acquiring the second image in the above manner, dual-image convolution processing can be performed directly based on the input image and the second image, or the input image and the second image can be further processed to perform dual-image convolution processing based on the further processed input image and the second image.

[0130] In a possible implementation, the input image and the second image may be preprocessed respectively, so that the preprocessed input image and the preprocessed second image are used as two inputs for dual-image convolution processing.

[0131] Optionally, the preprocessing may include image normalization, scaling the image to a fixed size, etc., which is not limited in the present application.

[0132] It should be noted that, regardless of whether the input image and the second image have been further processed, when performing dual-image convolution processing based on the input image and the second image obtained by feature enhancement processing of the input image, and determining the position of each question in the input image based on the second feature map obtained by the dual-image convolution processing, it can be achieved by the following steps:

[0133] Step 2031: Perform convolution processing based on the input image and the second image respectively to obtain a convolution feature map corresponding to the input image and a convolution feature map corresponding to the second image.

[0134] In a possible implementation, convolution processing may be performed on the input image and the first image based on the third convolution kernel and the fourth convolution kernel, respectively.

[0135] Optionally, the third convolution kernel is the same as the fourth convolution kernel, that is, a shared convolution method can be used to implement dual-image convolution processing based on the input image and the second image; or, the third convolution kernel is different from the fourth convolution kernel, that is, a non-shared convolution method can be used to implement dual-image convolution processing based on the input image and the second image.

[0136] Optionally, you can choose whether to use a shared convolution method or a non-shared convolution method to implement dual-image convolution processing based on the input image and the second image according to the scene requirements, and this application is not limited to this.

[0137] By performing convolution on the input image and the second image respectively, feature extraction can be performed on the input image and the second image dual images, so that more accurate features can be extracted to better determine the position of each question in the input image.

[0138] Step 2032: Perform feature fusion processing based on the convolution feature map corresponding to the input image and the convolution feature map corresponding to the second image to obtain a second feature map.

[0139] Optionally, when the third convolution kernel and the fourth convolution kernel are the same, that is, when dual-image convolution processing is performed in a shared convolution manner, feature fusion processing of the convolution feature map corresponding to the input image and the convolution feature map corresponding to the second image can be implemented according to the following formula (8):

[0140] Y2=f(w2(x0+x2)+b2) (8)

[0141] Among them, Y2 represents the second feature map, f is the activation function, w2 is the parameter of the third convolution kernel and the fourth convolution kernel, x0 represents the convolution feature map corresponding to the input image, x2 represents the convolution feature map corresponding to the second feature map, and b2 represents the bias value obtained after training.

[0142] Optionally, for the case where the third convolution kernel and the fourth convolution kernel are different, that is, for the case where the dual-image convolution processing is performed in a non-shared convolution manner, the feature fusion processing of the convolution feature map corresponding to the input image and the convolution feature map corresponding to the second image can be implemented according to the following formula (9):

[0143] Y2=f(w 2a *x0+w 2b *x2+b2) (9)

[0144] Among them, Y2 represents the second feature map, f is the activation function, and w 2a is the parameter of the third convolution kernel, w 2b are the parameters of the fourth convolution kernel, x0 represents the convolution feature map corresponding to the input image, x2 represents the convolution feature map corresponding to the second feature map, and b2 represents the bias value obtained after training.

[0145] Step 2033: Based on the second feature map, determine the position of each question in the input image.

[0146] In some possible implementations, reasoning may be performed based on the second feature map to determine the location of each question in the input image.

[0147] In some embodiments, the question detection process described in steps 2031 to 2033 above can be implemented through a question detection model, that is, the input image and the second image can be input into the question detection model to execute steps 2031 to 2033 through the question detection model, thereby determining the position of each question in the input image.

[0148] Optionally, the question detection model may be a YOLOv5 model, but is not limited thereto.

[0149] It should be noted that the specific position of each question in the input image can be determined through the above steps 2031 to 2033. Optionally, the specific position can be achieved by marking a box selection area in the input image, or the specific position can be identified by position coordinates, etc., but is not limited thereto.

[0150] After image classification, finger detection and topic detection are implemented through the above embodiments, step 204 may be used to extract the topic image of the user's finger from the input image.

[0151] In some embodiments, for step 204, when extracting the question image of the user's finger from the input image based on the classification result of the input image, the position of the user's finger on the input image, and the position of each question in the input image, for any question in the input image, the following method can be used to determine whether to extract the corresponding image as the question image of the user's finger:

[0152] In one possible implementation, if the classification result of the input image indicates that the input image includes a question, the position indicated by the user's finger on the input image is located in the question area, and the first framed area corresponding to the position indicated by the user's finger on the input image and the second framed area corresponding to the question have an intersecting area and the size of the intersecting area is greater than or equal to the first area threshold, then the image corresponding to the second framed area is cropped out from the input image as the question image of the user's finger.

[0153] In one possible implementation, if the classification result of the input image indicates that the input image includes a question, the position indicated by the user's finger on the input image is located in a blank area, the first framed area corresponding to the position indicated by the user's finger on the input image and the second framed area corresponding to the question have an intersecting area and the size of the intersecting area is greater than or equal to the second area threshold, then the image corresponding to the second framed area is cropped out from the input image as the question image of the user's finger.

[0154] In one possible implementation, if the classification result of the input image indicates that the input image does not include the question, the position indicated by the user's finger on the input image is located in the question area, and the first framed area corresponding to the position indicated by the user's finger on the input image and the second framed area corresponding to the question have an intersecting area and the size of the intersecting area is greater than or equal to the first area threshold, then the image corresponding to the second framed area is cropped out from the input image as the question image of the user's finger.

[0155] In one possible implementation, if the classification result of the input image indicates that the input image does not include the question, the position indicated by the user's finger on the input image is in a blank area, the first framed area corresponding to the position indicated by the user's finger on the input image and the second framed area corresponding to the question have an intersecting area and the size of the intersecting area is greater than or equal to the second area threshold, then the image corresponding to the second framed area is cropped out from the input image as the question image of the user's finger.

[0156] The first area threshold and the second area threshold may be any values, and the first area threshold and the second area threshold may be the same or different, which is not limited in the present application.

[0157] The above embodiment is described by taking the example of successfully extracting the title from the input image. In more possible implementations, it is possible that the title cannot be successfully extracted from the input image.

[0158] In one possible implementation, if the classification result of the input image indicates that the input image includes a question, the position indicated by the user's finger on the input image is located in a blank area, and there is no intersection between the first framed area corresponding to the position indicated by the user's finger on the input image and the second framed area corresponding to the question, a new input image is reacquired.

[0159] In one possible implementation, if the classification result of the input image indicates that the input image does not include the question, the position indicated by the user's finger on the input image is in a blank area, and there is no intersection between the first framed area corresponding to the position indicated by the user's finger on the input image and the second framed area corresponding to the question, a new input image is reacquired.

[0160] The method of re-obtaining a new input image can be found in the previous description and will not be described again here.

[0161] In one possible implementation, if the classification result of the input image indicates that the input image includes a question, the position indicated by the user's finger on the input image is located in the question area, and there is no intersection between the first framed area corresponding to the position indicated by the user's finger on the input image and the second framed area corresponding to the question, then based on the first image and the position indicated by the user's finger on the input image, the question image of the user's finger is re-extracted from the input image.

[0162] In one possible implementation, if the classification result of the input image indicates that the input image does not include the question, the position indicated by the user's finger on the input image is located in the question area, and there is no intersecting area between the first framed area corresponding to the position indicated by the user's finger on the input image and the second framed area corresponding to the question, then based on the first image and the position indicated by the user's finger on the input image, the question image of the user's finger is re-extracted from the input image.

[0163] In some embodiments, when re-extracting the title image of the user's finger from the input image based on the first image and the position pointed by the user's finger on the input image, it can be achieved by the following steps:

[0164] Step 1: binarize the second image to obtain a binarized image.

[0165] In a possible implementation, the Otsu's Thresholding Method (OTSU) method may be used to perform binarization processing on the second image to obtain a binary image.

[0166] It should be noted that the above is only an exemplary binarization processing method. In more possible implementations, other binarization processing methods may also be used, and this application does not limit this.

[0167] Step 2: Determine the target area based on the area information of the connected domain in the binary image and the position pointed by the user's finger on the input image.

[0168] The region information is used to indicate the size of the connected domain, for example, the region information is used to indicate the width and height of the connected domain. It should be noted that, in the case where there are multiple connected domains, the region information can be used to indicate the average size of the multiple connected domains, for example, the region information can be used to indicate the average width and average height of the multiple connected domains.

[0169] In a possible implementation, the size of the target area may be determined based on the size indicated by the area information; thus, the target area is determined based on the size of the target area with the position pointed by the user's finger on the input image as the center.

[0170] Optionally, the width and height of the target area can be determined based on the width and height indicated by the area information, that is, the position pointed by the user's finger on the input image can be taken as the center, and a rectangular area with the same center point as the center, a width equal to the determined width of the target area, and a height equal to the determined height of the target area can be taken as the target area.

[0171] For example, the width of the target area can be 3 times the value of the width indicated by the area information, and the height of the target area can be 3 times the value of the height indicated by the area information. That is, the position indicated by the user's finger on the input image can be taken as the center, and a rectangular area with the same center point as the above center, a width of 3 times the width indicated by the area information, and a height of 3 times the height indicated by the area information can be taken as the target area. That is, if the width indicated by the area information is avgW and the height indicated by the area information is avgH, then the target area is a rectangular area with the position indicated by the user's finger on the input image as the center, a width of 3*avgW, and a height of 3*avgH.

[0172] Step 3: Based on the intersection of the target area and the connected domain, extract the title image of the user's finger from the input image.

[0173] In a possible implementation, if the target area and any connected domain have an intersecting area and the size of the intersecting area is greater than or equal to a third area threshold, it is determined that a question exists at the position indicated by the user's finger on the input image; thus, when a question exists at the position indicated by the user's finger on the input image, an image corresponding to an area centered on the position indicated by the user's finger on the input image and having a target size can be cropped from the input image as the question image of the user's finger, wherein the target size is determined based on the width of the input image and the height of the connected domain.

[0174] Optionally, the target size may include a target width and a target height. The target width may be determined based on the width of the input image, and the target height may be determined based on the height of the connected domain, thereby cropping an image corresponding to a rectangular area centered on the position indicated by the user's finger on the input image, with a height equal to the target height and a width equal to the target width from the input image as the title image of the user's finger.

[0175] For example, the width of the input image can be used as the target width, and 5 times the height of the connected domain can be used as the target height. That is, the position pointed by the user's finger on the input image can be used as the center, and the image corresponding to the rectangular area with the same center point as the above center, the width of the input image, and the height of 5 times the height of the connected domain can be cropped from the input image as the title image of the user's finger.

[0176] After the question image is extracted through the above embodiment, text recognition may be performed based on the question image to obtain the question text information in the question image, thereby outputting question-answering information based on the question text information.

[0177] In a possible implementation, when performing text recognition based on the title image to obtain the title text information in the title image, optical character recognition (OCR) may be performed on the title image to obtain the title text information in the title image.

[0178] In a possible implementation, when outputting question-answer information based on the question text information, it can be implemented through the following steps:

[0179] Step 1: Determine the target topic with the greatest similarity to the topic text information from the target database and determine the similarity between the target topic and the topic text information.

[0180] The target database is used to store multiple questions, and the target database may store question-answering information for each question. Optionally, the question-answering information may include explanation videos and answer analysis, but is not limited thereto.

[0181] Optionally, the text similarity between the title content and the title text information of each title stored in the target database may be determined, and the title with the greatest text similarity is determined as the target title and the text similarity between the title content and the title text information of the target title is obtained.

[0182] Step 2: If the similarity between the target question and the question text information is greater than or equal to the similarity threshold, the question answering information of the target question in the target database is output as the question answering information of the question text information.

[0183] The similarity threshold may be any value, which is not limited in this application.

[0184] Optionally, the output of question-answering information can be achieved by playing a video explanation of the question, or by displaying the answer analysis of the question on the screen, etc., which is not limited in this application.

[0185] Optionally, if the similarity between the target topic and the topic text information is less than a similarity threshold, it is determined that there is no topic related to the topic text information in the target database.

[0186] The general process of the question extraction method for fingertip question checking provided in the embodiment of the present application can be found in Figure 3 , Figure 3 is a flowchart of a method for extracting questions for fingertip query according to an exemplary embodiment of the present application. Figure 3 As shown, when an input image is obtained, the blur judgment process can be used to determine whether the input image is blurry. When the input image is not blurry, the input image can be processed through the question image classification process, the finger detection process and the question detection process. The processing results of the above three processes are combined to realize finger question extraction, and then the questions are answered according to the extracted finger questions. The specific implementation method of each process can be found in the previous text and will not be repeated here.

[0187] Corresponding to the embodiments of the aforementioned method, the present application also provides embodiments of an apparatus and a terminal device to which it is applied.

[0188] like Figure 4 As shown, Figure 4 : is a block diagram of a question extraction device for fingertip question checking according to an exemplary embodiment of the present application, the device comprising:

[0189] A determination module 401 is used to determine a classification result of the input image in response to acquiring the input image, wherein the classification result is used to indicate whether the input image includes a title;

[0190] The determination module 401 is further configured to perform a dual-image convolution process based on the input image and a first image obtained by performing feature enhancement processing on the input image, so as to determine a position pointed by the user's finger on the input image based on the first feature map obtained by the dual-image convolution process;

[0191] The determination module 401 is further configured to perform a dual-image convolution process based on the input image and a second image obtained by performing feature enhancement processing on the input image, so as to determine the position of each question in the input image based on the second feature map obtained by the dual-image convolution process;

[0192] The question extraction module 402 is used to extract the question image of the user's finger from the input image based on the classification result of the input image, the position of the user's finger on the input image, and the position of each question in the input image.

[0193] In some embodiments, the determination module 401, when used to perform dual-image convolution processing based on the input image and the first image obtained by the feature enhancement processing of the input image, so as to determine the position of the user's finger on the input image based on the first feature map obtained by the dual-image convolution processing, is used to:

[0194] Performing convolution processing based on the input image and the first image respectively to obtain a convolution feature map corresponding to the input image and a convolution feature map corresponding to the first image;

[0195] Performing feature fusion processing based on the convolution feature map corresponding to the input image and the convolution feature map corresponding to the first image to obtain the first feature map;

[0196] Based on the first feature map, a position on the input image pointed by the user's finger is determined.

[0197] In some embodiments, the determining module 401, when used to perform convolution processing based on the input image and the first image respectively, is used to:

[0198] Based on a first convolution kernel and a second convolution kernel, performing convolution processing on the input image and the first image respectively;

[0199] The first convolution kernel is the same as the second convolution kernel, or the first convolution kernel is different from the second convolution kernel.

[0200] In some embodiments, the apparatus further comprises:

[0201] The first enhancement module is used to use a first Laplacian operator to perform feature enhancement processing on the input image to obtain the first image.

[0202] In some embodiments, the determination module 401, when used to perform dual-image convolution processing based on the input image and a second image obtained by feature enhancement processing of the input image, so as to determine the position of each question in the input image based on the second feature map obtained by the dual-image convolution processing, is used to:

[0203] Performing convolution processing based on the input image and the second image respectively to obtain a convolution feature map corresponding to the input image and a convolution feature map corresponding to the second image;

[0204] Performing feature fusion processing based on the convolution feature map corresponding to the input image and the convolution feature map corresponding to the second image to obtain the second feature map;

[0205] Based on the second feature map, the position of each question in the input image is determined.

[0206] In some embodiments, the determining module 401, when used to perform convolution processing based on the input image and the second image respectively, is used to:

[0207] Based on a third convolution kernel and a fourth convolution kernel, performing convolution processing on the input image and the second image respectively;

[0208] The third convolution kernel is the same as the fourth convolution kernel, or the third convolution kernel is different from the fourth convolution kernel.

[0209] In some embodiments, the apparatus further comprises:

[0210] The second enhancement module is used to use a second Laplace operator to perform feature enhancement processing on the input image to obtain the second image.

[0211] In some embodiments, the question extraction module 402, when used to extract the question image of the user's finger from the input image based on the classification result of the input image, the position of the user's finger on the input image, and the position of each question in the input image, is used to:

[0212] For any question in the input image, if the classification result of the input image indicates that the input image includes the question, the position pointed by the user's finger on the input image is located in the question area, and the first framed area corresponding to the position pointed by the user's finger on the input image and the second framed area corresponding to the question have an intersecting area and the size of the intersecting area is greater than or equal to a first area threshold, then cropping the image corresponding to the second framed area from the input image as the question image of the user's finger;

[0213] If the classification result of the input image indicates that the input image includes the title, the position pointed by the user's finger on the input image is located in a blank area, the first framed area and the second framed area have an intersection area and the size of the intersection area is greater than or equal to a second area threshold, then cropping an image corresponding to the second framed area from the input image as an image of the title of the user's finger;

[0214] If the classification result of the input image indicates that the input image does not include the title, the position pointed by the user's finger on the input image is located in the title area, the first framed area and the second framed area have an intersection area and the size of the intersection area is greater than or equal to a first area threshold, then cropping an image corresponding to the second framed area from the input image as the title image of the user's finger;

[0215] If the classification result of the input image indicates that the input image does not include the title, the position indicated by the user's finger on the input image is in a blank area, the first framed area and the second framed area have an intersection area and the size of the intersection area is greater than or equal to a second area threshold, then the image corresponding to the second framed area is cropped out from the input image as the title image of the user's finger.

[0216] In some embodiments, the apparatus further comprises:

[0217] A first acquisition module, configured to reacquire a new input image if the classification result of the input image indicates that the input image includes a question, the position pointed to by the user's finger on the input image is located in a blank area, and the first framed area and the second framed area have no intersecting area;

[0218] The title extraction module 402 is further configured to re-extract the title image of the user's finger from the input image based on the first image and the position pointed by the user's finger on the input image, if the classification result of the input image indicates that the input image includes a title, the position pointed by the user's finger on the input image is located in the title area, and the first framed area and the second framed area have no intersecting area;

[0219] The first acquisition module is further configured to reacquire a new input image if the classification result of the input image indicates that the input image does not include the question, the position pointed by the user's finger on the input image is in a blank area, and the first framed area and the second framed area have no intersecting area;

[0220] The title extraction module 402 is further used to re-extract the title image of the user's finger from the input image based on the first image and the position indicated by the user's finger on the input image, if the classification result of the input image indicates that the input image does not include the title, the position indicated by the user's finger on the input image is located in the title area, and the first framed area and the second framed area have no intersection.

[0221] In some embodiments, the title extraction module 402, when used to re-extract the title image of the user's finger from the input image based on the first image and the position of the user's finger on the input image, is used to:

[0222] performing binarization processing on the second image to obtain a binarized image;

[0223] Determine a target area based on the area information of the connected domain in the binary image and the position pointed by the user's finger on the input image;

[0224] Based on the intersection between the target area and the connected domain, a topic image of the user's finger is extracted from the input image.

[0225] In some embodiments, the region information is used to indicate the size of the connected domain;

[0226] The title extraction module 402, when used to determine the target area based on the area information of the connected domain in the binary image and the position pointed by the user's finger on the input image, is used to:

[0227] Determining the size of the target area based on the size indicated by the area information;

[0228] The target area is determined based on the size of the target area, with the position pointed by the user's finger on the input image as the center.

[0229] In some embodiments, the topic extraction module 402, when used to extract the topic image of the user's finger from the input image based on the intersection of the target area and the connected domain, is used to:

[0230] If the target area and the connected domain have an intersecting area and the size of the intersecting area is greater than or equal to a third area threshold, it is determined that a question exists at the position pointed by the user's finger on the input image;

[0231] In the case where there is a question at the position pointed by the user's finger on the input image, an image corresponding to an area centered on the position pointed by the user's finger on the input image and having a target size is cropped out from the input image as the question image of the user's finger, wherein the target size is determined based on the width of the input image and the height of the connected domain.

[0232] In some embodiments, the apparatus further comprises:

[0233] A blur detection module, used for performing blur detection on the input image to determine whether the input image is blurry;

[0234] A second acquisition module, configured to reacquire a new input image if the input image is blurred;

[0235] The determination module 401 is configured to execute the step of determining a classification result of the input image if the input image is not blurred.

[0236] In some embodiments, the blur detection module, when used to perform blur detection on the input image to determine whether the input image is blurry, is used to:

[0237] A dual-image convolution operation is performed based on the input image and a third image obtained by feature enhancement processing of the input image, so as to determine whether the input image is blurred based on a third feature map obtained by the dual-image convolution operation.

[0238] In some embodiments, the blur detection module, when used to perform a dual-image convolution operation based on the input image and a third image obtained by feature enhancement processing of the input image, so as to determine whether the input image is blurred based on the third feature map obtained by the dual-image convolution operation, is used to:

[0239] Performing convolution processing based on the input image and the third image respectively to obtain a convolution feature map corresponding to the input image and a convolution feature map corresponding to the third image;

[0240] Performing feature fusion processing based on the convolution feature map corresponding to the input image and the convolution feature map corresponding to the third image to obtain the third feature map;

[0241] Based on the third feature map, it is determined whether the input image is blurred.

[0242] In some embodiments, the blur detection module, when used to perform convolution processing based on the input image and the third image respectively, is used to:

[0243] Based on the fifth convolution kernel and the sixth convolution kernel, performing convolution processing on the input image and the third image respectively;

[0244] Among them, the fifth convolution kernel is the same as the sixth convolution kernel, or the fifth convolution kernel is different from the sixth convolution kernel.

[0245] In some embodiments, the apparatus further comprises:

[0246] The third enhancement module is used to use a third Laplace operator to perform feature enhancement processing on the input image to obtain the third image.

[0247] In some embodiments, the apparatus further comprises:

[0248] A text recognition module, used for performing text recognition based on the title image to obtain title text information in the title image;

[0249] The output module is used to output question-answering information based on the question text information.

[0250] In some embodiments, the output module, when used to output the question-answering information based on the question text information, is used to:

[0251] Determine a target topic with the greatest similarity to the topic text information from a target database and determine the similarity between the target topic and the topic text information, wherein the target database is used to store a plurality of topics;

[0252] If the similarity between the target topic and the topic text information is greater than or equal to a similarity threshold, the question-answering information of the target topic in the target database is output as the question-answering information of the topic text information.

[0253] In some embodiments, the output module is further configured to determine that there is no topic related to the topic text information in the target database if the similarity between the target topic and the topic text information is less than a similarity threshold.

[0254] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0255] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0256] This application also provides a terminal device, see Figure 5 , Figure 5 1 is a schematic diagram of a terminal device according to an exemplary embodiment of the present application. Figure 5 As shown, the terminal device includes a processor 510, a memory 520 and a network interface 530. The memory 520 is used to store computer instructions that can be run on the processor 510. The processor 510 is used to implement the question extraction method for fingertip answering provided in any embodiment of the present application when executing the computer instructions. The network interface 530 is used to implement input and output functions. In more possible implementations, the terminal device may also include other hardware, which is not limited in the present application.

[0257] The present application also provides a computer-readable storage medium, which can be in various forms. For example, in different examples, the computer-readable storage medium can be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof. In particular, the computer-readable medium can also be paper or other suitable medium capable of printing programs. A computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, the question extraction method for fingertip answering provided in any embodiment of the present application is implemented.

[0258] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the question extraction method for answering questions with fingertips provided in any embodiment of the present application.

[0259] Those skilled in the art will appreciate that one or more embodiments of the present application may be provided as methods, devices, terminal equipment, computer-readable storage media or computer program products. Therefore, one or more embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0260] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiment corresponding to the terminal device, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0261] The above specific embodiments of the present application are described. Other embodiments are within the scope of the present application. In some cases, the actions or steps recorded in the present application can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0262] The embodiments of the subject matter and functional operations described in this application can be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this application and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this application can be implemented as one or more computer programs, that is, one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or control the operation of the data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a question extraction device for fingertip answering. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0263] The processes and logic flows described in this application can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits), and the apparatus can also be implemented as special purpose logic circuits.

[0264] Computers suitable for executing computer programs include, for example, general and / or special microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, the computer does not necessarily have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0265] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0266] Although the application includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in the application can also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment can also be implemented separately in multiple embodiments or implemented with any suitable sub-combination. In addition, although the feature can work in some combinations as described above and even initially claim protection, one or more features from the claimed combination can be removed from the combination in some cases, and the claimed combination can point to a variation of a sub-combination or a sub-combination.

[0267] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0268] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the present application. In some cases, the actions described in the present application can be performed in a different order and still achieve the desired results. In addition, the processing depicted in the accompanying drawings does not necessarily require the specific order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0269] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the inventions claimed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not claimed in the present application. That is, the present application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof.

[0270] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for extracting questions for fingertip checking, characterized in that: The method comprises: In response to acquiring an input image, determining a classification result of the input image, the classification result being used to indicate whether the input image includes a title; Performing dual-image convolution processing based on the input image and a first image obtained by feature enhancement processing of the input image, so as to determine the position of the user's finger on the input image based on the first feature map obtained by the dual-image convolution processing; Performing dual-image convolution processing based on the input image and a second image obtained by feature enhancement processing of the input image, so as to determine the position of each question in the input image based on the second feature map obtained by the dual-image convolution processing; Based on the classification result of the input image, the position pointed by the user's finger on the input image, and the position of each question in the input image, a question image of the user's finger is extracted from the input image.

2. The method according to claim 1, characterized in that The performing dual-image convolution processing based on the input image and the first image obtained by the feature enhancement processing of the input image, so as to determine the position of the user's finger on the input image based on the first feature map obtained by the dual-image convolution processing, includes: Performing convolution processing based on the input image and the first image respectively to obtain a convolution feature map corresponding to the input image and a convolution feature map corresponding to the first image; Performing feature fusion processing based on the convolution feature map corresponding to the input image and the convolution feature map corresponding to the first image to obtain the first feature map; Based on the first feature map, a position on the input image pointed by the user's finger is determined.

3. The method according to claim 2, characterized in that The performing convolution processing based on the input image and the first image respectively includes: Based on a first convolution kernel and a second convolution kernel, performing convolution processing on the input image and the first image respectively; The first convolution kernel is the same as the second convolution kernel, or the first convolution kernel is different from the second convolution kernel.

4. The method according to claim 1, characterized in that: Before performing dual-image convolution processing based on the input image and the first image obtained by feature enhancement processing of the input image to determine the position of the user's finger on the input image based on the first feature map obtained by the dual-image convolution processing, the method further includes: A first Laplacian operator is used to perform feature enhancement processing on the input image to obtain the first image.

5. The method according to claim 1, characterized in that The step of performing dual-image convolution processing based on the input image and a second image obtained by feature enhancement processing of the input image, so as to determine the position of each question in the input image based on the second feature map obtained by the dual-image convolution processing, includes: Performing convolution processing based on the input image and the second image respectively to obtain a convolution feature map corresponding to the input image and a convolution feature map corresponding to the second image; Performing feature fusion processing based on the convolution feature map corresponding to the input image and the convolution feature map corresponding to the second image to obtain the second feature map; Based on the second feature map, the position of each question in the input image is determined.

6. The method according to claim 5, characterized in that The performing convolution processing based on the input image and the second image respectively comprises: Based on a third convolution kernel and a fourth convolution kernel, performing convolution processing on the input image and the second image respectively; The third convolution kernel is the same as the fourth convolution kernel, or the third convolution kernel is different from the fourth convolution kernel.

7. The method according to claim 1, characterized in that Before performing dual-image convolution processing based on the input image and the second image obtained by feature enhancement processing of the input image to determine the position of each question in the input image based on the second feature map obtained by the dual-image convolution processing, the method further includes: A second Laplacian operator is used to perform feature enhancement processing on the input image to obtain the second image.

8. The method according to claim 1, characterized in that The step of extracting the question image of the user's finger from the input image based on the classification result of the input image, the position of the user's finger on the input image, and the position of each question in the input image comprises: For any question in the input image, if the classification result of the input image indicates that the input image includes the question, the position pointed by the user's finger on the input image is located in the question area, and the first framed area corresponding to the position pointed by the user's finger on the input image and the second framed area corresponding to the question have an intersecting area and the size of the intersecting area is greater than or equal to a first area threshold, then cropping the image corresponding to the second framed area from the input image as the question image of the user's finger; If the classification result of the input image indicates that the input image includes the title, the position pointed by the user's finger on the input image is located in a blank area, the first framed area and the second framed area have an intersection area and the size of the intersection area is greater than or equal to a second area threshold, then cropping an image corresponding to the second framed area from the input image as an image of the title of the user's finger; If the classification result of the input image indicates that the input image does not include the title, the position pointed by the user's finger on the input image is located in the title area, the first framed area and the second framed area have an intersection area and the size of the intersection area is greater than or equal to a first area threshold, then cropping an image corresponding to the second framed area from the input image as the title image of the user's finger; If the classification result of the input image indicates that the input image does not include the title, the position indicated by the user's finger on the input image is in a blank area, the first framed area and the second framed area have an intersection area and the size of the intersection area is greater than or equal to a second area threshold, then the image corresponding to the second framed area is cropped out from the input image as the title image of the user's finger.

9. The method according to claim 8, characterized in that The method further comprises any of the following: If the classification result of the input image indicates that the input image includes the question, the position pointed by the user's finger on the input image is located in a blank area, and the first framed area and the second framed area have no intersecting area, then reacquiring a new input image; If the classification result of the input image indicates that the input image includes a title, the position pointed by the user's finger on the input image is located in the title area, and the first framed area and the second framed area have no intersecting area, then based on the first image and the position pointed by the user's finger on the input image, re-extracting the title image of the user's finger from the input image; If the classification result of the input image indicates that the input image does not include the question, the position pointed by the user's finger on the input image is located in a blank area, and the first framed area and the second framed area have no intersection area, then reacquire a new input image; If the classification result of the input image indicates that the input image does not include the title, the position indicated by the user's finger on the input image is located in the title area, and the first framed area and the second framed area have no intersection, then based on the first image and the position indicated by the user's finger on the input image, the title image of the user's finger is re-extracted from the input image.

10. The method according to claim 9, characterized in that The step of re-extracting the title image of the user's finger from the input image based on the first image and the position of the user's finger on the input image comprises: performing binarization processing on the second image to obtain a binarized image; Determine a target area based on the area information of the connected domain in the binary image and the position pointed by the user's finger on the input image; Based on the intersection between the target area and the connected domain, a topic image of the user's finger is extracted from the input image.

11. The method according to claim 10, characterized in that The region information is used to indicate the size of the connected domain; The determining of the target area based on the area information of the connected domain in the binary image and the position pointed by the user's finger on the input image includes: Determining the size of the target area based on the size indicated by the area information; The target area is determined based on the size of the target area, with the position pointed by the user's finger on the input image as the center.

12. The method according to claim 10, characterized in that The step of extracting the title image of the user's finger from the input image based on the intersection of the target area and the connected domain includes: If the target area and the connected domain have an intersecting area and the size of the intersecting area is greater than or equal to a third area threshold, it is determined that a question exists at the position pointed by the user's finger on the input image; In the case where there is a question at the position pointed by the user's finger on the input image, an image corresponding to an area centered on the position pointed by the user's finger on the input image and having a target size is cropped out from the input image as the question image of the user's finger, wherein the target size is determined based on the width of the input image and the height of the connected domain.

13. The method according to claim 1, characterized in that Before determining the classification result of the input image, the method further includes: Performing blur detection on the input image to determine whether the input image is blurred; If the input image is not blurred, performing the step of determining a classification result of the input image; If the input image is blurred, a new input image is acquired again.

14. The method according to claim 13, characterized in that The performing blur detection on the input image to determine whether the input image is blurry includes: A dual-image convolution operation is performed based on the input image and a third image obtained by feature enhancement processing of the input image, so as to determine whether the input image is blurred based on a third feature map obtained by the dual-image convolution operation.

15. The method according to claim 14, characterized in that The step of performing a dual-image convolution operation based on the input image and a third image obtained by feature enhancement processing of the input image, so as to determine whether the input image is blurred based on a third feature map obtained by the dual-image convolution operation, includes: Performing convolution processing based on the input image and the third image respectively to obtain a convolution feature map corresponding to the input image and a convolution feature map corresponding to the third image; Performing feature fusion processing based on the convolution feature map corresponding to the input image and the convolution feature map corresponding to the third image to obtain the third feature map; Based on the third feature map, it is determined whether the input image is blurred.

16. The method according to claim 15, characterized in that The performing convolution processing based on the input image and the third image respectively comprises: Based on the fifth convolution kernel and the sixth convolution kernel, performing convolution processing on the input image and the third image respectively; The fifth convolution kernel is the same as the sixth convolution kernel, or the fifth convolution kernel is different from the sixth convolution kernel.

17. The method according to claim 14, characterized in that Before performing a dual-image convolution operation based on the input image and a third image obtained by feature enhancement processing of the input image to determine whether the input image is blurred based on a third feature map obtained by the dual-image convolution operation, the method further includes: A third Laplacian operator is used to perform feature enhancement processing on the input image to obtain the third image.

18. The method according to claim 1, characterized in that After extracting the question image of the user's finger from the input image based on the classification result of the input image, the position of the user's finger on the input image, and the position of each question in the input image, the method further includes: Performing text recognition based on the title image to obtain title text information in the title image; Based on the question text information, question-answering information is output.

19. The method according to claim 18, characterized in that The outputting of question-answering information based on the question text information includes: Determine a target topic with the greatest similarity to the topic text information from a target database and determine the similarity between the target topic and the topic text information, wherein the target database is used to store a plurality of topics; If the similarity between the target topic and the topic text information is greater than or equal to a similarity threshold, the question-answering information of the target topic in the target database is output as the question-answering information of the topic text information.

20. The method according to claim 19, characterized in that After determining the target topic with the greatest similarity to the topic text information from the target database and determining the similarity between the target topic and the topic text information, the method further includes: If the similarity between the target topic and the topic text information is less than a similarity threshold, it is determined that there is no topic related to the topic text information in the target database.

21. A question extraction device for fingertip question checking, characterized in that: The device comprises: a determination module, configured to determine a classification result of the input image in response to acquiring the input image, wherein the classification result is used to indicate whether the input image includes a title; The determination module is further used to perform dual-image convolution processing based on the input image and a first image obtained by feature enhancement processing of the input image, so as to determine the position of the user's finger on the input image based on the first feature map obtained by the dual-image convolution processing; The determination module is further configured to perform dual-image convolution processing based on the input image and a second image obtained by performing feature enhancement processing on the input image, so as to determine the position of each question in the input image based on the second feature map obtained by the dual-image convolution processing; The question extraction module is used to extract the question image of the user's finger from the input image based on the classification result of the input image, the position of the user's finger on the input image, and the position of each question in the input image.

22. A terminal device, characterized in that: The terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the operations performed by the question extraction method for fingertip question checking as described in any one of claims 1 to 20 when executing the computer program.

23. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, and the program is used by a processor to execute the operations performed by the question extraction method for fingertip question checking as described in any one of claims 1 to 20.

24. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the operations performed by the question extraction method for fingertip question checking as described in any one of claims 1 to 20 are implemented.

Images