Page detection method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to the technical field of image processing, and particularly provides a page detection method and device, electronic equipment, a storage medium and a program product. The page detection method comprises the steps that an object detection model is adopted, object detection is carried out according to a to-be-detected image, and a target area where an indication object is located is obtained; adopting an indication confirmation model to obtain an indication confirmation result of the target area according to the to-be-detected image, the target area and the first limit matrix; if the indication confirmation result shows that the target area is valid, performing page detection according to the to-be-detected image by adopting a page detection model to obtain a target page area; and according to the target area and the target page area, cutting the to-be-detected image to obtain a page area image. Therefore, the page area pointed by the indication object can be accurately detected, so that the page area can be identified, and misjudgment of page detection is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a method, device, electronic device, storage medium and program product for book page detection. Background Art

[0002] With the rapid development of smart devices, finger and book page detection is usually performed through a learning device (such as a learning machine) in order to detect the content of the page pointed to by the user's finger in the book.

[0003] In the prior art, after the finger is detected in the image, the page area is usually detected and outputted through the image.

[0004] However, using this method, there are usually some misjudgments in the book page detection results. For example, an invalid book page area that is not pointed to by the user's finger may be output. Summary of the invention

[0005] The purpose of the embodiments of the present disclosure is to provide a method, device, electronic device, storage medium and program product for book page detection, so as to reduce misjudgment of book page detection during book page detection.

[0006] On the one hand, an embodiment of the present disclosure provides a method for detecting book pages, including:

[0007] An object detection model is used to perform object detection according to the image to be detected, and a target area where an indicated object is located is obtained; the indicated object is used to indicate a book page;

[0008] The indication confirmation model is adopted to obtain the indication confirmation result of the target area according to the image to be detected, the target area and the first limiting matrix; the indication confirmation result indicates whether the target area is valid; the first limiting matrix is ​​generated according to the limiting factor of the limiting function and is used to assist in judging the indication confirmation result;

[0009] If the confirmation result indicates that the target area is valid, a book page detection model is used to perform book page detection according to the image to be detected to obtain the target book page area;

[0010] The image to be detected is cropped according to the target area and the target book page area to obtain a book page area image.

[0011] In one embodiment, the indication confirmation model includes a first classification network, a first restriction network, and a first output network;

[0012] The indication confirmation model is adopted to obtain the indication confirmation result of the target area according to the image to be detected, the target area and the first limiting matrix, including:

[0013] According to the target area, a local image containing the indicated object is cut out from the image to be detected;

[0014] Inputting the local image into a first classification network to obtain a first image feature;

[0015] Inputting the first limiting matrix into the first limiting network to obtain a first limiting feature;

[0016] The first image feature and the first limiting feature are input into the first output network to obtain an indication confirmation result.

[0017] In one implementation, inputting the first image feature and the first limiting feature into the first output network to obtain an indication confirmation result includes:

[0018] Obtaining a first limiting value based on the first image feature and the first limiting feature through the first output network;

[0019] If the first limiting value is determined to be the first target limiting value corresponding to the first limiting matrix, then determining the indication confirmation result is that the target area is valid;

[0020] Otherwise, the determination indicates that the result of the determination is that the target area is invalid.

[0021] In one embodiment, before using the object detection model to perform object detection according to the image to be detected and obtaining the target area of ​​the target indicated object, the method includes:

[0022] The following steps are executed repeatedly until the image to be detected is obtained:

[0023] Acquire the collected image;

[0024] A fuzzy detection model is used to perform fuzziness detection according to the collected image and the second limiting matrix to obtain a fuzzy detection result; the fuzzy detection result indicates whether the collected image is clear; the second limiting matrix is ​​generated according to the limiting factor of the limiting function and is used to assist in judging the fuzzy detection result;

[0025] If the blur detection result shows that the image is clear, the captured image is determined as the image to be detected; otherwise, the captured image is discarded.

[0026] In one implementation, the fuzzy detection model includes a second classification network, a second restriction network, and a second output network;

[0027] The fuzzy detection model is used to perform fuzziness detection according to the collected image and the second limiting matrix to obtain fuzzy detection results, including:

[0028] Inputting the collected image into a second classification network to obtain a second image feature;

[0029] Inputting the second limiting matrix into the second limiting network to obtain a second limiting feature;

[0030] The second image feature and the second limiting feature are input into the second output network to obtain a blur detection result.

[0031] In one implementation, the second image feature and the second limiting feature are input into a second output network to obtain a blur detection result, including:

[0032] Obtaining a second limiting value according to the second image feature and the second limiting feature through a second output network;

[0033] If the second limiting value is determined to be the second target limiting value corresponding to the second limiting matrix, then the blur detection result is determined to be image blur;

[0034] Otherwise, the blur detection result is determined to be that the image is clear.

[0035] In one implementation, the image to be detected is cropped according to the target area and the target book page area to obtain the book page area image, including:

[0036] Obtaining the indicated position of the indicated object in the target area also output by the object detection model;

[0037] Determine the overlap area between the indicated position and the target page area;

[0038] Determine the proportion of the overlapping area in the target area;

[0039] If the proportion is higher than the set proportion threshold, a page area image containing the target page area is cropped from the image to be detected.

[0040] On the one hand, an embodiment of the present disclosure provides a device for detecting book pages, including:

[0041] A detection unit, used to use an object detection model to perform object detection according to the image to be detected, and obtain a target area where an indicated object is located; the indicated object is used to indicate a book page;

[0042] A confirmation unit, used to obtain an indication confirmation result of the target area according to the image to be detected, the target area and the first limiting matrix by adopting the indication confirmation model; the indication confirmation result indicates whether the target area is valid; the first limiting matrix is ​​generated according to the limiting factor of the limiting function and is used to assist in judging the indication confirmation result;

[0043] an obtaining unit, configured to, if the indication confirmation result indicates that the target area is valid, use a book page detection model to perform book page detection according to the image to be detected to obtain a target book page area;

[0044] The cropping unit is used to crop the image to be detected according to the target area and the target book page area to obtain a book page area image.

[0045] In one embodiment, the indication confirmation model includes a first classification network, a first restriction network, and a first output network;

[0046] The confirmation unit is used to:

[0047] According to the target area, a local image containing the indicated object is cut out from the image to be detected;

[0048] Inputting the local image into a first classification network to obtain a first image feature;

[0049] Inputting the first limiting matrix into the first limiting network to obtain a first limiting feature;

[0050] The first image feature and the first limiting feature are input into the first output network to obtain an indication confirmation result.

[0051] In one implementation, the confirmation unit is used to:

[0052] Obtaining a first limiting value based on the first image feature and the first limiting feature through the first output network;

[0053] If the first limiting value is determined to be the first target limiting value corresponding to the first limiting matrix, then determining the indication confirmation result is that the target area is valid;

[0054] Otherwise, the determination indicates that the result of the determination is that the target area is invalid.

[0055] In one embodiment, the detection unit is further used for:

[0056] The following steps are executed repeatedly until the image to be detected is obtained:

[0057] Acquire the collected image;

[0058] A fuzzy detection model is used to perform fuzziness detection according to the collected image and the second limiting matrix to obtain a fuzzy detection result; the fuzzy detection result indicates whether the collected image is clear; the second limiting matrix is ​​generated according to the limiting factor of the limiting function and is used to assist in judging the fuzzy detection result;

[0059] If the blur detection result shows that the image is clear, the captured image is determined as the image to be detected; otherwise, the captured image is discarded.

[0060] In one implementation, the fuzzy detection model includes a second classification network, a second restriction network, and a second output network;

[0061] The detection unit is also used to:

[0062] Inputting the collected image into a second classification network to obtain a second image feature;

[0063] Inputting the second limiting matrix into the second limiting network to obtain a second limiting feature;

[0064] The second image feature and the second limiting feature are input into the second output network to obtain a blur detection result.

[0065] In one embodiment, the detection unit is further used for:

[0066] Obtaining a second limiting value according to the second image feature and the second limiting feature through a second output network;

[0067] If the second limiting value is determined to be the second target limiting value corresponding to the second limiting matrix, then the blur detection result is determined to be image blur;

[0068] Otherwise, the blur detection result is determined to be that the image is clear.

[0069] In one implementation, the cutting unit is used to:

[0070] Obtaining the indicated position of the indicated object in the target area also output by the object detection model;

[0071] Determine the overlap area between the indicated position and the target page area;

[0072] Determine the proportion of the overlapping area in the target area;

[0073] If the proportion is higher than the set proportion threshold, a page area image containing the target page area is cropped from the image to be detected.

[0074] In one aspect, an electronic device is provided in an embodiment of the present disclosure, including:

[0075] Processor; and

[0076] The memory stores computer instructions, wherein the computer instructions are used to enable the processor to execute the steps of the method provided in any of the various optional implementations of the above-mentioned book page detection.

[0077] On the one hand, an embodiment of the present disclosure provides a computer-readable storage medium storing computer instructions for causing a computer to execute steps of a method provided in any of the above-mentioned various optional implementations of book page detection.

[0078] On the one hand, a computer program product is provided in an embodiment of the present disclosure, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the steps of the method provided in any of the various optional implementations of any of the above-mentioned book page detection.

[0079] The method for book page detection in the embodiment of the present disclosure, after detecting the existence of an indicating object in the image to be detected, also confirms the validity of the indicating object (i.e., whether the indicating object points to a book page). Only when the indicating object is valid, will the book page area image be detected and output, thereby outputting the book page area image pointed to by the user, so as to identify the content of the book page pointed to by the user, thereby reducing misjudgment of book page detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0081] Figure 1 It is a flow chart of a method for detecting book pages in an embodiment of the present disclosure.

[0082] Figure 2 It is a structural diagram of a fuzzy detection model in an embodiment of the present disclosure.

[0083] Figure 3 It is a detailed implementation flow chart of a book page detection in an embodiment of the present disclosure.

[0084] Figure 4 It is a structural block diagram of a device for detecting book pages in an embodiment of the present disclosure.

[0085] Figure 5 It is a structural schematic diagram of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0086] The technical solution of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described implementation is a part of the implementation of the present disclosure, rather than all the implementations. Based on the implementation in the present disclosure, all other implementations obtained by ordinary technicians in the field without creative work are within the scope of protection of the present disclosure. In addition, the technical features involved in the different implementations of the present disclosure described below can be combined with each other as long as they do not conflict with each other.

[0087] In the related art, the learning machine usually detects the page area through the captured image after detecting the finger. However, in some application scenarios, there may be multiple pages, and it is usually difficult to accurately output the page area indicated by the user. In addition, the user's finger may not be pointing on the page, and an invalid page area is usually detected. Therefore, there are usually some misjudgments in the page detection.

[0088] Based on the defects of the above-mentioned related technologies, the embodiments of the present disclosure provide a method, device, electronic device, storage medium and program product for book page detection, aiming to reduce misjudgment of book page detection during book page detection.

[0089] A method for detecting book pages is provided in an embodiment of the present disclosure. The method can be applied to electronic devices. The present disclosure does not limit the type of electronic devices. The method can be any type of device suitable for implementation, such as a terminal device and a server, etc. The present disclosure will not elaborate on this.

[0090] See also Figure 1 FIG. 1 is a flow chart of a method for detecting book pages in an embodiment of the present disclosure. Figure 1 The method is described below. The specific implementation process of the method is as follows:

[0091] Step 101: using an object detection model, performing object detection according to the image to be detected, and obtaining a target area indicating where the object is located.

[0092] Optionally, the object detection model can be built based on a target detection (You Only Look Once version5, yolov5) model, or can be built using other models, without limitation here.

[0093] The indicating object is used to indicate a book page, for example, the indicating object can be a user's finger, or a tool used to point to a book page, such as a stylus. The target area is the area where the indicating object is located in the image to be detected, and can be a polygonal area, such as a rectangular area.

[0094] Furthermore, the indicated position of the indicated object in the target area may also be obtained, for example, the indicated object is a finger of the user, and the indicated position is the area where the fingertip of the user is located.

[0095] In one implementation, the image to be detected is input into an object detection model to obtain a target area where the indicated object is located and an indicated position of the indicated object in the target area.

[0096] Furthermore, before executing step 101, the image to be processed may be preprocessed. Optionally, the preprocessing may include at least one of the following: scaling to a fixed size and image normalization.

[0097] Furthermore, before executing step 101, image screening may be performed to screen out clear images to be detected, thereby reducing the failure rate of subsequent book page detection.

[0098] In one implementation, when screening images, the following steps may be performed cyclically until an image to be detected is obtained:

[0099] S1011: Acquire the collected image.

[0100] Optionally, the captured image may be captured by a camera device provided in the electronic device.

[0101] Furthermore, the captured image may be preprocessed to obtain a preprocessed captured image.

[0102] S1012: Using a fuzzy detection model, perform fuzziness detection according to the collected image and the second limiting matrix to obtain a fuzzy detection result.

[0103] Combine the following Figure 2 The fuzzy detection model is explained. Figure 2 A structural diagram of a fuzzy detection model. Figure 2 In the embodiment, the blur detection model includes a second classification network, a second restriction network and a second output network. Exemplarily, the collected image is input into the second classification network to obtain the second image feature; the second restriction matrix is ​​input into the second restriction network to obtain the second restriction feature; the second image feature and the second restriction feature are input into the second output network to obtain the blur detection result.

[0104] Among them, the blur detection result indicates whether the captured image is clear; the second limiting matrix is ​​generated according to the limiting factor of the limiting function. The second classification network and the second limiting network are both used for feature extraction, and both can be built based on the mobilenetv3 model. Optionally, they can also be built based on other models, which is not limited here. Exemplarily, based on the mobilenetv3 model, the second classification network and the second limiting network are constructed respectively, and the penultimate layers in the second classification network and the second limiting network are merged and connected to the second output network for full connection.

[0105] Optionally, the size of the second limiting matrix and the acquired image may be the same or different. Each value in the second limiting matrix may be a limiting factor of the limiting function, and the second limiting matrix may be input into the transformation function to obtain a new second limiting matrix, and the new second limiting matrix may be used as the input of the second limiting network.

[0106] Exemplarily, when the second image feature and the second limiting feature are input into the second output network and the blur detection result is obtained, the following steps may be adopted:

[0107] Through the second output network, a second limiting value is obtained according to the second image feature and the second limiting feature; if the second limiting value is determined to be the second target limiting value corresponding to the second limiting matrix, the blur detection result is determined to be a blurred image; otherwise, the blur detection result is determined to be a clear image.

[0108] The limiting factor set of the limiting function includes multiple limiting factors, and the second limiting matrix can be generated based on one or more limiting factors. The limiting factors used in the second limiting matrix can be randomly selected, or can be the limiting factors with the highest accuracy rate tested, or can be selected in other ways, which are not limited here.

[0109] For example, in a high recall application scenario, a second limiting matrix can be generated based on a randomly selected limiting factor, and if the second limiting value corresponding to the determined limiting factor is the same as the second target limiting value, the image is determined to be blurred, otherwise, the image is determined to be clear. In this way, the accuracy of image blur detection can be improved.

[0110] In order to improve the accuracy of blur detection, multiple limiting factors can also be used to generate a second limiting matrix. If the second limiting values ​​and the second target limiting values ​​corresponding to the respective limiting factors are the same or the same proportion is higher than the specified proportion, the image is determined to be blurred, otherwise, the image is determined to be clear. In this way, it can be applied to scenes with high requirements for blur accuracy.

[0111] In the embodiment of the present application, the loop function is introduced into the neural network, and the auxiliary network is used for feature extraction to improve the generalization ability of the network. When the loop function is applied in the neural network, the loop function is called a restriction function, so that the restricted network (such as the second restricted network) in the model can be predicted according to the input and output of the loop function, and the domain of the loop function is the set of restricted factors. The output value of the loop function is the restricted value corresponding to the restricted factor.

[0112] The domain and range of the cyclic function have the same number of values ​​and are finitely discrete. The numbers in the domain are all different. Any number in the domain can be called the first number, and the cyclic function can start from the first number and get the second number (the next number). Then the second number is calculated to get the third number, and so on. The function is calculated from the last number that is not involved in the calculation to get the first number mentioned above, and the cyclic calculation of the function is completed.

[0113] In practical applications, the cyclic function can be used to represent the transformation of the five elements of gold, wood, water, fire, and earth, that is, gold generates water, water generates wood, wood generates fire, fire generates earth, and earth generates gold. The domain and range of the cyclic function are all the five elements of gold, wood, water, fire, and earth.

[0114] Exemplarily, in an embodiment of the present application, the restriction function may adopt a loop function: f(x)=x%5+1. The domain and range of the loop function are both (1, 2, 3, 4, 5). Wherein, x is an input variable, f(1)=2, f(2)=3, f(3)=4, f(4)=5, f(5)=1. When training the blur detection model, a second restriction matrix may be constructed based on the restriction factor of the restriction function, and training data may be constructed based on the image sample, the second restriction matrix and its corresponding restriction value, and the model may be trained based on the training data. Then the blurred image sample is trained with restriction factor 1 to obtain an output value of 2, trained with restriction factor 2 to obtain an output value of 3, trained with restriction factor 3 to obtain an output value of 4, trained with restriction factor 4 to obtain an output value of 5, and trained with restriction factor 5 to obtain an output value of 1. A clear image sample, i.e. a clear image, is trained with any restriction factor to obtain an output value of 0.

[0115] S1013: If the blur detection result shows that the image is clear, the captured image is determined as the image to be detected; otherwise, the captured image is discarded and S1011 is executed.

[0116] In this way, clear images to be detected can be screened out to reduce the failure rate of detection.

[0117] Step 102: using the indication confirmation model, according to the image to be detected, the target area and the first limiting matrix, obtaining an indication confirmation result of the target area.

[0118] The indication confirmation result indicates whether the target area is valid, that is, the validity of the target area is determined according to whether the indication object points to the page area. The first restriction matrix is ​​generated according to the restriction factor of the restriction function and is used to assist in determining the indication confirmation result.

[0119] Among them, the indication confirmation model includes a first classification network, a first limitation network and a first output network. The first classification network and the first limitation network are both used for feature extraction, and both can be constructed based on a lightweight neural network (ghostnet) or in other ways, which are not limited here.

[0120] In one implementation, when executing step 102, the following steps may be used:

[0121] S1021: According to the target area, a local image containing the indicated object is cropped from the image to be detected.

[0122] Exemplarily, if the target area is a rectangular area, a partial image containing only the rectangular area may be cropped.

[0123] Furthermore, the local image may be preprocessed, for example, the local image may be scaled to a fixed size and subjected to image standardization (for example, brightness standardization).

[0124] S1022: Input the local image into a first classification network to obtain a first image feature.

[0125] S1023: Input the first limiting matrix into the first limiting network to obtain the first limiting feature.

[0126] The first limiting matrix can be generated and applied based on the principle similar to the generation and application of the second limiting matrix, which is not described in detail here. The first limiting matrix and the second limiting matrix can be the same or different, and the limiting functions used by the two can be the same or different, which is not limited here.

[0127] S1024: Input the first image feature and the first limiting feature into the first output network to obtain an indication confirmation result.

[0128] In one embodiment, a first limiting value is obtained through a first output network based on a first image feature and a first limiting feature; if the first limiting value is determined to be a first target limiting value corresponding to a first limiting matrix, then the confirmation result is determined to indicate that the target area is valid; otherwise, the confirmation result is determined to indicate that the target area is invalid.

[0129] In this way, it is possible to determine whether the target area is valid, that is, whether the user's pointing object (such as a finger) is pointing on the page, based on whether the first limiting value corresponding to the limiting factor is the same as the first target limiting value, and to make a judgment in combination with the limiting function, thereby improving the accuracy of the judgment of the validity of the target area. This is because multiple judgments can improve the accuracy rate. If several limiting factors are used, the confirmation mechanism will be executed several times. The more limiting factors are used, the higher the accuracy of the validity judgment. Different numbers of limiting factors can be used for different scenarios and network sizes. For example, in high recall scenarios, fewer limiting factors are used to balance accuracy and efficiency. In scenarios with high accuracy requirements, more limiting factors are used to ensure high accuracy, thereby avoiding subsequent page detection errors caused by errors in validity judgment, and improving user experience.

[0130] Step 103: If the confirmation result indicates that the target area is valid, a book page detection model is used to perform book page detection according to the image to be detected to obtain a target book page area.

[0131] In one implementation, if it is determined that the target area is valid, the image to be detected is input into a book page detection model for forward reasoning to obtain the target book page area.

[0132] The book page detection model can be constructed by an image segmentation network (u2net small) model. In practical applications, other models can also be used for construction, which is not limited here. Optionally, the target book page area can be a quadrilateral, which can be represented by the coordinates of the four vertices of the quadrilateral, or by other methods, which is not limited here. Furthermore, the image to be processed can also be preprocessed.

[0133] Step 104: cropping the image to be detected according to the target area and the target book page area to obtain a book page area image.

[0134] In one implementation, when executing step 104, the following steps may be used:

[0135] S1041: Obtain the indicated position of the indicated object in the target area output by the object detection model.

[0136] The optional indication position may be a polygonal area. For example, if the indication object is a finger, the indication position is a rectangular area where the fingertip is located.

[0137] S1042: Determine an overlapping area between the indicated position and the target page area.

[0138] S1043: Determine the proportion of the overlapping area in the target area.

[0139] S1044: If the proportion is higher than the set proportion threshold, a page area image including the target page area is cropped from the image to be detected.

[0140] In actual applications, the percentage threshold can be set according to the actual application scenario and is not limited here.

[0141] Furthermore, when there are multiple target page areas, each target page area may be judged separately by setting a percentage threshold, so that only the page area image corresponding to the target page area indicated by the indicating object may be cropped.

[0142] Furthermore, after obtaining the book page region image and the indicated position, the book page content indicated by the user in the book page region image can be identified, and the associated content of the identified content can be obtained, and the associated content can be output.

[0143] For example, the recognized content may be a Chinese word, and the associated content may be the audio of the Chinese word, and the audio may be displayed and played.

[0144] Combine the following Figure 3 The above-mentioned embodiment is described. Figure 3 The following is a detailed implementation flow chart of a book page detection method. The process of the method includes:

[0145] Step 301: Acquire a captured image.

[0146] Step 302: Use a fuzzy detection model to perform fuzzy detection on the captured image.

[0147] Step 303: Determine whether the blur detection result indicates that the image is clear. If so, execute step 304; otherwise, execute step 301.

[0148] Step 304: using the object detection model, perform object detection based on the image to be detected, and obtain a target area indicating where the object is located.

[0149] Step 305: using the indication confirmation model, according to the image to be detected, the target area and the first limiting matrix, to obtain an indication confirmation result of the target area.

[0150] Step 306: Determine whether the indication confirmation result indicates that the target area is valid, if so, execute step 307, otherwise, execute step 301.

[0151] Step 307: cropping the image to be detected according to the target area and the target book page area to obtain a book page area image.

[0152] Among them, when executing steps 301 to 307, please refer to the above steps 101 to 104 for details, which will not be repeated here.

[0153] In the embodiment of the present application, through blur detection, clear images to be detected can be screened out, reducing the detection failure rate caused by image blur, and after detecting the indicated object, it is further confirmed whether the indicated object points to the book page. This is because only when the indicated object points to the book page is subsequent book page detection necessary to display relevant content of the book page to the user. In addition, when there are multiple book pages, the book page area pointed to by the user can be screened and cropped, thereby improving the effectiveness and accuracy of book page detection and improving user experience.

[0154] Based on the same inventive concept, the embodiment of the present disclosure also provides a device for detecting book pages. Since the principle of solving the problem by the above device and equipment is similar to that of a method for detecting book pages, the implementation of the above device can refer to the implementation of the method, and the repeated parts will not be repeated. The device can be applied to electronic devices. The present disclosure does not limit the type of electronic devices. It can be any type of device suitable for implementation, such as terminal devices and servers, etc. The present disclosure will not repeat this.

[0155] See also Figure 4 FIG. 1 is a block diagram of a book page detection device according to an embodiment of the present disclosure. In some embodiments, the book page detection device according to an embodiment of the present disclosure includes:

[0156] The detection unit 401 is used to use an object detection model to perform object detection according to the image to be detected, and obtain a target area where an indicated object is located; the indicated object is used to indicate a book page;

[0157] The confirmation unit 402 is used to obtain an indication confirmation result of the target area according to the image to be detected, the target area and the first limiting matrix by using the indication confirmation model; the indication confirmation result indicates whether the target area is valid; the first limiting matrix is ​​generated according to the limiting factor of the limiting function and is used to assist in judging the indication confirmation result;

[0158] The obtaining unit 403 is used to, if the indication confirmation result indicates that the target area is valid, use the book page detection model to perform book page detection according to the image to be detected to obtain the target book page area;

[0159] The cropping unit 404 is used to crop the image to be detected according to the target area and the target book page area to obtain a book page area image.

[0160] In one embodiment, the indication confirmation model includes a first classification network, a first restriction network, and a first output network;

[0161] The confirmation unit 402 is used for:

[0162] According to the target area, a local image containing the indicated object is cut out from the image to be detected;

[0163] Inputting the local image into a first classification network to obtain a first image feature;

[0164] Inputting the first limiting matrix into the first limiting network to obtain a first limiting feature;

[0165] The first image feature and the first limiting feature are input into the first output network to obtain an indication confirmation result.

[0166] In one implementation, the confirmation unit 402 is used to:

[0167] Obtaining a first limiting value based on the first image feature and the first limiting feature through the first output network;

[0168] If the first limiting value is determined to be the first target limiting value corresponding to the first limiting matrix, then determining the indication confirmation result is that the target area is valid;

[0169] Otherwise, the determination indicates that the result of the determination is that the target area is invalid.

[0170] In one implementation, the detection unit 401 is further configured to:

[0171] The following steps are executed repeatedly until the image to be detected is obtained:

[0172] Acquire the collected image;

[0173] A fuzzy detection model is used to perform fuzziness detection according to the collected image and the second limiting matrix to obtain a fuzzy detection result; the fuzzy detection result indicates whether the collected image is clear; the second limiting matrix is ​​generated according to the limiting factor of the limiting function and is used to assist in judging the fuzzy detection result;

[0174] If the blur detection result shows that the image is clear, the captured image is determined as the image to be detected; otherwise, the captured image is discarded.

[0175] In one implementation, the fuzzy detection model includes a second classification network, a second restriction network, and a second output network;

[0176] The detection unit 401 is also used for:

[0177] Inputting the collected image into a second classification network to obtain a second image feature;

[0178] Inputting the second limiting matrix into the second limiting network to obtain a second limiting feature;

[0179] The second image feature and the second limiting feature are input into the second output network to obtain a blur detection result.

[0180] In one implementation, the detection unit 401 is further configured to:

[0181] Obtaining a second limiting value according to the second image feature and the second limiting feature through a second output network;

[0182] If the second limiting value is determined to be the second target limiting value corresponding to the second limiting matrix, then the blur detection result is determined to be image blur;

[0183] Otherwise, the blur detection result is determined to be that the image is clear.

[0184] In one implementation, the cropping unit 404 is used to:

[0185] Obtaining the indicated position of the indicated object in the target area also output by the object detection model;

[0186] Determine the overlap area between the indicated position and the target page area;

[0187] Determine the proportion of the overlapping area in the target area;

[0188] If the proportion is higher than the set proportion threshold, a page area image containing the target page area is cropped from the image to be detected.

[0189] The method for book page detection in the embodiment of the present disclosure, after detecting the existence of an indicating object in the image to be detected, also confirms the validity of the indicating object (i.e., whether the indicating object points to a book page). Only when the indicating object is valid, will the book page area image be detected and output, thereby outputting the book page area image pointed to by the user, so as to identify the content of the book page pointed to by the user, thereby reducing misjudgment of book page detection.

[0190] In an embodiment of the present disclosure, an electronic device is provided, including:

[0191] Processor; and

[0192] The memory stores computer instructions, where the computer instructions are used to enable the processor to execute the method of any of the above embodiments.

[0193] In an embodiment of the present disclosure, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a computer to execute a method in any of the above-mentioned embodiments.

[0194] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device implements any of the above-mentioned embodiments of the method.

[0195] Figure 5 FIG. 5 shows a schematic diagram of the structure of an electronic device 5000. Figure 5 As shown, the electronic device 5000 includes: a processor 5010 and a memory 5020 , and optionally, may also include a power supply 5030 , a display unit 5040 , and an input unit 5050 .

[0196] The processor 5010 is the control center of the electronic device 5000. It connects various components using various interfaces and lines, and performs various functions of the electronic device 5000 by running or executing software programs and / or data stored in the memory 5020, thereby monitoring the electronic device 5000 as a whole.

[0197] In the embodiment of the present disclosure, the processor 5010 executes the various steps in the above embodiment when calling the computer program stored in the memory 5020.

[0198] Optionally, the processor 5010 may include one or more processing units; preferably, the processor 5010 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and applications, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 5010. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips.

[0199] The memory 5020 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, various applications, etc.; the data storage area may store data created according to the use of the electronic device 5000, etc. In addition, the memory 5020 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices, etc.

[0200] The electronic device 5000 also includes a power source 5030 (such as a battery) for supplying power to various components. The power source can be logically connected to the processor 5010 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0201] The display unit 5040 may be used to display information input by the user or information provided to the user and various menus of the electronic device 5000, and in the embodiment of the present disclosure, is mainly used to display the display interface of each application in the electronic device 5000 and objects such as text and pictures displayed in the display interface. The display unit 5040 may include a display panel 5041. The display panel 5041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc.

[0202] The input unit 5050 may be used to receive information such as numbers or characters input by the user. The input unit 5050 may include a touch panel 5051 and other input devices 5052. The touch panel 5051, also known as a touch screen, may collect touch operations of the user on or near it (such as operations of the user using a finger, a stylus, or any other suitable object or accessory on or near the touch panel 5051).

[0203] Specifically, the touch panel 5051 can detect the user's touch operation, detect the signal brought by the touch operation, convert these signals into touch point coordinates, send them to the processor 5010, and receive and execute the command sent by the processor 5010. In addition, the touch panel 5051 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. Other input devices 5052 can include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, power on / off keys, etc.), a trackball, a mouse, a joystick, etc.

[0204] Of course, the touch panel 5051 can cover the display panel 5041. When the touch panel 5051 detects a touch operation on or near it, it is transmitted to the processor 5010 to determine the type of touch event. Then the processor 5010 provides corresponding visual output on the display panel 5041 according to the type of touch event. Figure 5 In the embodiment, the touch panel 5051 and the display panel 5041 are used as two independent components to realize the input and output functions of the electronic device 5000, but in some embodiments, the touch panel 5051 and the display panel 5041 can be integrated to realize the input and output functions of the electronic device 5000.

[0205] The electronic device 5000 may also include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity light sensor, etc. Of course, according to the needs of specific applications, the electronic device 5000 may also include other components such as a camera. Since these components are not the key components used in the embodiments of the present disclosure, Figure 5 It is not shown and will not be described in detail.

[0206] Those skilled in the art will understand that Figure 5 The electronic device is merely an example and does not limit the electronic device, and may include more or less components than those shown in the figure, or may combine certain components, or may include different components.

[0207] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing the present disclosure, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

Claims

1. A method for detecting book pages, characterized in that: The method comprises: Using an object detection model, performing object detection according to the image to be detected, and obtaining a target area where an indicated object is located; the indicated object is used to indicate a book page; Adopting an indication confirmation model, according to the image to be detected, the target area and a first limiting matrix, obtaining an indication confirmation result of the target area; the indication confirmation result indicates whether the target area is valid; the first limiting matrix is ​​generated according to the limiting factor of the limiting function, and is used to assist in judging the indication confirmation result; If the indication confirmation result indicates that the target area is valid, a book page detection model is used to perform book page detection according to the image to be detected to obtain a target book page area; The image to be detected is cropped according to the target area and the target book page area to obtain a book page area image.

2. The method according to claim 1, characterized in that The indication confirmation model includes a first classification network, a first limiting network and a first output network; The adopting the indication confirmation model to obtain the indication confirmation result of the target area according to the image to be detected, the target area and the first limiting matrix includes: According to the target area, cutting out a partial image containing the indicated object from the image to be detected; Inputting the local image into the first classification network to obtain a first image feature; Inputting the first limiting matrix into the first limiting network to obtain a first limiting feature; The first image feature and the first limiting feature are input into the first output network to obtain the indication confirmation result.

3. The method according to claim 2, characterized in that The step of inputting the first image feature and the first limiting feature into the first output network to obtain the indication confirmation result includes: Obtaining a first limiting value based on the first image feature and the first limiting feature through the first output network; If it is determined that the first limiting value is the first target limiting value corresponding to the first limiting matrix, then determining that the indication confirmation result is that the target area is valid; Otherwise, it is determined that the indication confirmation result is that the target area is invalid.

4. The method according to any one of claims 1 to 3, characterized in that: Before using the object detection model to perform object detection according to the image to be detected and obtaining the target area of ​​the target indicated object, it includes: The following steps are executed repeatedly until the image to be detected is obtained: Acquire the collected image; A blur detection model is used to perform blur detection according to the collected image and the second limiting matrix to obtain a blur detection result; the blur detection result indicates whether the collected image is clear; the second limiting matrix is ​​generated according to the limiting factor of the limiting function and is used to assist in judging the blur detection result; If the blur detection result is that the image is clear, the acquired image is determined as the image to be detected; otherwise, the acquired image is discarded.

5. The method according to claim 4, characterized in that The fuzzy detection model includes a second classification network, a second limiting network and a second output network; The adopting of the fuzzy detection model to perform fuzziness detection according to the collected image and the second limiting matrix to obtain a fuzzy detection result includes: Inputting the collected image into the second classification network to obtain a second image feature; Inputting the second limiting matrix into the second limiting network to obtain a second limiting feature; The second image feature and the second limiting feature are input into the second output network to obtain the blur detection result.

6. The method according to claim 5, characterized in that The step of inputting the second image feature and the second limiting feature into the second output network to obtain the blur detection result includes: Obtaining a second limiting value according to the second image feature and the second limiting feature through the second output network; If it is determined that the second limiting value is the second target limiting value corresponding to the second limiting matrix, then the blur detection result is determined to be image blur; Otherwise, the blur detection result is determined to be that the image is clear.

7. The method according to any one of claims 1 to 3, characterized in that: The step of cropping the image to be detected according to the target area and the target book page area to obtain a book page area image includes: Obtaining the indicated position of the indicated object in the target area which is also output by the object detection model; Determine an overlapping area between the indicated position and the target page area; Determine the proportion of the overlapping area in the target area; If the proportion is higher than a set proportion threshold, a page area image including the target page area is cropped from the image to be detected.

8. A book page detection device, characterized in that: The device comprises: A detection unit, used to use an object detection model to perform object detection according to the image to be detected, and obtain a target area where an indicated object is located; the indicated object is used to indicate a book page; a confirmation unit, configured to obtain an indication confirmation result of the target area according to the image to be detected, the target area and a first limiting matrix by adopting an indication confirmation model; the indication confirmation result indicates whether the target area is valid; the first limiting matrix is ​​generated according to a limiting factor of a limiting function and is used to assist in determining the indication confirmation result; an obtaining unit, configured to, if the indication confirmation result indicates that the target area is valid, use a book page detection model to perform book page detection according to the image to be detected to obtain a target book page area; The cropping unit is used to crop the image to be detected according to the target area and the target book page area to obtain a book page area image.

9. An electronic device, characterized in that: include: processor; as well as A memory storing computer instructions, wherein the computer instructions are used to enable the processor to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Computer instructions are stored, and the computer instructions are used to make a computer execute the method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The invention comprises a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code is executed in a processor of an electronic device, the processor in the electronic device performs the method according to any one of claims 1 to 7.

Images