Image recognition and response processing method and device, equipment and storage medium

Abstract

According to the image recognition and response processing method and device, the equipment and the storage medium provided by the invention, the pre-trained detection network and recognition network are combined, and for the to-be-recognized page containing the illustration, the illustration is used as a recognition clue; and through a dual classification mechanism, i.e., the preliminary classification of the detection network and the fine classification of the recognition network are combined, misjudgment is effectively filtered, input requirements of various document formats, font changes and the like can be met, a relatively high recognition rate can be provided for the condition that the page to be recognized has fuzzy and incomplete characters, and the recognition efficiency is improved. A more reliable and efficient page number identification means is provided, and the accuracy and robustness of page number identification are improved.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to an image recognition and response processing method, device, equipment and storage medium. Background Art

[0002] In the paper-screen interaction scenario, traditional methods usually rely on OCR (Optical Character Recognition) technology and image search engines to meet the needs of electronic devices to accurately identify the page numbers of paper document pages. OCR technology scans and parses the text content in the paper document page, so that the electronic device can match the text content with the text of each page of the pre-stored paper document to achieve retrieval and page number recognition; for pages without text, the image search engine compares the page without text with multiple page images of paper documents stored in the database, and infers the page number based on the similarity of the image content.

[0003] However, in conditions of low light or impaired image quality, OCR technology often has difficulty accurately recognizing text, resulting in deviations in page number recognition results. Similarly, image search engines are also prone to mismatching page numbers when processing low-quality images or images with extremely high similarity, affecting the user's browsing experience. Summary of the invention

[0004] In view of this, in order to solve the above technical problems, the present application provides an image recognition and response processing method, device, equipment and storage medium.

[0005] Specifically, the present application is implemented through the following technical solutions:

[0006] According to a first aspect of an embodiment of the present application, an image recognition method is provided, and the method may include:

[0007] Acquire a page image of a paper document as a page to be identified;

[0008] Using a pre-trained detection network to detect whether the page to be identified includes a picture, and if the page to be identified includes a picture, determining a first classification result of the picture;

[0009] Extracting the illustration from the page to be identified, and inputting the illustration into a pre-trained recognition network to obtain a second classification result of the illustration; wherein both the detection network and the recognition network include a plurality of classification categories, and each classification category corresponds to all the illustrations of a single page in the paper document;

[0010] When the second classification result matches the first classification result, a corresponding page number of the to-be-identified page in the paper document is determined.

[0011] Optionally, the target network includes the detection network or the recognition network; each convolution layer of the target network includes at least one of the upper and lower convolutions and the left and right convolutions in the custom convolution operation; the upper and lower convolutions include horizontally dividing the first convolution kernel into two sub-convolution kernels of equal size and then calculating the difference between the upper and lower convolution results; the left and right convolutions include vertically dividing the second convolution kernel into two sub-convolution kernels of equal size and then calculating the difference between the left and right convolution results.

[0012] Optionally, when the page to be identified includes a picture, determining a first classification result of the picture includes:

[0013] Determine a bounding box of the image using the detection network, and determine a region of interest in the page to be identified based on the bounding box;

[0014] Through the custom convolution operation in the detection network, a first image feature of the region of interest is extracted, and a first classification result of the image is predicted based on the first image feature.

[0015] Optionally, the step of cutting the image from the page to be identified and inputting the image into a pre-trained recognition network to obtain a second classification result of the image includes:

[0016] Cropping the page to be recognized according to the bounding box of the image determined by the detection network to obtain the image and input it into the recognition network;

[0017] Through the custom convolution operation in the recognition network, a second picture feature of the picture is extracted, and a second classification result of the picture is predicted based on the second picture feature.

[0018] Optionally, the multiple convolution kernels used by each convolution layer of the target network include a first convolution kernel for implementing the up-down convolution in a first proportion, a second convolution kernel for implementing the left-right convolution in a second proportion, and a third convolution kernel for implementing the standard convolution in a third proportion;

[0019] The upper and lower convolution or the left and right convolution includes: dividing the corresponding convolution kernel into two equal-sized sub-convolution kernels; calculating the convolution result obtained by weighted summation of each sub-convolution kernel, and obtaining the absolute value of the difference between the two convolution results; and obtaining the corresponding final convolution result according to the absolute value.

[0020] Optionally, when the second classification result matches the first classification result, determining a corresponding page number of the to-be-identified page in the paper document includes:

[0021] Obtaining a first prediction category with the highest confidence in the first classification result, and obtaining a second prediction category with the highest confidence in the second classification result;

[0022] When the second prediction category meets the set conditions, it is determined that the second classification result matches the first classification result, and the page number of the page in the paper document corresponding to the second prediction category is obtained as the corresponding page number; the set conditions include: the confidence of the second prediction category is greater than or equal to a preset threshold, and the second prediction category is the same as the first prediction category.

[0023] Optionally, the method further comprises:

[0024] When the second classification result does not match the first classification result, extracting an image feature vector of the page to be identified to obtain a first feature vector of a fixed dimension;

[0025] The first feature vector is matched with a pre-stored feature vector of a first page of the paper document containing a picture, and a page number corresponding to the matched feature vector is determined as the corresponding page number.

[0026] Optionally, the target network further includes a pre-trained fuzzy judgment network; and the method further includes:

[0027] By means of the custom convolution operation in the fuzzy judgment network, edge features of the page to be identified are extracted and transmitted to subsequent layers of the fuzzy judgment network to obtain a fuzzy classification result of the page to be identified;

[0028] When the fuzzy classification result indicates that the page to be identified is not fuzzy, it is determined that the clarity of the page to be identified meets the set requirement; otherwise, it is determined that the clarity does not meet the set requirement.

[0029] When the definition meets the set requirement, it is detected whether the page to be identified includes a picture.

[0030] Optionally, the method further comprises:

[0031] In the case that the page to be identified does not include a picture, extracting the text content of the page to be identified, and matching it with the page text of each second page that does not include a picture in the pre-stored paper document, and obtaining the first page number corresponding to the matched page text;

[0032] Extracting an image feature vector of the page to be identified to obtain a second feature vector of a fixed dimension, matching the second feature vector with a pre-stored feature vector of the second page, and obtaining a second page number of the second page corresponding to the matched feature vector;

[0033] In the case that the first page number is the same as the second page number, the first page number is determined as the corresponding page number of the to-be-identified page in the paper document.

[0034] Optionally, the method further comprises:

[0035] When the first page number is different from the second page number, controlling the display of a second page corresponding to the first page number and the second page number on the interactive interface;

[0036] A page number selected by a user from a displayed second page through an interactive interface is received, and the selected page number is used as the corresponding page number.

[0037] According to a second aspect of an embodiment of the present application, there is provided an image recognition device, the device comprising:

[0038] A page acquisition module to be identified is used to acquire a page image of a paper document as a page to be identified;

[0039] The picture detection and recognition module is used to detect whether the page to be recognized includes a picture using a pre-trained detection network, and if the page to be recognized includes a picture, determine a first classification result of the picture; cut the picture from the page to be recognized, and input the picture into a pre-trained recognition network to obtain a second classification result of the picture; wherein the detection network and the recognition network both include multiple classification categories, each of which corresponds to all the pictures of a single page in the paper document;

[0040] The target page determination module is used to determine the corresponding page number of the page to be identified in the paper document when the second classification result matches the first classification result.

[0041] Optionally, the target network includes the detection network or the recognition network; each convolution layer of the target network includes at least one of the upper and lower convolutions and the left and right convolutions in the custom convolution operation; the upper and lower convolutions include horizontally dividing the first convolution kernel into two sub-convolution kernels of equal size and then calculating the difference between the upper and lower convolution results; the left and right convolutions include vertically dividing the second convolution kernel into two sub-convolution kernels of equal size and then calculating the difference between the left and right convolution results.

[0042] Optionally, based on the custom convolution operation, the image detection and recognition module is specifically used to:

[0043] Determine a bounding box of the image using the detection network, and determine a region of interest in the page to be identified based on the bounding box;

[0044] Through the custom convolution operation in the detection network, a first image feature of the region of interest is extracted, and a first classification result of the image is predicted based on the first image feature.

[0045] Optionally, based on the custom convolution operation, the image detection and recognition module is specifically used to:

[0046] Cropping the page to be recognized according to the bounding box of the image determined by the detection network to obtain the image and input it into the recognition network;

[0047] Through the custom convolution operation in the recognition network, a second picture feature of the picture is extracted, and a second classification result of the picture is predicted based on the second picture feature.

[0048] Optionally, the multiple convolution kernels used by each convolution layer of the target network include a first convolution kernel for implementing the up-down convolution in a first proportion, a second convolution kernel for implementing the left-right convolution in a second proportion, and a third convolution kernel for implementing the standard convolution in a third proportion;

[0049] The upper and lower convolution or the left and right convolution includes: dividing the corresponding convolution kernel into two equal-sized sub-convolution kernels; calculating the convolution result obtained by weighted summation of each sub-convolution kernel, and obtaining the absolute value of the difference between the two convolution results; and obtaining the corresponding final convolution result according to the absolute value.

[0050] Optionally, the target page determination module is specifically used to:

[0051] Obtaining a first prediction category with the highest confidence in the first classification result, and obtaining a second prediction category with the highest confidence in the second classification result;

[0052] When the second prediction category meets the set conditions, it is determined that the second classification result matches the first classification result, and the page number of the page in the paper document corresponding to the second prediction category is obtained as the corresponding page number; the set conditions include: the confidence of the second prediction category is greater than or equal to a preset threshold, and the second prediction category is the same as the first prediction category.

[0053] Optionally, the device further comprises:

[0054] When the second classification result does not match the first classification result, extracting an image feature vector of the page to be identified to obtain a first feature vector of a fixed dimension;

[0055] The first feature vector is matched with a pre-stored feature vector of a first page of the paper document containing a picture, and a page number corresponding to the matched feature vector is determined as the corresponding page number.

[0056] Optionally, the target network further includes a pre-trained fuzzy judgment network; and the device further includes:

[0057] By means of the custom convolution operation in the fuzzy judgment network, edge features of the page to be identified are extracted and transmitted to subsequent layers of the fuzzy judgment network to obtain a fuzzy classification result of the page to be identified;

[0058] When the fuzzy classification result indicates that the page to be identified is not fuzzy, it is determined that the clarity of the page to be identified meets the set requirement; otherwise, it is determined that the clarity does not meet the set requirement.

[0059] When the definition meets the set requirement, it is detected whether the page to be identified includes a picture.

[0060] Optionally, the device further comprises:

[0061] In the case that the page to be identified does not include a picture, extracting the text content of the page to be identified, and matching it with the page text of each second page that does not include a picture in the pre-stored paper document, and obtaining the first page number corresponding to the matched page text;

[0062] Extracting an image feature vector of the page to be identified to obtain a second feature vector of a fixed dimension, matching the second feature vector with a pre-stored feature vector of the second page, and obtaining a second page number of the second page corresponding to the matched feature vector;

[0063] In the case that the first page number is the same as the second page number, the first page number is determined as the corresponding page number of the to-be-identified page in the paper document.

[0064] Optionally, the device further comprises:

[0065] When the first page number is different from the second page number, controlling the display of a second page corresponding to the first page number and the second page number on the interactive interface;

[0066] A page number selected by a user from a displayed second page through an interactive interface is received, and the selected page number is used as the corresponding page number.

[0067] According to a third aspect of an embodiment of the present application, a response processing method is provided, which may include:

[0068] When a page image of a target paper document is acquired, using the image recognition method described in any of the above embodiments, determining that the page image corresponds to a target page number in the target paper document;

[0069] According to the preset mapping relationship between the page of the target paper document and the device operation, the device is triggered to execute the device operation corresponding to the target page number.

[0070] According to a fourth aspect of an embodiment of the present application, a response processing device is provided, the device comprising:

[0071] A page number determination module is used to determine, when a page image of a target paper document is acquired, a target page number in the target paper document corresponding to the page image by using an image recognition method;

[0072] The trigger execution module is used to trigger the device to execute the device operation corresponding to the target page number according to the preset mapping relationship between the page of the target paper document and the device operation.

[0073] According to the fifth aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory and a processor; the memory is used to store a computer program; the processor is used to execute the above-mentioned image recognition method or response processing method by calling the computer program.

[0074] According to a sixth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned image recognition method or response processing method is implemented.

[0075] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0076] In the technical solution provided by the present application, by combining the pre-trained detection network and recognition network, for the page to be identified that contains the illustrations, not only the illustrations are used as identification clues, but also the double classification mechanism, i.e., the preliminary classification of the detection network and the fine classification of the recognition network, is combined to effectively filter out misjudgments, provide a more reliable and efficient means of page number recognition, and improve the accuracy and robustness of page number recognition. Compared with pure text recognition or image search engines in related technologies, this method can cope with input requirements such as diverse document formats and font changes, and can also provide a higher recognition rate for situations where there are blurred or incomplete text on the page to be identified, thereby improving the automation level of document processing and user experience.

[0077] It should be understood that the above general description and the following detailed description are only exemplary and explanatory and cannot limit the present application. In addition, any embodiment in the present application does not need to achieve all the above effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0079] Figure 1 It is a schematic diagram of an image recognition method flow chart shown in an exemplary embodiment of the present application;

[0080] Figure 2 It is a schematic diagram of a processing of upper and lower convolutions shown in an exemplary embodiment of the present application;

[0081] Figure 3 It is a schematic diagram of a process of dividing a convolution kernel of an upper and lower convolution and a left and right convolution into two sub-convolution kernels of equal size, shown in an exemplary embodiment of the present application;

[0082] Figure 4 It is a flowchart of a step of matching a first classification result with a second classification result and determining a corresponding page number of a page to be identified, shown in an exemplary embodiment of the present application;

[0083] Figure 5 This is a flowchart of a step of using a fuzzy judgment network to detect whether a page to be identified is fuzzy, shown in an exemplary embodiment of the present application;

[0084] Figure 6 It is a flowchart of a page number recognition step combining text search and feature vector search for double verification when a page to be recognized does not include an illustration, as shown in an exemplary embodiment of the present application;

[0085] Fig. 7A is a flowchart of an image recognition method shown in an exemplary embodiment of the present application;

[0086] Figure 7B It is an example of a 4*4 convolution kernel and convolution image data covered by the convolution kernel shown in an exemplary embodiment of the present application;

[0087] Figure 7C It is a schematic diagram of segmentation when a 4*4 convolution kernel and convolution image data covered by the convolution kernel are convolved up and down, as shown in an exemplary embodiment of the present application;

[0088] Fig.7D It is a schematic diagram of the segmentation situation when a 4*4 convolution kernel and the convolution image data covered by the convolution kernel are subjected to left and right convolutions as shown in an exemplary embodiment of the present application;

[0089] Figure 8 is a structural schematic diagram of an image recognition device shown in an exemplary embodiment of the present application;

[0090] Fig. 9is a structural schematic diagram of a response processing device shown in an exemplary embodiment of the present application;

[0091] Fig.10 It is a hardware schematic diagram of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0092] In the paper-screen interaction scenario, the traditional method mainly relies on OCR technology and image search engines to meet the needs of electronic devices to accurately identify the page numbers of paper documents. OCR technology scans and parses the text information in the document page with high precision, and attempts to match the extracted text content with the text of each page of the document pre-stored in the database, so as to achieve rapid retrieval and recognition of page numbers. For pages that do not contain obvious text or have very little text, this method will no longer be applicable.

[0093] At this time, image search engines become another option. They compare the current page image without text with the paper document page image stored in the database, and infer the page number based on the similarity of the image content. This can make up for the shortcomings of OCR in pages with missing text to a certain extent. However, in the case of poor lighting conditions, damaged image quality, or a large number of pages with similar layouts in the document, image search engines are prone to page number mismatches, which in turn affects the user's browsing efficiency and experience.

[0094] Specifically, insufficient light can significantly reduce image clarity and increase the difficulty of OCR recognition; impaired image quality may include noise, blur, distortion and other factors, further affecting the accuracy of OCR and image search engines. In addition, when there are a large number of pages with similar layouts but different contents in a document, image search engines may find it difficult to accurately distinguish them, causing confusion in page number recognition.

[0095] The above problems are common in various application scenarios, such as page number recognition of boot manuals for learning machines in the education field, or page number recognition of paper documents for other consumer electronic products. A more accurate and robust image page number recognition method is needed to improve the user experience. Therefore, in order to improve the accuracy of page number recognition in paper-screen interactive scenarios, the present application provides an image recognition method for page number recognition of page images of paper documents in paper-screen interactive scenarios, which can be executed by any electronic device that needs to recognize the page number of an input image, depending on the actual application scenario. This method is not only suitable for the recognition of boot manuals for learning machines, but can also be widely used in other scenarios that require high-precision page number recognition, such as library book management, electronic archive organization, etc., providing a new solution for improving work efficiency and user experience.

[0096] See also Figure 1 As shown, the present application provides an image recognition method which may include the following steps:

[0097] S101, obtaining a page image of a paper document as a page to be identified;

[0098] The page image refers to a digital representation of one or more document pages included in the paper document, which can be stored in a picture format such as JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics) or TIFF (Tagged Image File Format), retaining the visual content of the original document page. In this embodiment, the page image is used as the page to be identified, which means that the user expects the electronic device to identify the corresponding page number of the page image in the paper document.

[0099] The page image can be obtained by scanning, taking photos, etc., wherein the scanning method can be implemented by a scanner, a scanning pen or a device with a scanning function, and the taking photos can be implemented by a camera, a device with a photo taking function, etc. For example, some intelligent learning devices are provided with one or more grooves, platforms or adjustable brackets for stably placing paper documents, and are equipped with at least one high-definition camera for accurately focusing and capturing the image of the document page of the placed paper document.

[0100] S102, using a pre-trained detection network to detect whether the page to be identified includes an illustration, and if the page to be identified includes an illustration, determining a first classification result of the illustration; the detection network includes a plurality of classification categories, each classification category corresponding to all illustrations of a single page in the paper document;

[0101] Illustrations refer to images or graphic elements on the page to be identified, which may include but are not limited to photos, illustrations, charts, schematic diagrams, cartoons or hand-drawn sketches. When the page to be identified is obtained, it is first detected whether the page to be identified includes illustrations. This process can be achieved through image processing technology or machine learning algorithms. For example, different areas of the page to be identified can be distinguished by an edge detection algorithm, and key features such as color and texture can be combined to analyze and determine whether it is an illustration area; a machine learning model trained with a large amount of annotated image data, such as the above-mentioned detection network, can also be used to learn the difference between illustrations and text, and can accurately detect whether an illustration is included on a new page to be identified. In this embodiment, the situation where the page to be identified includes illustrations may include: the page to be identified only includes illustrations, or the page to be identified contains both text and illustrations.

[0102] In this embodiment, a pre-trained detection network is used to implement picture detection and preliminary classification functions. The detection network is pre-trained to detect whether the page to be identified includes pictures, and to perform preliminary classification operations on the detected pictures. The first classification result can be obtained by extracting local area features of the pictures on the page to be identified, including but not limited to visual features such as edges, textures, color distribution, and basic attribute features such as shape, size, and proportion, and preliminarily classifying the pictures based on the local area features. The first classification result can at least include the probability distribution of the pictures belonging to each set classification category.

[0103] Based on the classification result of the detection network in this embodiment, the corresponding page number of the page to be identified in the paper document is determined. Therefore, each classification category of the detection network is preset to correspond to all the illustrations of a single page in the paper document. That is, for each document page with illustrations in the paper document, all the illustrations included in the document page correspond to a classification category of the detection network, so as to use the detection network to identify and distinguish the illustrations from different document pages of the paper document.

[0104] The detection network can be designed and implemented based on a convolutional neural network or a deep learning architecture, such as a Yolo V5 network, or an R-CNN (Regions with CNN features, regional convolutional neural network) series network, or a Transformer-based network for design and implementation, which can detect or extract the features of the illustrations in the page to be identified through a series of convolutional layers, pooling layers, activation functions and other structures. The output layer of the detection network is designed to have multiple neurons, each neuron corresponds to a specific classification category, and each classification category is directly mapped to all the illustrations included in a page with illustrations in the paper document, so that the classification result of the network can locate the illustrations contained in the page to be identified in the corresponding source page of the paper document. For example, the page numbers of each document page with illustrations in the paper document can be directly used as the classification category. When the classification category output by the network is 11, it means that the illustration of the page to be identified belongs to the eleventh document page of the paper document.

[0105] In order to realize the image detection and preliminary classification functions of the above detection network, a large amount of labeled data containing the location and category labels of the image can be used to train the network in the pre-training stage of the detection network, and the bounding box detection and initial classification errors can be minimized by optimizing the internal parameters of the network. The loss function of the detection network can include two parts: the image detection and positioning error and the image classification error. The importance of the two errors can be balanced by adjusting the weight of the loss function. Based on the structural design of the output layer of the above detection network, the pre-training of the detection network can be achieved in the following ways: (1) Dataset preparation: For each page of the paper document with an illustration, a large number of page image samples are collected, which may include multiple samples of the same document page under different shooting conditions, different clarity levels, etc., and each page image sample is annotated with the illustration position and classification category, and the classification category refers to the classification category set in the detection network; (2) Image preprocessing: Perform necessary preprocessing operations on the images, such as scaling, cropping, normalization, etc., to ensure that they have consistent size and format. Image enhancement (such as rotation, flipping, adding noise, etc.) can also be performed to increase the diversity of the dataset and the generalization ability of the model; (3) Model training: Use the preprocessed dataset to train the detection network. During the training process, the output of the network is calculated through forward propagation, and the network parameters are adjusted using the backpropagation algorithm to minimize the loss function.

[0106] After the above detection network training is completed, the detection network can be deployed in actual application scenarios, so that the network can be used to process the page image of the received paper document to automatically identify and classify which page of the paper document the illustration contained in the page image belongs to.

[0107] S103, extracting the illustration from the page to be identified, and inputting the illustration into a pre-trained recognition network to obtain a second classification result of the illustration; wherein the recognition network includes a plurality of classification categories, each classification category corresponding to all illustrations of a single page in the paper document;

[0108] The recognition network is used to accurately classify the illustrations in the page to be recognized. The network uses the illustrations themselves as network input, and can automatically extract more complex and detailed illustration feature representations from the illustrations through structures such as multi-layer convolutional neural networks, which may include but are not limited to local details of the illustrations such as texture patterns, color gradients, shape features such as contours, symmetry, and higher-level semantic features such as object parts, attributes, contextual relationships, etc., so as to accurately classify the illustrations according to the illustration feature representations to obtain a second classification result, which may at least include the probability distribution of the illustrations belonging to each classification category in the recognition network. Among them, the above-mentioned detection network can also accurately locate the bounding box of the illustration in the page to be recognized through features such as edge strength and direction changes, and the input of the recognition network can be obtained by extracting the illustration from the page to be recognized using the bounding box.

[0109] Based on the recognition network in this embodiment, each classification category corresponds to all the illustrations of a single page in the paper document, that is, all the illustrations included in each page with illustrations in the paper document correspond to a classification category of the detection network, so as to use the recognition network to accurately identify the illustrations from different pages of the paper document. The detection network of the detection network can be designed and implemented based on a convolutional neural network or a deep learning architecture, such as a shufflenetv2 classification network, a MobileNet series network, etc. In the pre-training stage, the recognition network can be trained by a large amount of labeled data containing only the illustrations themselves and category labels, and the classification errors can be minimized by optimizing the internal parameters.

[0110] The recognition network can detect or extract the feature representation of the illustrations through a series of convolutional layers, pooling layers, activation functions and other structures. Its output layer can be designed to have multiple neurons, each neuron corresponds to a specific classification category, and each classification category is directly mapped to all the illustrations included in a page with illustrations in the paper document, so that the classification result of the network can locate the illustrations included in the page to be identified in the corresponding source page of the paper document. For example, the page numbers of each document page with illustrations in the paper document can be directly used as the classification category. When the classification category output by the network is 11, it means that the input illustration belongs to the eleventh document page of the paper document.

[0111] The above-mentioned detection network and recognition network can also adopt the same basic network structure, and adaptively modify the network layer of the basic network structure according to their respective task requirements. For example, if the basic network structure is a classification network, then for the detection network, a network structure for realizing the image detection function should be added on the basis of the structure of the classification network.

[0112] S104: When the second classification result matches the first classification result, determine a corresponding page number of the to-be-identified page in the paper document.

[0113] The second classification result matches the first classification result when: there are target classification categories in the two classification results that meet their respective preset confidence requirements, and the target classification category in the second classification result is the same as the page in the paper document corresponding to at least one target classification category in the first classification result.

[0114] If the second classification result matches the first classification result, it can be determined that the second classification result for the picture on the page to be identified is accurate or acceptable. Based on the one-to-one correspondence between the classification categories of the detection network and the recognition network and the pages with pictures in the paper document, the corresponding page in the paper document can be determined based on the target classification category in the second classification result that meets the preset reliability requirements, and then the corresponding page number of the page to be identified in the paper document can be determined.

[0115] In the disclosed embodiment, by combining the pre-trained detection network and recognition network, the illustrations in the paper documents interacting with the electronic device are double-classified and verified, thereby improving the accuracy and robustness of page number recognition. For the pages to be recognized that contain illustrations, not only are the illustrations used as recognition clues, but also the dual classification mechanism, i.e., the preliminary classification of the detection network and the fine classification of the recognition network, is combined to effectively filter out misjudgments and ensure the accuracy of the recognition results. In addition, due to the uniqueness of the illustrations and their relatively fixed position in the document, compared with the pure text recognition of the related technology, it can better cope with challenges such as diverse document formats and font changes, especially when there are blurred or incomplete text in the document, the illustration recognition can maintain a high recognition rate, providing a more reliable and efficient means of page number recognition, greatly improving the automation level of document processing and user experience.

[0116] In some embodiments, based on the limitations of standard convolution operations, in order to more comprehensively capture the feature information in the image and enhance the effectiveness and richness of the extracted features to better adapt to the complex and changeable pages to be identified, this embodiment proposes an innovative convolution processing method. For the detection network and the recognition network (collectively referred to as the target network) described in the aforementioned embodiments, each convolution layer of the target network introduces at least one of the upper and lower convolutions or left and right convolutions in the custom convolution operation, thereby broadening the field of view of the convolution kernel and enabling the network to capture more multidimensional and detailed feature information.

[0117] Among them, the upper and lower convolutions include horizontally dividing the first convolution kernel into two sub-convolution kernels of equal size and then calculating the difference between the upper and lower convolution results; the left and right convolutions include vertically dividing the second convolution kernel into two sub-convolution kernels of equal size and then calculating the difference between the left and right convolution results.

[0118] Each convolution layer of the target network may include multiple convolution kernels, and the element weights and bias items of each convolution kernel are learned in the pre-training process so that the network can automatically learn feature representation from the network input data. Based on the predefined upper and lower convolution and left and right convolution operations introduced in this embodiment, the multiple convolution kernels may include at least a first convolution kernel for implementing upper and lower convolutions, and at least one of the second convolution kernels for implementing left and right convolutions, and may also include a standard convolution kernel, which realizes feature extraction by multiplying each element in the convolution kernel with the element at the corresponding position of the image to be convolved, and adding all the products. That is, each convolution layer of the target network can be set to include only upper and lower convolutions or left and right convolutions in the convolution layer, or it can also introduce at least one of upper and lower convolutions or left and right convolutions in the custom convolution operation on the basis of retaining the standard convolution operation.

[0119] When the first convolution kernel is used to extract features, the processing method for implementing upper and lower convolutions includes: dividing the first convolution kernel into two sub-convolution kernels with smaller dimensions in the horizontal direction, corresponding to the upper and lower parts of the horizontal dividing line respectively; in the coverage area of ​​the first convolution kernel for the image to be convolved, for each sub-convolution kernel, multiplying each element in the sub-convolution kernel with the element at the corresponding position of the image in the coverage area, and adding all the products to obtain the convolution result of the sub-convolution kernel; obtaining the difference between the convolution results of the upper and lower sub-convolution kernels, and further combining the bias term to obtain the final upper and lower convolution results. The upper and lower convolution method can extract the difference features in the vertical direction of the two image blocks corresponding to the coverage area of ​​the two sub-convolution kernels, thereby increasing the diversity and information content of the features and enhancing the network's ability to understand the image content.

[0120] For example, Figure 2 The weights and bias items of the 4*3 first convolution kernel shown are learned in the pre-training process, and cover the 4*3 pixel area X on the current image to be convolved. When performing upper and lower convolution operations, the first convolution kernel can be divided into a 2*3 upper convolution kernel A1 and a lower convolution kernel A2 along the horizontal dividing line L1 shown in the figure. The upper convolution kernel A1 corresponds to the pixel area X1, and the lower convolution kernel A2 corresponds to the pixel area X2. When calculating the convolution results of the sub-convolution kernels, the weights of A1 are multiplied by the pixels at the corresponding positions on X1 and the products are added to obtain the convolution result M1 of A1, and the weights of A2 are multiplied by the pixels at the corresponding positions of X2 and the products are added to obtain the convolution result M2 of A2, and then the difference between the convolution results M1 and M2 is calculated, and the bias item of the first convolution kernel can be further combined to obtain the final convolution result M of the first convolution kernel performing upper and lower convolution on the pixel area X.

[0121] Similarly, when the second convolution kernel is used to extract features, the processing method for implementing left and right convolution includes: dividing the second convolution kernel into two smaller sub-convolution kernels in the vertical direction, corresponding to the left and right parts of the vertical dividing line respectively, and the convolution result acquisition principle of each sub-convolution kernel is the same as that of the above-mentioned first convolution kernel. The difference between the convolution results of the left and right sub-convolution kernels can be obtained, and the final left and right convolution results can be further combined with the bias term. The calculation method of the sub-convolution kernel in the left and right convolution is the same as the calculation method of the sub-convolution kernel in the above-mentioned upper and lower convolutions, and this application will not repeat it here.

[0122] After introducing a custom convolution operation for the convolution layer of the target network, in the process of training the target network using the preprocessed data set, the output of the network is calculated through forward propagation, and the parameters of each layer of the network are adjusted using a back propagation algorithm, including adjusting the weights and bias terms of multiple convolution kernels of each convolution layer to minimize the loss function and learn the weight parameters on each convolution kernel for implementing the custom convolution operation.

[0123] In the embodiments of the present disclosure, local area difference features of two corresponding image blocks can be extracted based on the upper and lower / left and right convolutions. By introducing customized upper and lower convolutions and left and right convolutions into each convolution layer of the detection network or the recognition network, the perceptual field of view of the convolution kernel is broadened, and the network's ability to capture complex and subtle features in the image is enhanced, so that more effective features can be extracted from images with poor lighting conditions or poor image quality, which promotes the model's fusion processing of global and local information of the image, thereby improving the accuracy and robustness of the network in detection and recognition tasks.

[0124] In some embodiments, based on the custom convolution operation introduced into the detection network in the above embodiment, the determination of the first classification result of the image in the above step S102 when the page to be identified includes an image can be implemented as follows:

[0125] The detection network is used to determine the bounding box of the image, and the region of interest in the page to be identified is determined based on the bounding box; the first image feature of the region of interest is extracted through the custom convolution operation in the detection network, and the first classification result of the image is predicted based on the first image feature.

[0126] The bounding box of the image is used to describe the position and range of the image on the page to be identified. The positions of the upper left corner and the lower right corner of the bounding box can be determined by four parameters: the minimum X coordinate, the minimum Y coordinate, the maximum X coordinate, and the maximum Y coordinate. When the detection network determines that the page to be identified includes an image using the image detection capability learned in the pre-training stage, the bounding box of the image on the page to be identified can be determined at the same time.

[0127] The region of interest in the page to be identified refers to the local area range clearly defined by the bounding box of the picture determined by the detection network, and can directly correspond to the local area framed by the bounding box in the page to be identified. In order to flexibly adapt to the processing requirements of different scenarios, the region of interest can also be set as a local area focused on the periphery of the picture, which is expanded around the bounding box according to the preset size boundaries with the bounding box as the center.

[0128] The first picture feature is a feature representation of the region of interest obtained through the convolution layer of the detection network, which is used to describe the content, shape, texture and other attributes of the picture, and serves as the data basis for the preliminary classification of the detection network. After detecting the picture on the page to be identified, the detection network can use the convolution kernel corresponding to the upper and lower convolutions or the left and right convolutions set in the convolution layer to extract the feature representation of the region of interest as the first picture feature; or, based on the need for the detection network to use the convolution layer to extract the features of the page to be identified for picture detection, after detecting the picture, the features of the picture can be determined from the extracted features of the page to be identified based on the position of the picture boundary box, and the features of the picture can be used as the first picture feature.

[0129] After obtaining the first image feature, the detection network can predict the probability distribution of the classification category to which the image belongs through the learned classification parameters, thereby generating the first classification result.

[0130] In the disclosed embodiment, the detection network is used to accurately locate the image bounding box, thereby accurately delineating the area of ​​interest in the page to be identified, thereby achieving precise focusing on the target content, and utilizing the custom convolution operation in the detection network to efficiently capture rich and effective first image features from the area of ​​interest, thereby achieving prediction of image classification based on the first image features, thereby improving recognition accuracy and efficiency.

[0131] In some embodiments, based on the custom convolution operation introduced into the recognition network in the above embodiments, the above step S103 of cutting out the image from the page to be recognized and inputting the image into the pre-trained recognition network to obtain the second classification result of the image can be implemented as follows:

[0132] The page to be identified is cropped according to the bounding box of the illustration determined by the detection network to obtain the illustration and input it into the recognition network; through the custom convolution operation in the recognition network, a second illustration feature of the illustration is extracted, and a second classification result of the illustration is predicted based on the second illustration feature.

[0133] After obtaining the bounding box of the image on the page to be identified determined by the detection network, the image area of ​​the image can be extracted from the page to be identified using the position information of the bounding box, and the cropped image is used as the input of the recognition network. In order to ensure the quality of the image, the cropping operation can remove irrelevant background information as much as possible while ensuring the integrity of the image content.

[0134] When the recognition network receives the picture, it can use the convolution kernel corresponding to the up-down convolution or left-right convolution and / or standard convolution operation set in the convolution layer to extract the feature representation of the input picture as the second picture feature. Based on the extracted second picture feature, the classifier in the recognition network can be used to accurately classify and predict the picture. The classifier will output the second classification result of the picture according to different combinations of features and weight distribution, indicating the probability distribution of the picture coming from each page with pictures in the paper document.

[0135] In the disclosed embodiment, redundant background information is removed by accurately cropping the illustrations in the page to be identified, the cropped illustrations are directly input into the recognition network, and feature extraction is performed using a custom convolution operation in the recognition network. Compared with feature extraction using only a standard convolution kernel, the unique features of the illustrations can be deeply mined and the richness and effectiveness of the features can be enhanced, providing richer and more accurate information for classification prediction, reducing unnecessary computational effort, and improving the processing efficiency and accuracy of the recognition network.

[0136] In some embodiments, for the above-mentioned target network, i.e., the detection network or the recognition network, the multiple convolution kernels used by each convolution layer of the target network may include a first convolution kernel of a first proportion for realizing the upper and lower convolution, a second convolution kernel of a second proportion for realizing the left and right convolution, and a third convolution kernel of a third proportion for realizing standard convolution; the sum of the three proportions is 1.

[0137] That is, each convolution layer of the target network includes three convolution operations, namely, upper and lower convolution, left and right convolution, and standard convolution, and the corresponding number of convolution kernels is allocated according to the set ratio. The standard convolution includes: for the image area covered by the third convolution kernel, each weight of the third convolution kernel is multiplied by the pixel value of the corresponding position on the covered image area, and all products are added to obtain the convolution result of the third convolution kernel on the covered image area; the upper and lower convolution or left and right convolution performs feature extraction according to the convolution process introduced in the above embodiment.

[0138] For example, each convolution layer of the detection network is provided with 10 convolution kernels. Assuming that when defining the convolution kernel of the convolution kernel of the convolution layer, the number of the first convolution kernel, the second convolution kernel and the third convolution kernel accounts for 1 / 4, 1 / 4, and 1 / 2 respectively, then according to this ratio, it can be set that 2 of the 10 convolution kernels of each convolution layer are the first convolution kernels for realizing the upper and lower convolutions, 2 convolution kernels are used to realize the left and right convolutions, and the remaining 6 convolution kernels are used as the third convolution kernels for realizing standard convolution.

[0139] Among them, the upper and lower convolution or the left and right convolution can also include the following processing steps: dividing the corresponding convolution kernel into two equal-sized sub-convolution kernels; calculating the convolution result obtained by weighted summation of each sub-convolution kernel, and obtaining the absolute value of the difference between the two convolution results; and obtaining the corresponding final convolution result according to the absolute value.

[0140] The above-mentioned custom convolution operation involves dividing the standard convolution kernel into two sub-convolution kernels of equal size in the vertical or horizontal direction, and using two sub-convolution kernels to perform convolution operations on corresponding positions of the image area covered by the standard convolution kernel, and finally calculating the difference between the convolution results of the two sub-convolution kernels as the convolution result of the custom convolution operation, which reflects the local difference of the image area in the vertical or horizontal direction.

[0141] In the process of dividing the convolution kernel into sub-convolution kernels of equal size, in order to be able to divide the standard convolution kernel into two sub-convolution kernels of equal size in the vertical or horizontal direction, the height of the first convolution kernel corresponding to the upper and lower convolutions is set to an even number, and the width of the second convolution kernel corresponding to the left and right convolutions is set to an even number. Since all convolution kernels of the same convolution layer are usually the same size, the height and width of the convolution kernel of this layer can be set based on the custom convolution operation included in the convolution layer of the detection network or recognition network. For example, in the case where the convolution layer includes upper and lower convolutions, the height of the convolution kernel can be set to an even number without limiting the width; in the case where the convolution layer includes left and right convolutions, the width of the convolution kernel can be set to an even number without limiting the height; in the case where the convolution layer includes both upper and lower convolutions and left and right convolutions, the width and height of the convolution kernel can be set to be even numbers.

[0142] When the height of the first convolution kernel corresponding to the upper and lower convolutions or the width of the second convolution kernel corresponding to the left and right convolutions is an odd number, the weights of the bottom or top row of the first convolution kernel can be reused, and adjacent rows with the same weights are added to the reused rows to make the height of the first convolution kernel an even number. The weights of the rightmost or leftmost column of the second convolution kernel can be reused, and adjacent columns with the same weights are added to the reused columns to make the width of the second convolution kernel an even number.

[0143] For example, Figure 3As shown, the first convolution kernel K1 is 3*3, and the weight elements X31, X32 and X33 of the bottom row of K1 can be reused, and adjacent rows with weight elements X31, X32 and X33 are added to the bottom row. The K1 is updated to 4*3, so that K1 can be divided into two sub-convolution kernels of 2*3; similarly, the second convolution kernel K2 is 3*3, and the weight elements X13, X23 and X33 of the rightmost row of K2 can be reused, and adjacent columns with weight elements X13, X23 and X33 are added to the rightmost row. The K1 is updated to 3*4, so that K2 can be divided into two sub-convolution kernels of 3*2.

[0144] Based on the above description of the implementation steps of the upper and lower convolution kernels and the left and right convolution, the convolution process can be represented by the following formula: Y = F (abs (W1 * X1-W2 * X2) + b). Among them, W1 and W2 represent two equal-sized sub-convolution kernels obtained by dividing the convolution kernel W along the horizontal or vertical direction, (W1 + W2) represents the convolution kernel W, X1 and X2 represent the image blocks corresponding to the sub-convolution kernels, (X1 + X2) constitutes the image area covered by the convolution kernel W, b represents the bias term of the convolution kernel W, abs () represents the absolute value of the difference, and F () represents the activation function to map the input of the neuron to the output. The mapping is usually nonlinear to facilitate the network to learn complex patterns and data representations.

[0145] In the embodiments of the present disclosure, local changes or edge information of an image or feature map in the vertical or horizontal direction can be captured through a custom convolution operation, thereby enriching the feature content. Compared with feature extraction only through standard convolution, different types of convolution operations realize multi-dimensional feature capture, making the network more robust to changes in rotation, tilt, etc. in the image, providing more effective feature representations, and facilitating the detection network and recognition network to better understand the image content, thereby improving the accuracy of image classification.

[0146] In some embodiments, in the aforementioned step S104, when the second classification result matches the first classification result, determining the corresponding page number of the page to be identified in the paper document can first determine the classification categories that meet the respective set confidence requirements in the first classification result and the second classification result, and then perform matching judgment and corresponding page number determination, see Figure 4 The flowchart shown can be implemented by including the following steps:

[0147] S401, obtaining a first prediction category with the highest confidence in the first classification results;

[0148] The first classification result is generated by the detection network, which is used for image detection and preliminary classification of images. The preliminary classification of images is a rough classification, so the first classification result may include one or more first prediction categories with the highest confidence. For example, if the first classification result includes two second prediction categories with a confidence of 40%, then the first classification result includes two first prediction categories with the highest confidence.

[0149] S402, obtaining a second prediction category with the highest confidence in the second classification results;

[0150] S403, determining whether the second prediction category meets a set condition, wherein the set condition includes: the confidence of the second prediction category is greater than or equal to a preset threshold, and the second prediction category is the same as the first prediction category;

[0151] That is, the second prediction category must not only satisfy that its corresponding confidence is greater than or equal to the preset threshold, but also satisfy that the second prediction category is the same as the first prediction category with the highest confidence included in the first classification result. Wherein, each classification category based on the detection network and the recognition network corresponds to all the illustrations of a single page in the paper document, then the second prediction category is the same as the first prediction category means that the pages in the paper document corresponding to the two prediction categories are the same page. In the case where the first classification category includes multiple first prediction categories with the highest confidence, if the second prediction category is the same as any of the first prediction categories, it can be determined that the second prediction category is the same as the first prediction category.

[0152] S404a, if the set condition is met, determining that the second classification result matches the first classification result, and obtaining the page number of the page in the paper document corresponding to the second predicted category as the corresponding page number;

[0153] S404b: If the set condition is not met, it is determined that the second classification result does not match the first classification result, and a preset mismatch processing flow is executed.

[0154] If the second prediction category meets the set conditions, it means that the prediction results of the detection network and the recognition network point to the same prediction category and the prediction category has a high confidence level, which can be used as the prediction result of the illustration for the page to be identified. Then, according to the page number information of each page in the paper document stored in advance, the page number of the page in the paper document corresponding to the second prediction category can be obtained. Based on the prediction that the illustration on the page to be identified comes from the page in the paper document corresponding to the second prediction category, it can be determined that the page to be identified is the same page as the page in the paper document corresponding to the second prediction category. Therefore, the page number of the page in the paper document corresponding to the second prediction category can be used as the corresponding page number of the page to be identified in the paper document.

[0155] If the second prediction category does not meet the set conditions, a preset mismatch processing flow can be executed, such as further identifying the page to be identified by performing image feature extraction or text content extraction for further matching processing. For details, please refer to the subsequent embodiments.

[0156] In the embodiment of the present disclosure, by setting a condition requiring that the confidence of the second prediction category is greater than or equal to a preset threshold, and requiring the second prediction category to be the same as the first prediction category, it is ensured that the prediction results with high confidence in the recognition network used for accurate classification of the illustrations are further considered, and the consistency between the prediction results of different networks is enhanced to enhance the possibility of correct prediction results, thereby reducing misjudgments and uncertainties and improving the accuracy of the overall prediction.

[0157] In some embodiments, in the case where the second classification result does not match the first classification result, that is, the case where the second prediction category does not meet the set conditions as described in the above embodiment, the preset mismatch processing flow may include at least one of image feature extraction or text content extraction for further matching processing of the page to be identified. The image feature extraction for further matching processing of the page to be identified may be implemented by the following steps:

[0158] The image feature vector of the page to be identified is extracted to obtain a first feature vector of fixed dimension; the first feature vector is matched with a feature vector of a first page containing a picture in the pre-stored paper document, and the page number corresponding to the matched feature vector is determined as the corresponding page number.

[0159] The image feature vector extraction refers to a method of converting the image information of the page to be identified into a numerical form, which is used to represent the key features of the image. The first feature vector extracted may include but is not limited to the color, texture, shape, etc. of the page to be identified. Regarding the feature vector extraction process, it can be implemented by a preset feature extraction algorithm such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (Oriented FAST and Rotated BRIEF, with directional FAST feature points and rotated BRIEF descriptors), etc.

[0160] The first feature vector can also be determined based on the feature representation extracted by the detection network using the convolution layer for the input page to be identified when performing image detection, that is, the feature map output by one or some convolution layers in the detection network can be further processed (such as through global average pooling or global maximum pooling) to obtain a feature vector of fixed dimension as the first feature vector.

[0161] When executing this embodiment, for each first page including illustrations in the paper document, a fixed-dimensional feature vector of the first page is obtained through the same image feature vector extraction method, and the feature vector of each first page and the page number information of the first page can be pre-stored in a preset database such as an Elasticsearch library to facilitate fast search and calculation of feature vector similarity.

[0162] After extracting the first feature vector of the page to be identified during the application process, the first feature vector of the page to be identified is compared with the feature vectors of each pre-stored first page to find the most similar or most matching feature vector pair. The distance between the feature vectors, such as Euclidean distance, cosine similarity, etc., can be calculated, and the degree of matching can be judged according to the distance. At the same time, a matching threshold can be set as a boundary, and the feature vector of the first page whose distance is greater than the matching threshold is determined as the matched feature vector. Based on the pre-stored feature vectors of each first page and the page number information of the first page, the page number of the first page where the matched feature vector is located can be obtained and used as the corresponding page number of the page to be identified in the paper document.

[0163] In the disclosed embodiment, in the case where the second classification result does not match the first classification result, by calculating the distance between the feature vectors, the similarity between the page to be identified and the pre-stored page can be accurately quantified, and the page with the content closest to the page to be identified can be accurately identified, thereby determining the exact page number of the page to be identified in the paper document. In addition, the feature vector matching method is not affected by factors such as page layout, font size, color change, etc., and accurately identifies based on the principle of page content similarity, which is applicable to a variety of different paper document types and enhances the flexibility of image recognition.

[0164] The method of further matching the text content extraction described in the above embodiment may include the following steps: using OCR technology or other text extraction technology, identifying and extracting the text content of the page to be identified, and matching the text content with the page text of each first page containing the illustration in the pre-stored paper document, and determining the corresponding page number of the matched page text (i.e., the text content matching degree is greater than the set similarity threshold such as 96%) as the corresponding page number of the page to be identified in the paper document. The processing flow of text content extraction and matching can be performed after the image feature extraction and further matching of the page to be identified fails, as a supplementary solution for the image feature vector.

[0165] In some embodiments, in order to improve the efficiency and accuracy of text extraction or image processing, this embodiment adds a blurred image judgment step before the aforementioned step S102, which is used to pre-identify blurred images caused by unclear shooting, inaccurate focus, or damage during transmission, thereby avoiding unnecessary processing of such blurred images, saving computing resources, and improving the accuracy of subsequent image recognition. The blurred image judgment may include the following steps:

[0166] The custom convolution operation introduced in the above embodiment is used to obtain the page features of the page to be identified, and based on the page features, it is detected whether the clarity of the page to be identified meets the set requirements; when the clarity meets the set requirements, it is detected whether the page to be identified includes an illustration.

[0167] That is, at least one of the convolution processing methods of the upper and lower convolutions and the left and right convolutions in the custom convolution operation described in the aforementioned embodiment can be used to perform page feature extraction on the page to be identified in step S101 using a convolution operation, and the key information of the page can be stored in the form of a feature map, and then the clarity can be predicted using the extracted page based on a pre-built clarity evaluation algorithm. The set requirements can be quantified as a clarity threshold according to the needs of the application scenario, and when the clarity score of the page to be identified is higher than or equal to the clarity threshold, it is determined that the clarity of the page to be identified meets the set requirements.

[0168] In some embodiments, the fuzzy image determination step described in the above embodiment can be implemented by a pre-trained fuzzy determination network. Figure 5 As shown in the flowchart, in the blurred image determination step, the convolution operation is used to obtain the page features of the page to be identified, and the clarity of the page to be identified is detected according to the page features to determine whether it meets the set requirements, which may include the following processing flow:

[0169] S501, inputting the page to be identified into a pre-trained fuzzy judgment network;

[0170] The fuzzy judgment network is used to identify whether the image input to the network is clear. It can realize fuzzy judgment through an input layer, a hidden layer such as multiple convolutional layers, a pooling layer, an activation layer, etc., and an output layer. The input layer receives the input image, the hidden layer extracts and converts the input image, and the output layer outputs the judgment result according to the extracted features. The output layer of the network can be designed to output two categories, clear and fuzzy, and is implemented by a softmax function or other multi-classification function to convert the output of the hidden layer into a probability distribution of the two categories. The final judgment result usually takes the category with a higher probability as the output, that is, if the probability of the clear category is higher than that of the fuzzy category, the image is judged to be clear; otherwise, it is judged to be fuzzy.

[0171] In the pre-training process of the fuzzy judgment network, the training set includes clear images and fuzzy images, and each image in the training set is labeled to clarify whether it is a clear image or a fuzzy image. Then, the fuzzy judgment network is trained using the training set so that the network continuously learns from the training set how to judge whether the image is clear based on its features. The pre-training can be done in a supervised learning manner, that is, the network receives an input image and a corresponding label (clear or fuzzy) in each iteration, and adjusts the network parameters according to the error between the network output and the true label through a gradient descent algorithm, an Adam algorithm, etc. to minimize a loss function such as a cross entropy loss function.

[0172] S502, extracting edge features of the page to be identified through the custom convolution operation in the fuzzy judgment network and transmitting them to subsequent layers of the fuzzy judgment network to obtain a fuzzy classification result of the page to be identified;

[0173] In this embodiment, based on the fact that the edges of clear pages are sharp and the edges of blurred pages are relatively blurred, the clarity of the edge can be used as one of the judgment indicators of whether the image is blurred. Therefore, after the convolution layer of the blur judgment network introduces a custom convolution operation, the edge features in the page to be identified can be better captured through at least one of the upper and lower convolutions or the left and right convolutions.

[0174] The extracted edge features are used as the input of the middle layer of the fuzzy judgment network and are passed to subsequent network layers, such as more convolution layers, pooling layers, activation layers, etc., and more convolution operations are performed to extract higher-level features, or pooling operations are performed to reduce the dimension of the features and retain important information. At the same time, the application of activation functions can increase the nonlinearity of the network and improve the feature expression ability. After a series of network layer processing, the final features are passed to the output layer of the fuzzy judgment network. The output layer makes classification decisions based on the complex features received, outputs the probability distribution of two categories (clear and fuzzy) through mechanisms such as the softmax function, and takes the category with a higher probability as the final fuzzy classification result. When determining the final fuzzy classification result, when obtaining the category with a higher probability between the clear and fuzzy categories, it can be further determined whether the higher probability is greater than or equal to the set threshold, and the fuzzy classification result is determined to be valid when it is greater than or equal to the set threshold.

[0175] In order to more accurately judge the blurriness of an image, the blur judgment network captures and integrates multiple types of image features, including but not limited to edge features, by using a multi-layer convolutional neural network. Each layer of convolution extracts different types of features and passes them to subsequent layers for further processing and abstraction.

[0176] S503a, when the fuzzy classification result indicates that the page to be identified is not fuzzy, determining that the clarity of the page to be identified meets the set requirements, and detecting whether the page to be identified includes a picture;

[0177] S503b, when the fuzzy classification result indicates that the page to be identified is fuzzy, it is determined that the clarity does not meet the set requirements, and a preset interactive prompt processing flow is executed.

[0178] If the clarity of the page to be identified does not meet the set requirements, the subsequent processing flow for the page to be identified, that is, the process of detecting whether the accompanying image is included, will be terminated, and the preset interactive prompt processing flow will be executed. The preset interactive prompt processing flow is used to display an error message to the user to inform that the collected page to be identified is blurry and a clearer version needs to be uploaded again, or the page to be identified can be enhanced such as sharpening and denoising to re-evaluate its clarity.

[0179] For example, if the page to be identified is judged as blurry during the fuzzy classification process, the page will not be checked for images, but a warning message such as "The page is blurry, please upload a clearer version" will be displayed to the user. At the same time, a button can be provided to allow the page to be identified to be re-uploaded. If re-uploaded, the clarity assessment and subsequent processing flow will be performed again after receiving the updated page to be identified.

[0180] In the disclosed embodiment, by introducing a custom convolution operation into the fuzzy judgment network, the fuzzy judgment network is enhanced in capturing edge features of the input image. Compared with traditional convolution kernels, the convolution operation can more effectively extract subtle differences between blurred and clear pages, thereby improving the accuracy and robustness of fuzzy classification.

[0181] In some embodiments, when the page to be identified does not include an image, that is, when the page to be identified is a pure text page, the text content of the page to be identified can be identified and extracted through OCR technology or other text recognition technology, and then the extracted text content can be matched with the page text of each second page that does not contain an image in the pre-stored paper document, and the page text most similar to the extracted text content is found, and the page number of the corresponding second page is used as the corresponding page number of the page to be identified in the paper document. However, text extraction is susceptible to various factors, such as page quality, font style, and layout complexity, which affect the accuracy of text extraction, thereby causing subsequent matching failures or matching errors. Therefore, this embodiment provides a processing method that combines text content recognition and page feature vector recognition, see Figure 6 As shown in the flowchart, the processing method may include the following processing steps:

[0182] S601, extracting the text content of the page to be identified, and matching it with the pre-stored page text of each second page that does not contain a picture in the paper document, and obtaining a first page number corresponding to the matched page text;

[0183] S602, extracting an image feature vector from the page to be identified to obtain a second feature vector of a fixed dimension, and matching the second feature vector with a pre-stored feature vector of the second page to obtain a second page number of the second page corresponding to the matched feature vector;

[0184] That is, the page text content, page number information and the fixed-dimensional feature vector extracted from the image feature vector of each second page in the paper document that does not contain illustrations are pre-stored and can be stored in a preset database such as the Elasticsearch library to facilitate fast text matching and feature vector similarity calculation.

[0185] In the case where the page to be identified does not include an image, the text content on the page to be identified can be extracted by text recognition technology, and the second feature vector of a fixed dimension of the page to be identified can be extracted by the image feature vector, and the text content and the second feature vector are matched with the page text and the feature vector of the second page stored in the preset database, respectively, to obtain the first page number of the page text that best matches the text content, and the second page number of the feature vector of the second page that is most similar to the second feature vector. The above steps S601 and S602 can be performed in parallel to improve processing efficiency.

[0186] S603, determining whether the first page number is the same as the second page number;

[0187] S604a, if they are the same, determining the first page number as the corresponding page number of the page to be identified in the paper document;

[0188] In this embodiment, the possible page numbers of the same page to be identified in the paper document are determined by two different methods, text matching and feature vector matching. When the results obtained by the two methods are verified to be consistent, the page number recognition result is determined to be valid, thereby accurately determining the corresponding page number of the page to be identified. The first page number is the same as the second page number, indicating that the page numbers determined by the two different methods of text matching and feature vector matching point to the same page of the paper document, which increases the accuracy and reliability of the recognition result, so that the first page number can be determined as the corresponding page number of the page to be identified in the paper document.

[0189] S604b: If different, execute the preset interactive processing flow.

[0190] The preset interactive processing flow is a processing method when the first page number and the second page number are different, and can be designed according to user needs and scenarios. For example, the preset interactive processing flow can be set to prompt the user that the recognition fails.

[0191] In one example, in order to improve the user experience and ensure the accuracy of page number recognition, when the first page number is different from the second page number, the second page corresponding to the first page number and the second page number can be controlled to be displayed on the interactive interface; then, the page number selected by the user from the displayed second page through the interactive interface is received, and the selected page number is used as the corresponding page number of the page to be identified in the paper document, thereby ensuring the flexibility of recognition, enhancing user participation, and improving the accuracy of the overall processing.

[0192] In the disclosed embodiment, the double verification method of text matching and image feature vector matching of the page to be identified is used to improve the accuracy and robustness of page number recognition and effectively reduce the occurrence of misrecognition.

[0193] In some embodiments, a response processing method is also provided, which can be executed by any electronic device capable of running a program, including but not limited to a mobile phone, a computer, a tablet, a learning machine, etc. The response processing method uses the image recognition method described in any of the above embodiments to recognize the acquired image and perform subsequent device processing. The method may at least include the following steps:

[0194] When a page image of the target paper document is obtained, the image recognition method described in any of the above embodiments is used to determine that the page image corresponds to a target page number in the target paper document; based on a preset mapping relationship between the page of the target paper document and the device operation, the device is triggered to execute the device operation corresponding to the target page number.

[0195] Before executing the response processing method, a mapping relationship between pages and device operations is predefined and stored in the electronic device. The mapping relationship indicates the correspondence between each page of the target paper document or specific content on a page and one or more device operations. The mapping relationship can be flexibly set according to actual needs. For example, the mth page on the target paper device corresponds to playing the set demonstration video P on the electronic device, and the m+1th page corresponds to opening another application of the electronic device and demonstrating the usage steps of the application in the interactive interface of the application.

[0196] After determining the target page number of the page image through image recognition methods, the electronic device can automatically find and trigger the device operation associated with the target page number according to the preset mapping relationship. The user does not need to operate the device manually, but can control the operation of the device by simply displaying a specific page in the paper document, greatly improving the convenience and efficiency of operation.

[0197] The above response processing method can be widely used in various scenarios that require the combination of paper documents and electronic equipment interaction and automated control, such as smart home control, industrial automation production lines, library book management, medical equipment operation demonstrations, new equipment startup guide manuals and real-time guidance demonstrations, etc. By introducing image recognition technology, a more intelligent and user-friendly interaction method and device response process can be achieved, thereby improving user experience and work efficiency.

[0198] For the process of the electronic device acquiring the page image of the target paper document, the electronic device may include or be connected to an image acquisition device such as a scanner, a camera, a camera lens group or a scanning / recognition area of ​​a learning machine, etc. When the image acquisition device is in a working state, it continuously monitors whether there is an object placed in the scanning area, and when it is detected that a paper document is placed in the scanning area, it starts the image capture function to acquire the image corresponding to the page of the paper document displayed in the current scanning area as the page image. During the image acquisition process, the image acquisition device may apply image processing techniques such as automatic exposure control, color correction and noise reduction to improve the quality and clarity of the image.

[0199] Next, in order to enable those skilled in the art to better understand the image recognition method and response processing method provided by this application, this embodiment takes the scenario in which the power-on boot manual of the learning machine triggers the instructions of the real-time demonstration manual of the learning machine as an example to exemplify and illustrate the above-mentioned image recognition method and response processing method.

[0200] In educational scenarios, the learning machine is used as an auxiliary learning tool with rich built-in learning resources and interactive teaching to facilitate students' independent learning. In order to make it easier for novice users to learn how to use the learning machine, a boot guide manual is designed for the learning machine and a built-in boot guide application. After the user opens the boot guide application, the boot guide manual is placed in the scanning area of ​​the learning machine. The boot guide application will automatically identify which page of the guide manual is placed in the scanning area, and then explain the relevant learning machine knowledge to the user based on the pre-created content mapping table and the content of the page, or open the corresponding application for a real-time demonstration of the function.

[0201] Since the content of each page of the boot guide manual is different, some pages contain both text and pictures, and some pages only include pictures or only text. In order to improve the page recognition accuracy of the boot guide application, the image recognition method provided by this application can be used to implement the page recognition of the application, see Fig. 7A The flowchart shown may include the following processing steps:

[0202] S701, page image acquisition: acquiring a page image in the boot manual;

[0203] The boot manual can be placed directly in the scanning area of ​​the learning machine, and an image acquisition device such as a high-definition camera configured on the learning machine can capture the image of the manual page displayed in the scanning area; alternatively, the user can take a photo of the designated manual page of the boot manual and upload it to the learning machine via an album or the like.

[0204] S702, fuzzy image judgment: preprocess the page image to obtain a first image that meets the input layer requirements of a pre-trained fuzzy judgment network, and input the first image into the fuzzy judgment network so that the fuzzy judgment network extracts edge features of the first image through a custom convolution operation and performs subsequent feature conversion, and outputs a classification category and confidence level of whether the first image belongs to fuzzy or clear; when the output classification category is fuzzy and the confidence level is greater than a fuzzy threshold, the first image is determined to be fuzzy.

[0205] This embodiment uses a convolution-based neural network for fuzzy classification, which can be a Mobilenet V3 network. In actual applications, other types of networks can also be used for classification judgment. The classification task based on the fuzzy judgment network is to judge whether the input image is blurred. Since the edge features of blurred images are weakened, the standard convolution operation may not be able to extract effective edge information. Therefore, this embodiment improves the memory of the standard convolution operation to enhance the richness and effectiveness of the extracted image edge features.

[0206] like Figure 7B Shown on the left is a 4*4 convolution kernel. If the image data currently undergoing convolution calculation is the 4*4 image data shown on the right, the calculation process of the standard convolution operation can be expressed as: Y=F(W*X+b), where F is the activation function, W is the convolution kernel parameter, i.e. the 4*4 matrix on the left side of the above figure, X is the image data currently undergoing convolution calculation, i.e. the 4*4 matrix on the right side of the above figure, and b is the bias. Each weight on the convolution kernel is multiplied by the element at the corresponding position on the image data, and all the products are added together to obtain the result value of W*X.

[0207] The custom convolution operations introduced in this embodiment include upper and lower convolution and left and right convolution. Figure 7B The 4*4 convolution kernel and image data shown in the figure, the upper and lower convolution operations may include: dividing the parameters of the convolution kernel into two equal parts along the horizontal direction, such as Figure 7C As shown, W1 is the first row w11 to w14 and the second row w21 to w24, W2 is the third row w31 to w34 and the fourth row w41 to w44. Similarly, the image data currently undergoing convolution operation is also divided into upper and lower parts. X1 is the first row x11 to x14 and the second row x21 to x24, X2 is the third row x31 to x34 and the fourth row x41 to x44. The calculation process of upper and lower convolution can be expressed as: Y=F(abs(W1*X1-W2*X2)+b), where the calculation process of W1*X1 and W2*X2 is the same as the calculation process and principle of the standard convolution operation.

[0208] Similarly, the left and right convolution operations can include: dividing the parameters of the convolution kernel into two equal parts along the vertical direction, such as Fig.7DAs shown, W3 is the first column w11 to w41 and the second column w12 to w42, W4 is the third column w13 to w43 and the fourth column w14 to w44. Similarly, the image data currently undergoing convolution operation is also divided into left and right parts. X3 is the first column x11 to x41 and the second column x12 to x42, X4 is the third column x13 to x43 and the fourth column x14 to x44. The calculation process of left and right convolution can be expressed as: Y=F(abs(W3*X3-W4*X4)+b), where the calculation process of W3*X3 and W4*X4 is the same as the calculation process and principle of the standard convolution operation.

[0209] The custom convolution operation of this embodiment is more conducive to extracting image edge features, and is better than the standard convolution operation for judging whether an image is blurred, thereby enhancing the generalization ability of the fuzzy judgment network. In order to facilitate the operation of upper and lower convolutions or left and right convolutions, the convolution kernel size included in the convolution layer of the fuzzy judgment network can be set to an even number * even number.

[0210] S703, image detection and rough classification processing:

[0211] When the page image is identified as not a fuzzy image by the fuzzy judgment network, the page image is detected by using a pre-trained detection network, and when it is detected that the page image includes the picture, the detection frame of the picture and the first classification category with the highest confidence after the detection network performs preliminary classification on the picture are output; the multiple classification categories set in the output layer of the detection network correspond to the page numbers of the first manual pages containing the pictures in the boot manual of the learning machine one by one;

[0212] Before the page image is input into the detection network, the page image is preprocessed, such as scaling to a standard size and image data normalization, to obtain a second image that meets the input layer requirements of the detection network, and the second image is input into the detection network.

[0213] When the detection frame information of the accompanying picture is obtained, the detection frame information and the page image are input into the pre-trained recognition network to enter the accompanying picture precise classification processing in step S704; when the page image does not include the accompanying picture, the text and image search processing is entered in step S706.

[0214] The detection network used in this embodiment is the Yolo v5 network. When performing convolution operations, left and right convolutions and upper and lower convolution operations can be introduced in some convolution layers of the detection network, and the convolution operations occupied by the convolution kernels used to implement standard convolution, left and right convolution, and upper and lower convolution in the convolution layer are set to be 1 / 2, 1 / 4, and 1 / 4, respectively. The specific ratio can be set according to actual needs. After introducing upper and lower convolutions and left and right convolutions, the generalization ability of the detection network and the detection ability of the matching map are improved. Other convolutional networks can also be used in actual applications, and the ratio of upper and lower convolutions and left and right convolutions can be adjusted according to the application scenario.

[0215] S704, accurate classification and processing of pictures:

[0216] According to the received detection frame information and the page image, a picture image corresponding to the detection frame is cut out from the page image, and the picture image is input into the recognition network, so that the recognition network predicts the classification category and confidence of the picture image based on the parameters learned in the pre-training stage, and obtains the second classification category with the highest confidence; the multiple classification categories set in the output layer of the recognition network correspond to the page numbers of the first manual pages containing pictures in the boot manual of the learning machine one by one;

[0217] The recognition network used in this embodiment is the ShuffleNetV2 network. When performing convolution operations, left and right convolutions and upper and lower convolution operations can be introduced for some convolution layers of the recognition network, and the convolution operations occupied by the convolution kernels used to implement standard convolution, left and right convolution, and upper and lower convolution in the convolution layer are set to be 1 / 2, 1 / 4, and 1 / 4, respectively. The specific ratio can be set according to actual needs. After introducing upper and lower convolutions and left and right convolutions, the generalization ability of the recognition network and the detection ability of the matching pictures are improved. Other convolutional networks can also be used in actual applications, and the ratio of upper and lower convolutions and left and right convolutions can be adjusted according to the application scenario. After introducing the upper and lower convolution kernels and left and right convolutions, the network can be helped to extract the edge features of the image well, and the recognition ability of the recognition network can be improved for accurate classification of matching pictures.

[0218] S705, dual network classification result verification processing:

[0219] With respect to the second classification category of the identification network output and the first classification category of the detection network output, when the page number corresponding to the second classification category is the same as that of the first classification category and the confidence of the second classification category is greater than the set confidence threshold, the page number corresponding to the second classification category is determined as the corresponding page number of the page image obtained in step S701 in the boot manual; otherwise, the text and image search process of step S706 is entered.

[0220] S706, text and image search processing:

[0221] The image feature vector, page text and corresponding page number of each page of the boot manual are pre-stored in a preset database such as an Elasticsearch library.

[0222] When entering the text and image search process, the text content of the page image can be extracted using OCR technology, and a text search is performed in the preset database according to the text content to find the page text most similar to the text content, and the search result is determined to be valid when the similarity is greater than a first threshold, and the page number corresponding to the most similar page text is determined as the corresponding page number of the page image in the boot manual;

[0223] Alternatively, an image feature vector extraction algorithm may be used to extract a third feature vector of a fixed dimension of the page image, and a similar vector search may be performed in the preset database based on the third feature vector. The similarity of the feature vectors may be determined by calculating the vector distance to find an image feature vector of a manual page of the boot manual whose similarity to the third feature vector is greater than a second threshold value, and the page number corresponding to the image feature vector of the most similar manual page may be determined as the corresponding page number of the page image in the boot manual.

[0224] This embodiment implements the image feature vector extraction using the Mobilenetv3 small network, and other networks may also be used in actual applications. When performing convolution operations, left-right convolution and upper-lower convolution operations may be introduced for some convolution layers of the recognition network, and the convolution operation ratios of the convolution kernels used to implement standard convolution, left-right convolution, and upper-lower convolution in the convolution layer are set to 1 / 2, 1 / 4, and 1 / 4, respectively. The specific ratios may be set according to actual needs.

[0225] The text content search or the third feature vector search described above can be used separately, that is, only one of them can be used. In order to improve the accuracy of image page recognition, when entering the text and image search process through the dual network classification result verification process, it is preferred to use the third feature vector search method of the page image to determine the corresponding page number; when entering the text and image search process when the page image does not include an illustration, the text content search and the third feature vector search of the page image can be used at the same time, and the two page numbers determined by the two methods are compared. If the two page numbers are the same, the page number is used as the corresponding page number, otherwise an unrecognizable prompt result is output. Compared with the related technology that only uses pure text recognition, the recognition accuracy and flexibility are improved.

[0226] Through the overall processing flow of the above-mentioned image recognition method and the upper and lower convolutions or left and right convolutions operations introduced for each network, the richness and effectiveness of the extracted features are enhanced, the accuracy of image page number recognition and the robustness of network processing are effectively improved, and effective feature extraction and image page number recognition can also be provided for page number images with poor image quality due to poor lighting conditions or other conditions.

[0227] Based on the image recognition method, in the application scenario of the learning machine, the response processing method can be applied to the learning machine, and the method may include the following steps:

[0228] When a page image of one of the pages of the power-on boot manual is obtained, the above-mentioned image recognition method is used to determine that the page image corresponds to a target page number of the power-on boot manual; according to the content mapping table of the power-on boot manual page and the boot demonstration operation preset in the learning machine, the learning machine is triggered to execute the boot demonstration operation associated with the power-on boot manual page corresponding to the target page number.

[0229] For example, the backend of the boot guide application creates a content mapping table to associate each page in the boot guide manual with a specific demonstration video, animation demonstration, or a specific functional interface of a specific application. For example, page 5 of the manual is associated with the "mental arithmetic correction" function. Before and after the user places a page of the manual on the learning machine lens, the learning machine captures the input image of the manual page, and identifies the corresponding page number of the input image in the boot guide manual through the above-mentioned image recognition method. By querying the content mapping table, the operation associated with the page number is determined and started. Assuming that the recognized page number is page 5, the boot guide application attempts to start the pre-installed oral arithmetic APP, and guides the user to click on the page according to the "step 1-step 2" interaction to enter the functional interface, such as the correction mode; guide the user to click on the correction mode, and give a correction demonstration for the oral arithmetic problem on the input image on the current page.

[0230] Corresponding to the above-mentioned embodiment of the image recognition method, see Figure 8 As shown, the present application also provides an embodiment of an image recognition device, the device comprising:

[0231] The to-be-recognized page acquisition module 801 is used to acquire a page image of a paper document as a to-be-recognized page;

[0232] The picture detection and recognition module 802 is used to detect whether the page to be recognized includes a picture using a pre-trained detection network, and if the page to be recognized includes a picture, determine a first classification result of the picture; cut the picture from the page to be recognized, and input the picture into a pre-trained recognition network to obtain a second classification result of the picture; wherein the detection network and the recognition network both include multiple classification categories, each of which corresponds to all the pictures of a single page in the paper document;

[0233] The target page determination module 803 is used to determine the corresponding page number of the to-be-identified page in the paper document when the second classification result matches the first classification result.

[0234] In some embodiments, the target network includes the detection network or the recognition network; each convolution layer of the target network includes at least one of the upper and lower convolutions and the left and right convolutions in the custom convolution operation; the upper and lower convolutions include horizontally dividing the first convolution kernel into two sub-convolution kernels of equal size and then calculating the difference between the upper and lower convolution results; the left and right convolutions include vertically dividing the second convolution kernel into two sub-convolution kernels of equal size and then calculating the difference between the left and right convolution results.

[0235] In some embodiments, based on the custom convolution operation, the image detection and recognition module is specifically used to:

[0236] Determine a bounding box of the image using the detection network, and determine a region of interest in the page to be identified based on the bounding box;

[0237] Through the custom convolution operation in the detection network, a first image feature of the region of interest is extracted, and a first classification result of the image is predicted based on the first image feature.

[0238] In some embodiments, based on the custom convolution operation, the image detection and recognition module is specifically used to:

[0239] The page to be identified is cropped according to the boundary box of the image determined by the detection network to obtain the image and input it into the recognition network;

[0240] Through the custom convolution operation in the recognition network, a second picture feature of the picture is extracted, and a second classification result of the picture is predicted based on the second picture feature.

[0241] In some embodiments, the multiple convolution kernels used in each convolution layer of the target network include a first convolution kernel for implementing the up-down convolution in a first ratio, a second convolution kernel for implementing the left-right convolution in a second ratio, and a third convolution kernel for implementing the standard convolution in a third ratio;

[0242] The upper and lower convolution or the left and right convolution includes: dividing the corresponding convolution kernel into two equal-sized sub-convolution kernels; calculating the convolution result obtained by weighted summation of each sub-convolution kernel, and obtaining the absolute value of the difference between the two convolution results; and obtaining the corresponding final convolution result according to the absolute value.

[0243] In some embodiments, the target page determination module is specifically used to:

[0244] Obtaining a first prediction category with the highest confidence in the first classification result, and obtaining a second prediction category with the highest confidence in the second classification result;

[0245] When the second prediction category meets the set conditions, it is determined that the second classification result matches the first classification result, and the page number of the page in the paper document corresponding to the second prediction category is obtained as the corresponding page number; the set conditions include: the confidence of the second prediction category is greater than or equal to a preset threshold, and the second prediction category is the same as the first prediction category.

[0246] In some embodiments, the apparatus further comprises:

[0247] When the second classification result does not match the first classification result, extracting an image feature vector of the page to be identified to obtain a first feature vector of a fixed dimension;

[0248] The first feature vector is matched with a pre-stored feature vector of a first page of the paper document containing a picture, and a page number corresponding to the matched feature vector is determined as the corresponding page number.

[0249] In some embodiments, the target network further includes a pre-trained fuzzy judgment network; the device further includes:

[0250] By means of the custom convolution operation in the fuzzy judgment network, edge features of the page to be identified are extracted and transmitted to subsequent layers of the fuzzy judgment network to obtain a fuzzy classification result of the page to be identified;

[0251] When the fuzzy classification result indicates that the page to be identified is not fuzzy, it is determined that the clarity of the page to be identified meets the set requirement; otherwise, it is determined that the clarity does not meet the set requirement.

[0252] When the definition meets the set requirement, it is detected whether the page to be identified includes a picture.

[0253] In some embodiments, the apparatus further comprises:

[0254] In the case that the page to be identified does not include a picture, extracting the text content of the page to be identified, and matching it with the page text of each second page that does not include a picture in the pre-stored paper document, and obtaining the first page number corresponding to the matched page text;

[0255] Extracting an image feature vector of the page to be identified to obtain a second feature vector of a fixed dimension, matching the second feature vector with a pre-stored feature vector of the second page, and obtaining a second page number of the second page corresponding to the matched feature vector;

[0256] In the case that the first page number is the same as the second page number, the first page number is determined as the corresponding page number of the to-be-identified page in the paper document.

[0257] In some embodiments, the apparatus further comprises:

[0258] When the first page number is different from the second page number, controlling the display of a second page corresponding to the first page number and the second page number on the interactive interface;

[0259] A page number selected by a user from a displayed second page through an interactive interface is received, and the selected page number is used as the corresponding page number.

[0260] Corresponding to the embodiment of the above-mentioned response processing method, see Fig. 9 As shown, the present application also provides an embodiment of a response processing device, the device comprising:

[0261] A page number determination module 901 is used to determine a target page number in the target paper document corresponding to the page image by using the image recognition method described in any of the above embodiments when a page image of the target paper document is acquired;

[0262] The trigger execution module 902 is used to trigger the device to execute the device operation corresponding to the target page number according to the preset mapping relationship between the page of the target paper document and the device operation.

[0263] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0264] The present application also provides an electronic device. The structural diagram of the electronic device is as follows: Fig.10As shown, the electronic device 1000 includes at least one processor 1001, a memory 1002 and a bus 1003, and the at least one processor 1001 is electrically connected to the memory 1002; the memory 1002 is configured to store at least one computer-executable instruction, and the processor 1001 is configured to execute the at least one computer-executable instruction, thereby performing the steps of any image recognition method or response processing method provided in any embodiment or any optional implementation manner in the present application.

[0265] Furthermore, the processor 1001 may be a Field-Programmable Gate Array (FPGA) or other devices with logic processing capabilities, such as a Microcontroller Unit (MCU) or a Central Process Unit (CPU).

[0266] An embodiment of the present application also provides another readable storage medium storing a computer program, which is used to implement the steps of any image recognition method or response processing method provided in any embodiment or any optional implementation manner of the present application when executed by a processor.

[0267] The readable storage medium provided in the embodiments of the present application includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the readable storage medium includes any medium that can store or transmit information in a readable form by a device (e.g., a computer).

[0268] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An image recognition method, characterized in that: The method comprises: Acquire a page image of a paper document as a page to be identified; Using a pre-trained detection network to detect whether the page to be identified includes a picture, and if the page to be identified includes a picture, determining a first classification result of the picture; Extracting the illustration from the page to be identified, and inputting the illustration into a pre-trained recognition network to obtain a second classification result of the illustration; wherein both the detection network and the recognition network include a plurality of classification categories, and each classification category corresponds to all the illustrations of a single page in the paper document; When the second classification result matches the first classification result, a corresponding page number of the to-be-identified page in the paper document is determined.

2. The method according to claim 1, characterized in that The target network includes the detection network or the recognition network; each convolution layer of the target network includes at least one of the upper and lower convolutions and the left and right convolutions in the custom convolution operation; the upper and lower convolutions include horizontally dividing the first convolution kernel into two sub-convolution kernels of equal size and then calculating the difference between the upper and lower convolution results; the left and right convolutions include vertically dividing the second convolution kernel into two sub-convolution kernels of equal size and then calculating the difference between the left and right convolution results.

3. The method according to claim 2, characterized in that When the page to be identified includes a picture, determining a first classification result of the picture includes: Determine a bounding box of the image using the detection network, and determine a region of interest in the page to be identified based on the bounding box; Through the custom convolution operation in the detection network, a first image feature of the region of interest is extracted, and a first classification result of the image is predicted based on the first image feature.

4. The method according to claim 2, characterized in that: The step of cutting the image from the page to be identified and inputting the image into a pre-trained recognition network to obtain a second classification result of the image includes: The page to be identified is cropped according to the boundary box of the image determined by the detection network to obtain the image and input it into the recognition network; Through the custom convolution operation in the recognition network, a second picture feature of the picture is extracted, and a second classification result of the picture is predicted based on the second picture feature.

5. The method according to any one of claims 2 to 4, characterized in that: The multiple convolution kernels used by each convolution layer of the target network include a first convolution kernel for implementing the up-down convolution in a first proportion, a second convolution kernel for implementing the left-right convolution in a second proportion, and a third convolution kernel for implementing the standard convolution in a third proportion; The upper and lower convolution or the left and right convolution includes: dividing the corresponding convolution kernel into two equal-sized sub-convolution kernels; calculating the convolution result obtained by weighted summation of each sub-convolution kernel, and obtaining the absolute value of the difference between the two convolution results; and obtaining the corresponding final convolution result according to the absolute value.

6. The method according to claim 1, characterized in that When the second classification result matches the first classification result, determining a corresponding page number of the to-be-identified page in the paper document includes: Obtaining a first prediction category with the highest confidence in the first classification result, and obtaining a second prediction category with the highest confidence in the second classification result; When the second prediction category meets the set conditions, it is determined that the second classification result matches the first classification result, and the page number of the page in the paper document corresponding to the second prediction category is obtained as the corresponding page number; the set conditions include: the confidence of the second prediction category is greater than or equal to a preset threshold, and the second prediction category is the same as the first prediction category.

7. The method according to claim 1, characterized in that The method further comprises: When the second classification result does not match the first classification result, extracting an image feature vector of the page to be identified to obtain a first feature vector of a fixed dimension; The first feature vector is matched with a pre-stored feature vector of a first page of the paper document containing a picture, and a page number corresponding to the matched feature vector is determined as the corresponding page number.

8. The method according to claim 2, characterized in that: The target network also includes a pre-trained fuzzy judgment network; the method also includes: By means of the custom convolution operation in the fuzzy judgment network, edge features of the page to be identified are extracted and transmitted to subsequent layers of the fuzzy judgment network to obtain a fuzzy classification result of the page to be identified; If the fuzzy classification result indicates that the page to be identified is not fuzzy, determining that the clarity of the page to be identified meets the set requirement, otherwise determining that the clarity does not meet the set requirement; When the definition meets the set requirement, it is detected whether the page to be identified includes a picture.

9. The method according to claim 1, characterized in that: The method further comprises: In the case that the page to be identified does not include a picture, extracting the text content of the page to be identified, and matching it with the page text of each second page that does not include a picture in the pre-stored paper document, and obtaining the first page number corresponding to the matched page text; Extracting an image feature vector of the page to be identified to obtain a second feature vector of a fixed dimension, matching the second feature vector with a pre-stored feature vector of the second page, and obtaining a second page number of the second page corresponding to the matched feature vector; In the case that the first page number is the same as the second page number, the first page number is determined as the corresponding page number of the to-be-identified page in the paper document.

10. The method according to claim 9, characterized in that The method further comprises: When the first page number is different from the second page number, controlling the display of a second page corresponding to the first page number and the second page number on the interactive interface; A page number selected by a user from a displayed second page through an interactive interface is received, and the selected page number is used as the corresponding page number.

11. A response processing method, characterized in that: The method comprises: When a page image of a target paper document is acquired, using the image recognition method according to any one of claims 1 to 10, determining that the page image corresponds to a target page number in the target paper document; According to the preset mapping relationship between the page of the target paper document and the device operation, the device is triggered to execute the device operation corresponding to the target page number.

12. An image recognition device, characterized in that: The device comprises: A page acquisition module to be identified is used to acquire a page image of a paper document as a page to be identified; The picture detection and recognition module is used to detect whether the page to be recognized includes a picture using a pre-trained detection network, and if the page to be recognized includes a picture, determine a first classification result of the picture; cut the picture from the page to be recognized, and input the picture into a pre-trained recognition network to obtain a second classification result of the picture; wherein the detection network and the recognition network both include multiple classification categories, each of which corresponds to all the pictures of a single page in the paper document; The target page determination module is used to determine the corresponding page number of the page to be identified in the paper document when the second classification result matches the first classification result.

13. A response processing device, characterized in that: The device comprises: A page number determination module, configured to, when a page image of a target paper document is acquired, determine a target page number in the target paper document corresponding to the page image by using the image recognition method according to any one of claims 1 to 10; The trigger execution module is used to trigger the device to execute the device operation corresponding to the target page number according to the preset mapping relationship between the page of the target paper document and the device operation.

14. An electronic device, characterized in that: include: Memory, processor; The memory is used to store computer programs; The processor is used to call the computer program to implement the method according to any one of claims 1-11.

15. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images