Image recognition and response processing method, device, equipment and storage medium

Abstract

The application provides an image recognition and response processing method, device and equipment and a storage medium. By combining a pre-trained detection network and an identification network, for a to-be-recognized page containing a matching image, the matching image is used as an identification clue, and a double classification mechanism, i.e., the combination of preliminary classification of the detection network and fine classification of the identification network, is used to effectively filter misjudgments, so that the input requirements of various document formats and font changes can be met, and a high recognition rate can be provided for the case that the to-be-recognized page has blurred and incomplete characters. A more reliable and efficient page number identification method is provided, and the accuracy and robustness of page number identification are improved.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to an image recognition and response processing method, apparatus, device and storage medium. Background Technology

[0002] In paper-screen interaction scenarios, traditional methods for accurately identifying page numbers on paper documents by electronic devices typically rely on OCR (Optical Character Recognition) technology and image search engines. OCR technology scans and parses the text content on paper document pages, allowing the electronic device to match this text with pre-stored text from various pages of the paper document to achieve retrieval and page number recognition. For pages without text, the image search engine compares this text-free page with multiple page images of paper documents stored in a database, inferring the page number based on the similarity of the image content.

[0003] However, in low light or with poor image quality, OCR technology often struggles to accurately recognize text, leading to discrepancies in page number recognition. Similarly, image search engines are prone to mismatched page numbers when processing low-quality or highly similar images, impacting the user's browsing experience. Summary of the Invention

[0004] In view of this, in order to solve the above-mentioned technical problems, this application provides an image recognition and response processing method, apparatus, device and storage medium.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] According to a first aspect of the embodiments of this application, an image recognition method is provided, the method including:

[0007] Obtain page images of paper documents as the pages to be recognized;

[0008] A pre-trained detection network is used to detect whether the page to be identified includes an image, and if the page to be identified includes an image, a first classification result for the image is determined.

[0009] The image is extracted from the page to be identified and input into a pre-trained recognition network to obtain a second classification result of the image; wherein, both the detection network and the recognition network include multiple classification categories, and each classification category corresponds to all the images on a single page in the paper document;

[0010] If the second classification result matches the first classification result, the corresponding page number of the page to be identified in the paper document is determined.

[0011] Optionally, the target network includes the detection network or the recognition network; each convolutional layer of the target network includes at least one of the custom convolutional operations, namely, vertical convolution and horizontal convolution; the vertical convolution includes horizontally dividing the first convolutional kernel into two equal-sized sub-convolutional kernels and then calculating the difference between the vertical and horizontal convolutional results; the horizontal convolution includes vertically dividing the second convolutional kernel into two equal-sized sub-convolutional kernels and then calculating the difference between the horizontal and horizontal convolutional results.

[0012] Optionally, when the page to be identified includes an image, determining the first classification result of the image includes:

[0013] The detection network is used to determine the bounding box of the image, and the region of interest in the page to be identified is determined based on the bounding box.

[0014] The first image feature of the region of interest is extracted through the custom convolution operation in the detection network, and the first classification result of the image is predicted based on the first image feature.

[0015] Optionally, the step of cropping the image from the page to be identified and inputting the image into a pre-trained recognition network to obtain a second classification result for the image includes:

[0016] The page to be identified is cropped based on the bounding box of the image determined by the detection network to obtain the image and input into the recognition network;

[0017] The second image feature of the image is extracted through the custom convolution operation in the recognition network, and the second classification result of the image is predicted based on the second image feature.

[0018] Optionally, the multiple convolutional kernels used in each convolutional layer of the target network include a first convolutional kernel with a first ratio for implementing the vertical convolution, a second convolutional kernel with a second ratio for implementing the horizontal convolution, and a third convolutional kernel with a third ratio for implementing standard convolution.

[0019] The vertical convolution or the horizontal convolution includes: dividing the corresponding convolution kernel into two equal-sized sub-convolution kernels; calculating the convolution result obtained by weighted summation of each sub-convolution kernel, and obtaining the absolute value of the difference between the two convolution results; and obtaining the corresponding final convolution result based on the absolute value.

[0020] Optionally, determining the corresponding page number of the page to be identified in the paper document when the second classification result matches the first classification result includes:

[0021] Obtain the first predicted category with the highest confidence in the first classification result, and obtain the second predicted category with the highest confidence in the second classification result;

[0022] If the second predicted category meets the set conditions, it is determined that the second classification result matches the first classification result, and the page number of the page in the paper document corresponding to the second predicted category is obtained as the corresponding page number; the set conditions include: the confidence level of the second predicted category is greater than or equal to a preset threshold, and the second predicted category is the same as the first predicted category.

[0023] Optionally, the method further includes:

[0024] If the second classification result does not match the first classification result, image feature vector extraction is performed on the page to be identified to obtain a first feature vector of fixed dimensions.

[0025] The first feature vector is matched with the feature vector of the first page containing the illustration in the pre-stored paper document, and the page number corresponding to the matched feature vector is determined as the corresponding page number.

[0026] Optionally, the target network further includes a pre-trained fuzzy judgment network; the method further includes:

[0027] By using the custom convolution operation in the fuzzy judgment network, the edge features of the page to be identified are extracted and passed to the subsequent layers of the fuzzy judgment network to obtain the fuzzy classification result of the page to be identified.

[0028] If the fuzzy classification result indicates that the page to be identified is not blurry, the clarity of the page to be identified is determined to meet the set requirements; otherwise, the clarity is determined to not meet the set requirements.

[0029] If the clarity meets the set requirements, detect whether the page to be identified includes an image.

[0030] Optionally, the method further includes:

[0031] If the page to be identified does not contain an image, the text content of the page to be identified is extracted and matched with the page text of each second page in the pre-stored paper document that does not contain an image, and the first page number corresponding to the matched page text is obtained.

[0032] The image feature vector of the page to be identified is extracted to obtain a second feature vector of fixed dimension, and then matched with the feature vector of the pre-stored second page to obtain the second page number of the second page corresponding to the matched feature vector;

[0033] If the first page number and the second page number are the same, the first page number is determined as the corresponding page number of the page to be identified in the paper document.

[0034] Optionally, the method further includes:

[0035] If the first page number and the second page number are different, control the display of the second page corresponding to the first page number and the second page number on the interactive interface;

[0036] Receive the page number selected by the user from the second page displayed through the interactive interface, and use the selected page number as the corresponding page number.

[0037] According to a second aspect of the embodiments of this application, an image recognition apparatus is provided, the apparatus comprising:

[0038] The page to be identified module is used to acquire page images of paper documents as pages to be identified;

[0039] The image detection and recognition module is used to detect whether the page to be recognized includes an image using a pre-trained detection network, and if the page to be recognized includes an image, determine a first classification result for the image; crop the image from the page to be recognized, and input the image into a pre-trained recognition network to obtain a second classification result for the image; wherein, both the detection network and the recognition network include multiple classification categories, and each classification category corresponds to all images on a single page in the paper document;

[0040] The target page determination module is used to determine the corresponding page number of the page to be identified in the paper document when the second classification result matches the first classification result.

[0041] Optionally, the target network includes the detection network or the recognition network; each convolutional layer of the target network includes at least one of the custom convolutional operations, namely, vertical convolution and horizontal convolution; the vertical convolution includes horizontally dividing the first convolutional kernel into two equal-sized sub-convolutional kernels and then calculating the difference between the vertical and horizontal convolutional results; the horizontal convolution includes vertically dividing the second convolutional kernel into two equal-sized sub-convolutional kernels and then calculating the difference between the horizontal and horizontal convolutional results.

[0042] Optionally, based on the custom convolution operation, the image detection and recognition module is specifically used for:

[0043] The detection network is used to determine the bounding box of the image, and the region of interest in the page to be identified is determined based on the bounding box.

[0044] The first image feature of the region of interest is extracted through the custom convolution operation in the detection network, and the first classification result of the image is predicted based on the first image feature.

[0045] Optionally, based on the custom convolution operation, the image detection and recognition module is specifically used for:

[0046] The page to be identified is cropped based on the bounding box of the image determined by the detection network to obtain the image and input into the recognition network;

[0047] The second image feature of the image is extracted through the custom convolution operation in the recognition network, and the second classification result of the image is predicted based on the second image feature.

[0048] Optionally, the multiple convolutional kernels used in each convolutional layer of the target network include a first convolutional kernel with a first ratio for implementing the vertical convolution, a second convolutional kernel with a second ratio for implementing the horizontal convolution, and a third convolutional kernel with a third ratio for implementing standard convolution.

[0049] The vertical convolution or the horizontal convolution includes: dividing the corresponding convolution kernel into two equal-sized sub-convolution kernels; calculating the convolution result obtained by weighted summation of each sub-convolution kernel, and obtaining the absolute value of the difference between the two convolution results; and obtaining the corresponding final convolution result based on the absolute value.

[0050] Optionally, the target page determination module is specifically used for:

[0051] Obtain the first predicted category with the highest confidence in the first classification result, and obtain the second predicted category with the highest confidence in the second classification result;

[0052] If the second predicted category meets the set conditions, it is determined that the second classification result matches the first classification result, and the page number of the page in the paper document corresponding to the second predicted category is obtained as the corresponding page number; the set conditions include: the confidence level of the second predicted category is greater than or equal to a preset threshold, and the second predicted category is the same as the first predicted category.

[0053] Optionally, the device further includes:

[0054] If the second classification result does not match the first classification result, image feature vector extraction is performed on the page to be identified to obtain a first feature vector of fixed dimensions.

[0055] The first feature vector is matched with the feature vector of the first page containing the illustration in the pre-stored paper document, and the page number corresponding to the matched feature vector is determined as the corresponding page number.

[0056] Optionally, the target network further includes a pre-trained fuzzy judgment network; the device further includes:

[0057] By using the custom convolution operation in the fuzzy judgment network, the edge features of the page to be identified are extracted and passed to the subsequent layers of the fuzzy judgment network to obtain the fuzzy classification result of the page to be identified.

[0058] If the fuzzy classification result indicates that the page to be identified is not blurry, the clarity of the page to be identified is determined to meet the set requirements; otherwise, the clarity is determined to not meet the set requirements.

[0059] If the clarity meets the set requirements, detect whether the page to be identified includes an image.

[0060] Optionally, the device further includes:

[0061] If the page to be identified does not contain an image, the text content of the page to be identified is extracted and matched with the page text of each second page in the pre-stored paper document that does not contain an image, and the first page number corresponding to the matched page text is obtained.

[0062] The image feature vector of the page to be identified is extracted to obtain a second feature vector of fixed dimension, and then matched with the feature vector of the pre-stored second page to obtain the second page number of the second page corresponding to the matched feature vector;

[0063] If the first page number and the second page number are the same, the first page number is determined as the corresponding page number of the page to be identified in the paper document.

[0064] Optionally, the device further includes:

[0065] If the first page number and the second page number are different, control the display of the second page corresponding to the first page number and the second page number on the interactive interface;

[0066] Receive the page number selected by the user from the second page displayed through the interactive interface, and use the selected page number as the corresponding page number.

[0067] According to a third aspect of the embodiments of this application, a response processing method is provided, the method including:

[0068] When a page image of the target paper document is obtained, the image recognition method described in any of the above embodiments is used to determine the target page number in the target paper document corresponding to the page image;

[0069] Based on the preset mapping relationship between the target paper document's pages and device operations, the device is triggered to execute the device operation corresponding to the target page number.

[0070] According to a fourth aspect of the embodiments of this application, a response processing apparatus is provided, the apparatus comprising:

[0071] The page number determination module is used to determine the target page number in the target paper document by using an image recognition method when a page image of the target paper document is obtained.

[0072] The trigger execution module is used to trigger the device to execute the device operation corresponding to the target page number based on the preset mapping relationship between the pages of the target paper document and the device operation.

[0073] According to a fifth aspect of the embodiments of this application, an electronic device is provided, the electronic device comprising: a memory and a processor; the memory being used to store a computer program; the processor being used to execute the above-described image recognition method or response processing method by invoking the computer program.

[0074] According to a sixth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, wherein the program, when executed by a processor, implements the above-described image recognition method or response processing method.

[0075] The technical solutions provided in this application embodiment may include the following beneficial effects:

[0076] In the technical solution provided in this application, by combining a pre-trained detection network and a recognition network, for pages containing images, not only are the images used as recognition cues, but a dual classification mechanism—combining the preliminary classification of the detection network with the fine classification of the recognition network—effectively filters out false positives, providing a more reliable and efficient page number recognition method, and improving the accuracy and robustness of page number recognition. Compared to pure text recognition or image search engines in related technologies, this method can handle input requirements such as diverse document formats and font variations, and can also provide a high recognition rate for pages with blurred or incomplete text, improving the automation level of document processing and user experience.

[0077] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Furthermore, no embodiment in this application needs to achieve all the effects described above. Attached Figure Description

[0078] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0079] Figure 1 This is a schematic flowchart illustrating an image recognition method according to an exemplary embodiment of this application;

[0080] Figure 2 This is a schematic diagram illustrating an exemplary embodiment of the present application of a top-bottom convolution process;

[0081] Figure 3 This is a schematic diagram illustrating an exemplary embodiment of the present application, showing the process of dividing the convolution kernel of vertical convolution and horizontal convolution into two sub-convolution kernels of equal size;

[0082] Figure 4 This is a flowchart illustrating the steps of matching a first classification result with a second classification result and determining the corresponding page number of the page to be identified, as shown in an exemplary embodiment of this application.

[0083] Figure 5 This is a flowchart illustrating an exemplary embodiment of the present application of a step for detecting whether a page to be identified is blurry using a fuzzy judgment network;

[0084] Figure 6 This is a flowchart illustrating a page number recognition step that combines text search and feature vector search for dual verification when the page to be identified does not include an image, as shown in an exemplary embodiment of this application.

[0085] Figure 7A This is a flowchart illustrating an image recognition method according to an exemplary embodiment of this application;

[0086] Figure 7B This application provides an exemplary embodiment illustrating a 4x4 convolution kernel and an example of the convolutional image data covered by the kernel.

[0087] Figure 7C This is a schematic diagram illustrating the segmentation of a 4*4 convolution kernel and the convolutional image data covered by the kernel during vertical convolution, as shown in an exemplary embodiment of this application.

[0088] Figure 7D This is a schematic diagram illustrating the segmentation of a 4*4 convolution kernel and the convolutional image data covered by the kernel when performing left and right convolutions, according to an exemplary embodiment of this application.

[0089] Figure 8 This is a schematic diagram of the structure of an image recognition device shown in an exemplary embodiment of this application;

[0090] Figure 9This is a schematic diagram of the structure of a response processing device shown in an exemplary embodiment of this application;

[0091] Figure 10 This is a hardware schematic diagram of an electronic device illustrated in an exemplary embodiment of this application. Detailed Implementation

[0092] In paper-screen interaction scenarios, traditional methods rely primarily on OCR technology and image search engines to address the need for electronic devices to accurately identify page numbers in paper documents. OCR technology scans and parses the text information on document pages with high precision, attempting to match the extracted text content with the text of each page in a pre-stored database to achieve rapid page number retrieval and recognition. However, this method is no longer suitable for pages that do not contain obvious text or have very little text.

[0093] At this point, image search engines become another option. By comparing the current page image without text with the page images of paper documents stored in the database, they infer page numbers based on the similarity of the image content. This can, to some extent, make up for the shortcomings of OCR in pages with missing text. However, in cases of poor lighting conditions, damaged image quality, or a large number of pages with similar layouts in the document, image search engines are prone to mismatching page numbers, which in turn affects the user's browsing efficiency and experience.

[0094] Specifically, insufficient lighting significantly reduces image clarity, increasing the difficulty of OCR recognition; degraded image quality may include noise, blur, distortion, and other factors, further affecting the accuracy of OCR and image search engines. Furthermore, when a document contains many pages with similar layouts but different content, image search engines may struggle to distinguish them accurately, causing confusion in page number recognition.

[0095] The aforementioned problems are prevalent in various application scenarios. For example, page number recognition in the boot-up manuals of educational devices or page number recognition in paper documents of other consumer electronics products requires higher precision and robust image page number recognition methods to improve user experience. Therefore, to improve the page number recognition accuracy in paper-screen interaction scenarios, this application provides an image recognition method for page number recognition of paper document images in such scenarios. This method can be executed by any electronic device that needs to recognize the page numbers of the input image, depending on the specific application scenario. This method is not only applicable to the recognition of boot-up manuals for educational devices but can also be widely applied to other scenarios requiring high-precision page number recognition, such as library book management and electronic archive organization, providing a new solution for improving work efficiency and user experience.

[0096] See Figure 1 As shown, this application provides an image recognition method that may include the following steps:

[0097] S101, Obtain page images of the paper document as the pages to be recognized;

[0098] The page image refers to a digital representation of one or more pages of the paper document. It can be stored in image formats such as JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), or TIFF (Tagged Image File Format), preserving the visual content of the original document page. In this embodiment, using this page image as the page to be recognized indicates that the user expects the electronic device to identify the corresponding page number of that page image in the paper document.

[0099] The image of this page can be obtained through scanning, photographing, or other methods. Scanning can be achieved using a scanner, scanning pen, or other device with scanning capabilities, while photographing can be achieved using a camera or other device with photographing capabilities. For example, some smart learning devices are equipped with one or more recesses, platforms, or adjustable stands for stably placing paper documents, and are equipped with at least one high-definition camera to accurately focus on and capture images of the document pages.

[0100] S102, a pre-trained detection network is used to detect whether the page to be identified includes an image, and if the page to be identified includes an image, a first classification result of the image is determined; the detection network includes multiple classification categories, and each classification category corresponds to all the images on a single page in the paper document;

[0101] The accompanying image refers to an image or graphic element on the page to be identified, which may include, but is not limited to, photographs, illustrations, charts, diagrams, comics, or hand-drawn sketches. Upon obtaining the page to be identified, the process first checks whether the page includes an accompanying image. This process can be implemented using image processing techniques or machine learning algorithms. For example, edge detection algorithms can be used to distinguish different areas of the page to be identified, and key features such as color and texture can be combined to determine whether an area contains an accompanying image. Alternatively, a machine learning model trained on a large amount of labeled image data, such as the detection network mentioned above, can learn the difference between accompanying images and text, and accurately detect whether an accompanying image is present on a new page to be identified. In this embodiment, the cases where the page to be identified includes an accompanying image may include: the page to be identified contains only an accompanying image, or the page to be identified contains both text and an accompanying image.

[0102] In this embodiment, a pre-trained detection network is used to implement the image detection and preliminary classification functions. The detection network is pre-trained to detect whether the page to be identified includes an image and to perform a preliminary classification operation on the detected image. This can be achieved by extracting local region features of the page containing the image, including but not limited to visual features such as edges, textures, and color distribution, as well as basic attribute features such as shape, size, and proportion. Based on these local region features, the image is preliminarily classified to obtain the first classification result. The first classification result can at least include the probability distribution of the image belonging to each set category.

[0103] Based on the classification results of the detection network in this embodiment, which are used to determine the corresponding page number of the page to be identified in the paper document, each classification category of the detection network is pre-set to correspond to all the illustrations on a single page in the paper document. That is, for each document page with illustrations in the paper document, all the illustrations included on that document page correspond to a classification category of the detection network, so as to facilitate the identification and differentiation of illustrations from different document pages of the paper document using the detection network.

[0104] This detection network can be designed and implemented based on convolutional neural networks or deep learning architectures, such as YOLOv5 networks, R-CNN (Regions with CNN features) series networks, or Transformer-based networks. It can detect or extract image features from the page to be identified through a series of convolutional layers, pooling layers, activation functions, and other structures. The output layer of this detection network is designed with multiple neurons, each corresponding to a specific classification category. Each classification category directly maps to all images included on a page with images in the paper document, allowing the network's classification result to locate the corresponding source page of the paper document containing the images on the page to be identified. For example, the page numbers of each document page with images in the paper document can be directly used as the classification category. Therefore, when the network outputs a classification category of 11, it indicates that the image on the page to be identified belongs to the eleventh page of the paper document.

[0105] To achieve the image detection and preliminary classification functions of the aforementioned detection network, a large amount of labeled data containing image location and category labels can be used to train the network during the pre-training phase. The bounding box detection and initial classification errors are minimized by optimizing the network's internal parameters. The loss function of this detection network can include two parts: image detection and localization error and image classification error. The importance of these two errors is balanced by adjusting the weights of the loss function. Based on the above structural design of the output layer of the detection network, the pre-training of the detection network can be achieved in the following ways: (1) Dataset preparation: For each page of the paper document with illustrations, a large number of page image samples are collected. These samples can include multiple samples of the same document page under different shooting conditions and different clarity. The illustration position and classification category are labeled for each page image sample. The classification category refers to the classification category set in the detection network. (2) Image preprocessing: The images are preprocessed as necessary, such as scaling, cropping, normalization, etc., to ensure that they have consistent size and format. Image enhancement (such as rotation, flipping, adding noise, etc.) can also be performed to increase the diversity of the dataset and the generalization ability of the model. (3) Model training: The detection network is trained using the preprocessed dataset. During the training process, the output of the network is calculated through forward propagation, and the network parameters are adjusted using the backpropagation algorithm to minimize the loss function.

[0106] After the detection network is trained, it can be deployed in real-world application scenarios. This allows the network to process page images of received paper documents, automatically identifying and classifying which page of the paper document the accompanying image belongs to.

[0107] S103, extract the image from the page to be identified, and input the image into a pre-trained recognition network to obtain a second classification result of the image; wherein, the recognition network includes multiple classification categories, and each classification category corresponds to all images on a single page in the paper document;

[0108] The recognition network is used to accurately classify the images in the page to be recognized. The network takes the images themselves as input and can automatically extract more complex and detailed image feature representations from the images using structures such as multi-layer convolutional neural networks. These features may include, but are not limited to, local details such as texture patterns, color gradients, shape features such as contours and symmetry, and higher-level semantic features such as object components, attributes, and contextual relationships. Based on these feature representations, the network accurately classifies the images to obtain a second classification result. This second classification result can at least include the probability distribution of the image belonging to each category in the recognition network. Furthermore, the detection network can accurately locate the bounding box of the image in the page to be recognized using features such as edge strength and direction changes. The input to the recognition network can be obtained by extracting the image from the page to be recognized using this bounding box.

[0109] In this embodiment, each classification category of the recognition network corresponds to all the images on a single page of the paper document. That is, all the images on each page of the paper document that contains images correspond to a classification category of the detection network, facilitating accurate identification of images from different pages of the paper document using the recognition network. This detection network can be designed and implemented based on convolutional neural networks or deep learning architectures, such as the ShuffleNetv2 classification network or the MobileNet series of networks. During the pre-training phase, the recognition network can be trained using a large amount of labeled data containing only the images themselves and category labels, and the classification error can be minimized by optimizing the internal parameters.

[0110] This recognition network can detect or extract feature representations of images through a series of convolutional layers, pooling layers, activation functions, and other structures. Its output layer can be designed with multiple neurons, each corresponding to a specific classification category. Each classification category directly maps to all images included on a page with an image in the paper document, allowing the network's classification result to locate the corresponding source page of the paper document containing the image on the page to be recognized. For example, the page numbers of each document page with images in the paper document can be directly used as the classification category. Therefore, when the network outputs a classification category of 11, it indicates that the input image belongs to the eleventh page of the paper document.

[0111] The detection network and the recognition network mentioned above can also use the same basic network structure, and the network layers of the basic network structure can be modified adaptively according to their respective task requirements. For example, if the basic network structure is a classification network, then a network structure to realize the image detection function should be added on the basis of the classification network structure for the detection network.

[0112] S104, if the second classification result matches the first classification result, determine the corresponding page number of the page to be identified in the paper document.

[0113] The matching of the second classification result with the first classification result means that there is a target classification category in both classification results that meets their respective preset reliability requirements, and the target classification category in the second classification result is the same as the page in the paper document corresponding to at least one target classification category in the first classification result.

[0114] If the second classification result matches the first classification result, it can be determined that the second classification result for the image on the page to be identified is accurate or acceptable. Since the classification categories of the detection network and the recognition network correspond one-to-one with the pages with images in the paper document, the corresponding page in the paper document can be determined based on the target classification category in the second classification result that meets the preset reliability requirements, and thus the corresponding page number of the page to be identified in the paper document can be determined.

[0115] In this embodiment, by combining a pre-trained detection network and a recognition network, dual classification verification is performed on the illustrations in paper documents interacting with electronic devices, improving the accuracy and robustness of page number recognition. For pages containing illustrations, not only are the illustrations used as recognition cues, but a dual classification mechanism—combining the initial classification of the detection network with the fine classification of the recognition network—effectively filters out false positives and ensures the accuracy of the recognition results. Furthermore, due to the uniqueness of the illustrations and their relatively fixed positions in the document, they are better able to handle challenges such as diverse document formats and font variations compared to pure text recognition in related technologies. Especially when there is blurred or incomplete text in the document, illustration recognition can maintain a high recognition rate, providing a more reliable and efficient page number recognition method, greatly improving the automation level of document processing and user experience.

[0116] In some embodiments, based on the limitations of standard convolution operations, in order to capture feature information in images more comprehensively and enhance the effectiveness and richness of feature extraction to better adapt to complex and varied pages to be identified, this embodiment proposes an innovative convolution processing method. For the detection network and recognition network (collectively referred to as the target network) described in the foregoing embodiments, each convolutional layer of the target network introduces at least one of the custom convolution operations, either vertical convolution or horizontal convolution, thereby broadening the field of view of the convolution kernel and enabling the network to capture more multidimensional and detailed feature information.

[0117] The upper and lower convolutions include horizontally dividing the first convolution kernel into two equal-sized sub-convolution kernels and then calculating the difference between the upper and lower convolution results; the left and right convolutions include vertically dividing the second convolution kernel into two equal-sized sub-convolution kernels and then calculating the difference between the left and right convolution results.

[0118] Each convolutional layer of the target network can include multiple convolutional kernels. The element weights and biases of each convolutional kernel are learned during pre-training so that the network can automatically learn feature representations from the network input data. Based on the predefined vertical and horizontal convolution operations introduced in this embodiment, the multiple convolutional kernels can include at least one of a first convolutional kernel for implementing vertical convolution and a second convolutional kernel for implementing horizontal convolution. It can also include a standard convolutional kernel, which extracts features by multiplying each element in the kernel with the element at the corresponding position in the image to be convolved and summing all products. That is, each convolutional layer of the target network can be configured to include only vertical or horizontal convolution, or it can retain the standard convolutional operation while introducing at least one of the custom vertical or horizontal convolution operations.

[0119] When the first convolutional kernel is used to extract features, its upper and lower convolution processing method includes: dividing the first convolutional kernel horizontally into two smaller sub-convolutional kernels, corresponding to the upper and lower parts of the horizontal dividing line respectively; within the coverage area of ​​the image to be convolved by the first convolutional kernel, for each sub-convolutional kernel, multiplying each element in the sub-convolutional kernel with the corresponding element in the image within the coverage area, and summing all products to obtain the convolution result of the sub-convolutional kernel; obtaining the difference between the convolution results of the upper and lower sub-convolutional kernels, which can be further combined with the bias term to obtain the final upper and lower convolution result. This upper and lower convolution method can extract the vertical difference features of two corresponding image patches within the coverage area of ​​the two sub-convolutional kernels, thereby increasing the diversity and information content of features and enhancing the network's ability to understand image content.

[0120] For example, such as Figure 2 The 4x3 first convolution kernel shown has weights and biases learned during pre-training, covering a 4x3 pixel region X in the image to be convolved. During the upper and lower convolution operations, this first convolution kernel can be divided into a 2x3 upper convolution kernel A1 and a lower convolution kernel A2 along the horizontal dividing line L1 shown in the diagram. The upper convolution kernel A1 corresponds to pixel region X1, and the lower convolution kernel A2 corresponds to pixel region X2. When calculating the convolution result of the sub-convolution kernels, the weights of A1 are multiplied by the corresponding pixels in X1, and the products are summed to obtain the convolution result M1 of A1. Similarly, the weights of A2 are multiplied by the corresponding pixels in X2, and the products are summed to obtain the convolution result M2 of A2. The difference between convolution results M1 and M2 is then calculated. This difference, combined with the bias term of the first convolution kernel, yields the final convolution result M of the first convolution kernel performing upper and lower convolutions on pixel region X.

[0121] Similarly, when the second convolutional kernel is used to extract features, the processing method for implementing left and right convolution includes: dividing the second convolutional kernel vertically into two smaller sub-convolutional kernels, corresponding to the left and right parts of the vertical dividing line, respectively. The principle for obtaining the convolution result of each sub-convolutional kernel is the same as that of the first convolutional kernel. The difference between the convolution results of the left and right sub-convolutional kernels is obtained, and the final left and right convolution result can be obtained by combining the bias term. The calculation method of the sub-convolutional kernels in the left and right convolution is the same as that of the sub-convolutional kernels in the up and down convolution, and will not be repeated here.

[0122] After introducing custom convolution operations into the convolutional layers of the target network, during the training of the target network using the preprocessed dataset, the network output is computed through forward propagation, and the parameters of each layer of the network are adjusted using the backpropagation algorithm, including adjusting the weights and bias terms of multiple convolutional kernels in each convolutional layer to minimize the loss function, and the weight parameters on each convolutional kernel used to implement the custom convolution operations are learned.

[0123] In this embodiment, local regional difference features of two corresponding image blocks can be extracted based on top-bottom / left-right convolution. By introducing custom top-bottom and left-right convolutions into each convolutional layer of the detection or recognition network, the perceptual field of the convolutional kernel is broadened, enhancing the network's ability to capture complex and subtle features in the image. This enables the extraction of more effective features even for images with poor lighting conditions or low image quality, promoting the model's fusion processing of global and local information in the image, thereby improving the accuracy and robustness of the network in detection and recognition tasks.

[0124] In some embodiments, based on the custom convolution operation introduced for the detection network in the above embodiments, the determination of the first classification result of the image in the case that the page to be identified includes an image, as described in step S102 above, can be achieved in the following way:

[0125] The detection network is used to determine the bounding box of the image, and the region of interest in the page to be identified is determined based on the bounding box; the custom convolution operation in the detection network is used to extract the first image feature of the region of interest, and the first classification result of the image is predicted based on the first image feature.

[0126] The bounding box of the accompanying image describes the position and extent of the image on the page to be identified. The position of the top-left and bottom-right corners of the bounding box can be determined by four parameters: minimum X-coordinate, minimum Y-coordinate, maximum X-coordinate, and maximum Y-coordinate. When the detection network determines that the page to be identified includes an image using its image detection capabilities learned during the pre-training phase, the bounding box of the image on the page to be identified can be determined simultaneously.

[0127] The region of interest (ROI) in the page to be identified refers to the local area defined by the bounding box of the image determined by the detection network, which directly corresponds to the local area defined by the bounding box in the page to be identified. To flexibly adapt to the processing needs of different scenarios, the ROI can also be set as a local area focused on the periphery of the image, obtained by expanding outwards from the bounding box according to a preset size boundary.

[0128] The first image feature is a feature representation of the region of interest obtained through the convolutional layers of the detection network. It describes the content, shape, texture, and other attributes of the image and serves as the data basis for the initial classification of the detection network. After detecting an image on the page to be identified, the detection network can extract the feature representation of the region of interest using the convolutional kernels corresponding to the top-bottom or left-right convolutions set in the convolutional layers, and use this as the first image feature. Alternatively, if the detection network needs to extract features of the page to be identified using convolutional layers for image detection, after detecting the image, the features of the image can be determined from the extracted features of the page to be identified based on the image bounding box position, and these features can be used as the first image feature.

[0129] After obtaining the first image feature, the detection network can predict the probability distribution of the category to which the image belongs by learning the classification parameters, thereby generating the first classification result.

[0130] In this embodiment of the disclosure, the bounding box of the image is accurately located by the detection network, thereby accurately delineating the region of interest in the page to be identified, achieving precise focusing on the target content. Furthermore, by utilizing the custom convolution operation in the detection network, rich and effective first image features are efficiently captured from the region of interest, and then the image classification is predicted based on the first image features, improving the accuracy and efficiency of the identification.

[0131] In some embodiments, based on the custom convolution operation introduced for the recognition network in the foregoing embodiments, the step S103 described above, which involves cropping the image from the page to be recognized and inputting the image into a pre-trained recognition network to obtain the second classification result of the image, can be implemented in the following way:

[0132] The page to be identified is cropped based on the bounding box of the image determined by the detection network to obtain the image, which is then input into the recognition network. The second image feature of the image is extracted through the custom convolution operation in the recognition network, and the second classification result of the image is predicted based on the second image feature.

[0133] After obtaining the bounding box of the image on the page to be recognized, as determined by the detection network, the image region of the image can be extracted from the page using the positional information of the bounding box. The cropped image is then used as input to the recognition network. To ensure the quality of the image, the cropping operation can remove irrelevant background information as much as possible while ensuring the integrity of the image content.

[0134] Upon receiving the accompanying image, the recognition network can extract feature representations of the input image using convolutional kernels corresponding to top-bottom or left-right convolutions and / or standard convolution operations within the convolutional layers. These feature representations serve as the second image feature. Based on the extracted second image feature, a classifier within the recognition network can accurately classify and predict the image. This classifier outputs a second classification result for the image, representing the probability distribution of the image originating from each page of the paper document containing the accompanying image, according to different combinations and weight allocations of the features.

[0135] In this embodiment, by precisely cropping the image in the page to be identified, redundant background information is removed. The cropped image is then directly input into the recognition network, and a custom convolution operation in the recognition network is used for feature extraction. Compared with using only standard convolution kernels for feature extraction, this method can deeply explore the unique features of the image and enhance the richness and effectiveness of the features, providing richer and more accurate information for classification prediction, reducing unnecessary computation, and improving the processing efficiency and accuracy of the recognition network.

[0136] In some embodiments, for the target network, i.e., the detection network or the recognition network, the multiple convolutional kernels used in each convolutional layer of the target network may include a first convolutional kernel with a first proportion for implementing the vertical convolution, a second convolutional kernel with a second proportion for implementing the horizontal convolution, and a third convolutional kernel with a third proportion for implementing the standard convolution; the sum of the three proportions is 1.

[0137] That is, each convolutional layer of the target network includes three convolution operations: vertical convolution, horizontal convolution, and standard convolution, with the number of convolutional kernels allocated to each according to a set ratio. The standard convolution includes: for the image region covered by the third convolutional kernel, multiplying each weight of the third convolutional kernel by the pixel value at the corresponding position in the covered image region, and summing all the products to obtain the convolution result of the third convolutional kernel on the covered image region; the vertical or horizontal convolution performs feature extraction according to the convolution process described in the aforementioned embodiment.

[0138] For example, each convolutional layer of the detection network has 10 convolutional kernels. Assuming that when defining the convolutional kernels of this convolutional layer, the proportions of the first convolutional kernel, the second convolutional kernel, and the third convolutional kernel are 1 / 4, 1 / 4, and 1 / 2 respectively, then according to this proportion, 2 of the 10 convolutional kernels in each convolutional layer can be set to implement the vertical convolution, 2 convolutional kernels can be used to implement the horizontal convolution, and the remaining 6 convolutional kernels can be used as the third convolutional kernel to implement the standard convolution.

[0139] The vertical convolution or the horizontal convolution may further include the following processing steps: dividing the corresponding convolution kernel into two equal-sized sub-convolution kernels; calculating the convolution result obtained by weighted summation of each sub-convolution kernel, and obtaining the absolute value of the difference between the two convolution results; and obtaining the corresponding final convolution result based on the absolute value.

[0140] The aforementioned custom convolution operation involves dividing a standard convolution kernel into two equally sized sub-convolution kernels in the vertical or horizontal direction. For the image region covered by the standard convolution kernel, the two sub-convolution kernels are used to perform convolution operations on the corresponding positions of the image region. Finally, the difference between the convolution results of the two sub-convolution kernels is calculated as the convolution result of the custom convolution operation. This convolution result reflects the local differences of the image region in the vertical or horizontal direction.

[0141] In the process of dividing a convolutional kernel into equally sized sub-kernels, to divide a standard convolutional kernel into two equally sized sub-kernels vertically or horizontally, the height of the first convolutional kernel corresponding to the vertical convolution is set to an even number, and the width of the second convolutional kernel corresponding to the horizontal convolution is set to an even number. Since all convolutional kernels in the same convolutional layer are typically the same size, the height and width of the convolutional kernels in the detection or recognition network can be set based on the custom convolutional operations included in the convolutional layer. For example, if the convolutional layer includes vertical convolutions, the height of the convolutional kernel can be set to an even number, with no width restriction; if the convolutional layer includes horizontal convolutions, the width of the convolutional kernel can be set to an even number, with no height restriction; if the convolutional layer includes both vertical and horizontal convolutions, both the width and height of the convolutional kernel can be set to even numbers.

[0142] If the height of the first convolution kernel corresponding to the top and bottom convolutions or the width of the second convolution kernel corresponding to the left and right convolutions are odd, then the weights of the bottom or top row of the first convolution kernel can be reused. By adding adjacent rows with the same weights to the reused rows, the height of the first convolution kernel becomes even. The weights of the rightmost or leftmost column of the second convolution kernel can be reused, and adjacent columns with the same weights can be added to the reused columns to make the width of the second convolution kernel even.

[0143] For example, such as Figure 3As shown, the first convolutional kernel K1 is 3*3. By reusing the weight elements X31, X32, and X33 in the bottom row of K1 and adding adjacent rows with weight elements X31, X32, and X33 to the bottom row, K1 is updated to 4*3, thus dividing K1 into two 2*3 sub-convolutional kernels. Similarly, the second convolutional kernel K2 is 3*3. By reusing the weight elements X13, X23, and X33 in the rightmost row of K2 and adding adjacent columns with weight elements X13, X23, and X33 to the rightmost row, K1 is updated to 3*4, thus dividing K2 into two 3*2 sub-convolutional kernels.

[0144] Based on the above description of the implementation steps of upper and lower convolution kernels and left and right convolution, the convolution process can be represented by the following formula: Y = F(abs(W1*X1-W2*X2)+b). Where W1 and W2 represent two equally sized sub-convolution kernels obtained by dividing the convolution kernel W along the horizontal or vertical direction, (W1+W2) represents the convolution kernel W, X1 and X2 represent the image patches corresponding to the sub-convolution kernels, (X1+X2) constitutes the image region covered by the convolution kernel W, b represents the bias term of the convolution kernel W, abs() represents calculating the absolute value of the difference, and F() represents the activation function to map the neuron's input to its output. This mapping is usually non-linear to facilitate the network learning complex patterns and data representations.

[0145] In this embodiment of the disclosure, custom convolution operations can capture local changes or edge information of images or feature maps in the vertical or horizontal directions, enriching the feature content. Compared with extracting features only through standard convolution, different types of convolution operations achieve multi-dimensional feature capture, making the network more robust to changes such as rotation and tilt in the image, providing more effective feature representations, which helps the detection network and recognition network to better understand the image content, thereby improving the accuracy of image classification.

[0146] In some embodiments, regarding the aforementioned step S104, where the second classification result matches the first classification result, determining the corresponding page number of the page to be identified in the paper document can be achieved by first determining the classification categories in the first and second classification results that meet their respective set confidence requirements, and then performing a matching judgment and determining the corresponding page number. See [link to documentation]. Figure 4 The flowchart shown can be implemented by including the following steps:

[0147] S401, Obtain the first predicted category with the highest confidence level in the first classification result;

[0148] The first classification result is generated by a detection network used for image detection and preliminary image classification. This preliminary classification is coarse, therefore the first classification result may include one or more first predicted categories with the highest confidence. For example, if the first classification result includes two second predicted categories with a confidence level of 40%, then the first classification result includes two first predicted categories with the highest confidence.

[0149] S402, Obtain the second predicted category with the highest confidence level in the second classification result;

[0150] S403, determine whether the second prediction category meets the set conditions, the set conditions include: the confidence level of the second prediction category is greater than or equal to a preset threshold, and the second prediction category is the same as the first prediction category;

[0151] In other words, the second predicted category must not only satisfy the condition that its corresponding confidence level is greater than or equal to a preset threshold, but also simultaneously satisfy the condition that the second predicted category is the same as the first predicted category with the highest confidence level included in the first classification result. Here, each classification category based on the detection network and the recognition network corresponds to all the images on a single page in the paper document. Therefore, the second predicted category being the same as the first predicted category means that the pages in the paper document corresponding to the two predicted categories are the same page. If the first classification category includes multiple first predicted categories with the highest confidence levels, then if the second predicted category is the same as any one of the first predicted categories, it can be determined that the second predicted category is the same as the first predicted category.

[0152] S404a, if the set conditions are met, then it is determined that the second classification result matches the first classification result, and the page number of the page in the paper document corresponding to the second predicted category is obtained as the corresponding page number;

[0153] S404b, if the set conditions are not met, it is determined that the second classification result does not match the first classification result, and a preset mismatch processing procedure is executed.

[0154] If the second prediction category meets the set conditions, it means that the prediction results of the detection network and the recognition network point to the same prediction category and the prediction category has a high confidence level. It can be used as the image prediction result for the page to be identified. Then, based on the page number information of each page in the pre-stored paper document, the page number of the page in the paper document corresponding to the second prediction category can be obtained. Since the image on the page to be identified is predicted to come from the page in the paper document corresponding to the second prediction category, it can be determined that the page to be identified and the page in the paper document corresponding to the second prediction category are the same page. Therefore, the page number of the page in the paper document corresponding to the second prediction category can be used as the corresponding page number of the page to be identified in the paper document.

[0155] If the second predicted category does not meet the set conditions, a preset mismatch processing procedure can be executed, such as further matching processing by extracting image features or text content of the page to be identified, as detailed in subsequent embodiments.

[0156] In this embodiment of the disclosure, by setting conditions requiring the confidence level of the second predicted category to be greater than or equal to a preset threshold, and requiring the second predicted category to be the same as the first predicted category, it is ensured that high-confidence prediction results in the recognition network used for accurate image classification are further considered. Furthermore, by ensuring consistency between the prediction results of different networks, the possibility of correct prediction results is enhanced, thereby reducing misjudgments and uncertainties and improving the overall accuracy of prediction.

[0157] In some embodiments, when the second classification result does not match the first classification result, i.e., when the second predicted category does not meet the set conditions as described in the foregoing embodiments, the preset mismatch processing procedure may include at least one of image feature extraction or text content extraction for further matching processing of the page to be identified. The further matching processing of image feature extraction of the page to be identified can be implemented through the following steps:

[0158] Image feature vectors are extracted from the page to be identified to obtain a first feature vector of fixed dimensions; the first feature vector is matched with the feature vectors of the first page containing the illustration in the pre-stored paper document, and the page number corresponding to the matched feature vector is determined as the corresponding page number.

[0159] Image feature vector extraction refers to a method of converting the image information of the page to be identified into a numerical form to represent the key features of the image. The extracted first feature vector may include, but is not limited to, the color, texture, and shape of the page to be identified. This feature vector extraction process can be implemented using preset feature extraction algorithms such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF).

[0160] The first feature vector can also be determined based on the feature representation extracted by the convolutional layer of the detection network for the input page to be identified when performing image detection. That is, a fixed-dimensional feature vector can be obtained by further processing the feature map output by one or more convolutional layers in the detection network (such as by global average pooling or global max pooling), which can then be used as the first feature vector.

[0161] When executing this embodiment, for each first page including illustrations in the paper document, a fixed-dimensional feature vector of the first page is obtained through the same image feature vector extraction method. The feature vector of each first page and the page number information of the first page can be pre-stored in a preset database such as the Elasticsearch library for easy searching and similarity calculation of feature vectors.

[0162] After extracting the first feature vector of the page to be identified during the application process, the first feature vector of the page to be identified is compared with the feature vectors of each pre-stored first page to find the most similar or best-matching feature vector pair. This can be done by calculating the distance between feature vectors, such as Euclidean distance or cosine similarity, and judging the degree of matching based on the distance. A matching threshold can also be set as a boundary; feature vectors of first pages whose distance is greater than the matching threshold are determined as the matched feature vectors. Based on the pre-stored feature vectors of each first page and their page number information, the page number of the first page containing the matched feature vector can be obtained and used as the corresponding page number of the page to be identified in the paper document.

[0163] In this embodiment, when the second classification result does not match the first classification result, the similarity between the page to be identified and the pre-stored page can be accurately quantified by calculating the distance between feature vectors. This allows for the accurate identification of the page most similar to the page to be identified, thereby determining the accurate page number of the page to be identified in the paper document. Furthermore, the feature vector matching method is unaffected by factors such as page layout, font size, and color changes. It accurately identifies pages based on the principle of page content similarity, making it applicable to various paper document types and enhancing the flexibility of image recognition.

[0164] The method for further matching the text content extraction described in the above embodiments may include the following steps: using OCR technology or other text extraction techniques to identify and extract the text content of the page to be identified, and matching this text content with the page text of each first page containing illustrations in the pre-stored paper document. The page number corresponding to the matched page text (i.e., text content matching degree greater than a set similarity threshold, such as 96%) is determined as the corresponding page number of the page to be identified in the paper document. This text content extraction and matching process can be performed after further matching processing for image feature extraction of the page to be identified fails, as a supplementary solution to the image feature vector.

[0165] In some embodiments, to improve the efficiency and accuracy of text extraction or image processing, this embodiment adds a blurred image judgment step before the aforementioned step S102. This step is used to pre-identify blurred images caused by unclear shooting, inaccurate focus, or damage during transmission, thereby avoiding unnecessary processing of such blurred images, saving computing resources, and improving the accuracy of subsequent image recognition. The blurred image judgment may include the following steps:

[0166] The page features of the page to be identified are obtained using the custom convolution operation introduced in the aforementioned embodiments, and the clarity of the page to be identified is detected based on the page features to determine whether the clarity of the page to be identified meets the set requirements; if the clarity meets the set requirements, it is detected whether the page to be identified includes an image.

[0167] That is, at least one of the convolution processing methods, namely vertical convolution and horizontal convolution, in the custom convolution operation described in the foregoing embodiments can be used to extract page features from the page to be identified in step S101. The key information of the page can be stored in the form of a feature map. Then, based on the pre-built clarity evaluation algorithm, the extracted page can be used to predict the clarity. The set requirements can be quantified and set as a clarity threshold according to the needs of the application scenario. When the clarity score of the page to be identified is higher than or equal to the clarity threshold, it is determined that the clarity of the page to be identified meets the set requirements.

[0168] In some embodiments, the blurred image determination step described in the above embodiments can be implemented using a pre-trained blurred image determination network. See also Figure 5 The flowchart shown illustrates the steps involved in determining the blurred image. The convolution operation is used to obtain the page features of the page to be identified, and the clarity of the page to be identified is detected based on these features to determine if it meets the set requirements. This process may include the following steps:

[0169] S501, The page to be identified is input into a pre-trained fuzzy judgment network;

[0170] The fuzzy image determination network is used to identify whether an image input to the network is sharp. It can achieve fuzzy image determination through an input layer, hidden layers (such as multiple convolutional layers, pooling layers, and activation layers), and an output layer. The input layer receives the input image, the hidden layers extract and transform features from the input image, and the output layer outputs the determination result based on the extracted features. The output layer of this network can be designed to output two categories: sharp and blurry. This can be achieved using a softmax function or other multi-classification function, converting the output of the hidden layer into a probability distribution of the two categories. The final determination result is usually the category with the higher probability as the output; that is, if the probability of the sharp category is higher than that of the blurry category, the image is determined to be sharp; otherwise, it is determined to be blurry.

[0171] In the pre-training process of this fuzzy image judgment network, the training set contains both sharp and blurry images. Each image in the training set is labeled to determine whether it is sharp or blurry. This training set is then used to train the fuzzy image judgment network, allowing it to continuously learn how to determine image sharpness based on its features. This pre-training can employ supervised learning, where the network receives an input image and its corresponding label (sharp or blurry) in each iteration. Based on the error between the network's output and the true label, the network parameters are adjusted using algorithms such as gradient descent or Adam to minimize a loss function, such as cross-entropy loss.

[0172] S502, by using the custom convolution operation in the fuzzy judgment network, the edge features of the page to be identified are extracted and passed to the subsequent layers of the fuzzy judgment network to obtain the fuzzy classification result of the page to be identified;

[0173] In this embodiment, since clear page edges are sharp while blurry page edges are relatively blurry, the clarity of the edges can be used as one of the indicators for judging whether an image is blurry. Therefore, after introducing a custom convolution operation into the convolutional layer of the blur judgment network, the edge features in the page to be identified can be better captured by at least one of the convolution operations of vertical convolution or horizontal convolution.

[0174] The extracted edge features are used as input to the intermediate layers of the fuzzy classification network and passed to subsequent network layers, such as more convolutional layers, pooling layers, and activation layers. More convolutional operations extract higher-level features, while pooling operations reduce the dimensionality of features and retain important information. Simultaneously, the application of activation functions increases the network's non-linearity and improves feature representation. After a series of network layer processing steps, the final features are passed to the output layer of the fuzzy classification network. The output layer makes a classification decision based on the received complex features, outputting the probability distributions of two categories (clear and fuzzy) through mechanisms such as the softmax function, and taking the category with the higher probability as the final fuzzy classification result. When determining the final fuzzy classification result, if the higher probability of the clear or fuzzy category is obtained, it can be further determined whether this higher probability is greater than or equal to a set threshold. If it is greater than or equal to the set threshold, the fuzzy classification result is considered valid.

[0175] To more accurately determine the degree of blur in an image, the blur determination network captures and integrates various types of image features, including but not limited to edge features. This is achieved by using a multi-layer convolutional neural network, where each convolutional layer extracts different types of features and passes them to subsequent layers for further processing and abstraction.

[0176] S503a, if the fuzzy classification result indicates that the page to be identified is not blurry, determine that the clarity of the page to be identified meets the set requirements, and detect whether the page to be identified includes an image;

[0177] S503b, if the fuzzy classification result indicates that the page to be identified is blurry, determine that the clarity does not meet the set requirements, and execute the preset interactive prompt processing flow.

[0178] If the clarity of the page to be identified does not meet the set requirements, the subsequent processing flow for that page (i.e., the process of checking whether it includes an image) will be terminated, and a preset interactive prompt processing flow will be executed. This preset interactive prompt processing flow is used to display an error message to the user, informing them that the collected page to be identified is blurry and needs to be re-uploaded as a clearer version. Alternatively, it suggests trying to enhance the page to be identified, such as sharpening or denoising, to reassess its clarity.

[0179] For example, if a page to be identified is judged to be blurry during the fuzzy classification process, the system will not continue to check whether the page contains an image. Instead, a warning message will be displayed to the user, such as "Page is blurry, please upload a clearer version." A button will also be provided to allow the user to re-upload the page. If the page is re-uploaded, the clarity assessment and subsequent processing will be performed again upon receiving the updated page.

[0180] In this embodiment of the disclosure, by introducing a custom convolution operation into the fuzzy judgment network, the network's ability to capture edge features of the input image is enhanced. Compared with traditional convolution kernels, this convolution operation can more effectively extract subtle differences between blurred and clear pages, thereby improving the accuracy and robustness of fuzzy classification.

[0181] In some embodiments, when the page to be identified does not include an image, i.e., the page to be identified is a plain text page, the text content of the page to be identified can be identified and extracted using OCR technology or other text recognition technology. Then, the extracted text content is matched with the page text of each second page in the pre-stored paper document that does not contain an image. The page text most similar to the extracted text content is found, and its corresponding page number is used as the corresponding page number of the page to be identified in the paper document. However, text extraction is easily affected by various factors, such as page quality, font style, and layout complexity, which can affect the accuracy of text extraction and lead to subsequent matching failures or errors. Therefore, this embodiment provides a processing method that combines text content recognition and page feature vector recognition. See [link to documentation]. Figure 6 The flowchart shown illustrates the steps involved in this process, which may include the following steps:

[0182] S601, extract the text content of the page to be identified, and match it with the page text of each second page in the pre-stored paper document that does not contain illustrations, and obtain the first page number corresponding to the matched page text;

[0183] S602, extract image feature vectors from the page to be identified to obtain a second feature vector of fixed dimensions, and match it with the feature vector of the pre-stored second page to obtain the second page number of the second page corresponding to the matched feature vector;

[0184] That is, the page text content, page number information, and fixed-dimensional feature vectors of each second page (excluding illustrations) in the paper document are pre-stored. These can be stored in a pre-set database such as Elasticsearch for easy text matching and feature vector similarity calculation.

[0185] When the page to be identified does not include an image, the text content on the page can be extracted using text recognition technology. Simultaneously, a fixed-dimensional second feature vector of the page to be identified can be extracted using the image feature vector. The text content and the second feature vector are then matched with page text and the feature vector of the second page stored in a preset database to obtain the first page number of the page text that best matches the text content, and the second page number of the feature vector of the second page that is most similar to the second feature vector. Steps S601 and S602 can be performed in parallel to improve processing efficiency.

[0186] S603, determine whether the first page number and the second page number are the same;

[0187] S604a, if they are the same, then the first page number is determined as the corresponding page number of the page to be identified in the paper document;

[0188] In this embodiment, the possible page numbers of the same page to be identified in the paper document are determined using two different methods: text matching and feature vector matching. If the results obtained by the two methods are consistent, the page number recognition result is deemed valid, thus accurately determining the corresponding page number of the page to be identified. The first page number being the same as the second page number indicates that the page number determined by both text matching and feature vector matching points to the same page in the paper document, increasing the accuracy and reliability of the recognition result. Therefore, the first page number can be determined as the corresponding page number of the page to be identified in the paper document.

[0189] If S604b is different, then the preset interactive processing flow will be executed.

[0190] The preset interactive processing flow serves as a handling method when the first page number differs from the second page number, and can be designed according to user needs and scenarios. For example, the preset interactive processing flow can be set to prompt the user that recognition failed.

[0191] In one example, to improve user experience and ensure the accuracy of page number recognition, when the first page number and the second page number are different, the second page corresponding to the first page number and the second page number can be displayed on the interactive interface. Then, the page number selected by the user from the displayed second page through the interactive interface is received, and the selected page number is used as the corresponding page number of the page to be recognized in the paper document. This ensures the flexibility of recognition, enhances user participation, and improves the overall processing accuracy.

[0192] In this embodiment of the disclosure, the accuracy and robustness of page number recognition are improved by using a dual verification method of text matching and image feature vector matching of the page to be identified, and the occurrence of false recognition is effectively reduced.

[0193] In some embodiments, a response processing method is also provided, which can be executed by an electronic device, including but not limited to any device capable of running a program such as a mobile phone, computer, tablet, or learning machine. This response processing method uses the image recognition method described in any of the above embodiments to recognize and subsequently process the acquired image. The method may include at least the following steps:

[0194] When a page image of the target paper document is obtained, the image recognition method described in any of the above embodiments is used to determine the target page number in the target paper document corresponding to the page image; according to the preset mapping relationship between the pages of the target paper document and device operations, the device is triggered to execute the device operation corresponding to the target page number.

[0195] Before executing the response processing method, the electronic device predefines and stores a mapping relationship between pages and device operations. This mapping relationship indicates the correspondence between each page or specific content on the target paper document and one or more device operations. This mapping relationship can be flexibly set according to actual needs. For example, the m-th page on the target paper device corresponds to playing the setting demonstration video P on the electronic device, and the (m+1)-th page corresponds to opening another application of this electronic device and demonstrating the usage steps of the application in the interactive interface of the application.

[0196] After the target page number of the page image is determined by image recognition method, the electronic device can automatically find and trigger the device operation associated with the target page number according to the preset mapping relationship. The user does not need to operate the device manually, but can control the operation of the device by simply displaying a specific page in the paper document, which greatly improves the convenience and efficiency of operation.

[0197] The above-mentioned response processing method can be widely applied to various scenarios that require the combination of paper documents and electronic device interaction and automated control, such as smart home control, industrial automated production lines, library book management, medical equipment operation demonstrations, new equipment start-up guides and real-time guidance demonstrations, etc. By introducing image recognition technology, a more intelligent and user-friendly interaction method and equipment response process can be achieved, improving user experience and work efficiency.

[0198] The electronic device, in acquiring page images of a target paper document, may include or be connected to an image acquisition device such as a scanner, camera, camera lens assembly, or the scanning / recognition area of ​​a learning machine. When the image acquisition device is operational, it continuously monitors whether an object is placed in the scanning area. Upon detecting a paper document in the scanning area, it activates an image capture function to acquire the image corresponding to the page of the paper document displayed in the current scanning area, which is then used as the page image. During image acquisition, the image acquisition device may apply image processing techniques such as automatic exposure control, color correction, and noise reduction to improve image quality and clarity.

[0199] Next, in order to enable those skilled in the art to better understand the image recognition method and response processing method provided in this application, this embodiment takes the scenario of triggering the real-time demonstration manual instruction content of the learning machine based on the learning machine's boot guide manual as an example to provide an exemplary explanation and description of the above image recognition method and response processing method.

[0200] In educational settings, learning machines serve as supplementary learning tools, incorporating rich learning resources and interactive teaching methods to facilitate students' self-directed learning. To help novice users learn how to use the learning machine, a startup guide manual and a built-in startup guide application have been designed. After opening the startup guide application, users place the startup guide manual on the scanning area of ​​the learning machine. The startup guide application will automatically identify which page of the manual is displayed in the scanning area, and then, based on a pre-created content mapping table and the content of that page, provide relevant explanations of the learning machine's features or open the corresponding application for a real-time demonstration of its functions.

[0201] Since the content of each page in the boot-up guide varies, with some pages containing both text and images, and others containing only images or only text, the image recognition method provided in this application can be used to improve the page recognition accuracy of the boot-up guide application in order to achieve page recognition for the application. (See [link to application]). Figure 7A The flowchart shown may include the following processing steps:

[0202] S701, Page Image Acquisition: Acquire the page image from the bootloader manual;

[0203] The boot-up manual can be placed directly on the scanning area of ​​the learning machine, and the image acquisition device on the learning machine, such as a high-definition camera, can capture the image of the manual page displayed in the scanning area; alternatively, the user can take a photo of a designated page of the boot-up manual and upload it to the learning machine via a photo album or other means.

[0204] S702, Blurred Image Judgment: The page image is preprocessed to obtain a first image that meets the input layer requirements of the pre-trained fuzzy judgment network, and the first image is input into the fuzzy judgment network so that the fuzzy judgment network extracts the edge features of the first image through a custom convolution operation and performs subsequent feature transformation, and outputs the classification category and confidence level of the first image as either blurry or clear; if the output classification category is blurry and the confidence level is greater than the blur threshold, the first image is determined to be blurry.

[0205] This embodiment employs a neural network based on convolutional operations for fuzzy classification, specifically the MobileNet V3 network. In practical applications, other types of networks can also be used for classification. The classification task based on this fuzzy classification network is to determine whether the input image is blurry. Since the edge features of a blurry image are weakened, standard convolutional operations may not be able to extract effective edge information. Therefore, this embodiment improves upon standard convolutional operations to enhance the richness and effectiveness of the extracted image edge features.

[0206] like Figure 7B The left side shows a 4x4 convolution kernel. If the image data for the current convolution calculation is the 4x4 image data shown on the right, the standard convolution operation can be represented as: Y = F(W*X + b), where F is the activation function, W is the convolution kernel parameter (the 4x4 matrix on the left of the image above), X is the image data for the current convolution calculation (the 4x4 matrix on the right of the image above), and b is the bias. Each weight on the convolution kernel is multiplied by the corresponding element in the image data, and all products are summed to obtain the result value W*X.

[0207] The custom convolution operations introduced in this embodiment include top-bottom convolution and left-right convolution. For Figure 7B The 4x4 convolution kernel and image data shown can be used for vertical convolution operations, which may include dividing the parameters of the convolution kernel into two equal parts along the horizontal direction, such as... Figure 7C As shown, W1 represents the first row (w11 to w14) and the second row (w21 to w24), W2 represents the third row (w31 to w34) and the fourth row (w41 to w44). Similarly, the image data currently undergoing convolution is also divided into upper and lower parts. X1 represents the first row (x11 to x14) and the second row (x21 to x24), X2 represents the third row (x31 to x34) and the fourth row (x41 to x44). The calculation process of upper and lower convolution can be expressed as: Y = F(abs(W1*X1 - W2*X2) + b), where the calculation process of W1*X1 and W2*X2 is the same as the calculation process and principle of standard convolution operation.

[0208] Similarly, left and right convolution operations can include: dividing the parameters of the convolution kernel into two equal parts along the vertical direction, such as... Figure 7DAs shown, W3 consists of the first column w11 to w41 and the second column w12 to w42, W4 consists of the third column w13 to w43 and the fourth column w14 to w44. Similarly, the image data currently undergoing convolution is also divided into left and right parts. X3 consists of the first column x11 to x41 and the second column x12 to x42, X4 consists of the third column x13 to x43 and the fourth column x14 to x44. The calculation process of left and right convolution can be expressed as: Y = F(abs(W3*X3-W4*X4)+b), where the calculation process of W3*X3 and W4*X4 is the same as the calculation process and principle of standard convolution operation.

[0209] The custom convolution operation in this embodiment is more conducive to extracting image edge features and performs better than the standard convolution operation in determining whether an image is blurred, thus enhancing the generalization ability of the fuzzy image detection network. To facilitate vertical or horizontal convolution operations, the kernel size of the convolutional layers in the fuzzy image detection network can be set to an even number * an even number.

[0210] S703, Image Detection and Coarse Classification Processing:

[0211] If the page image is identified as not being a blurred image by the fuzzy judgment network, a pre-trained detection network is used to detect the accompanying image in the page image. If the page image is found to contain an accompanying image, the detection box of the accompanying image is output and the first classification category with the highest confidence after the detection network performs a preliminary classification of the accompanying image is output. The multiple classification categories set in the output layer of the detection network correspond one-to-one with the page numbers of each first manual page containing the accompanying image in the learning machine's boot-up guide manual.

[0212] Before inputting the page image into the detection network, the page image is preprocessed, such as scaling to a standard size and image data standardization, to obtain a second image that meets the input layer requirements of the detection network, and then the second image is input into the detection network.

[0213] If the detection box information of the accompanying image is obtained, the detection box information and the page image are input into a pre-trained recognition network, and the image is accurately classified in step S704; if the page image does not include the accompanying image, the text and image search process is performed in step S706.

[0214] This embodiment uses the YOLOv5 network for detection. During convolution operations, left-right and top-bottom convolutions can be introduced into some convolutional layers of the detection network. The proportions of convolutional operations for the kernels used to implement standard convolution, left-right convolution, and top-bottom convolution are set to 1 / 2, 1 / 4, and 1 / 4 respectively. The specific proportions can be set according to actual needs. Introducing top-bottom and left-right convolutions improves both the generalization ability and the image detection capability of the detection network. In practical applications, other convolutional networks can also be used, and the proportions of top-bottom and left-right convolutions can be adjusted according to the application scenario.

[0215] S704, with precise image categorization:

[0216] Based on the received detection box information and the page image, the corresponding image is cropped from the page image and input into the recognition network. The recognition network then predicts the classification category and confidence level of the image based on the parameters learned in the pre-training stage, and obtains the second classification category with the highest confidence level. The multiple classification categories set in the output layer of the recognition network correspond one-to-one with the page numbers of each page of the first manual containing the illustrations in the learning machine's boot-up guide.

[0217] This embodiment uses the ShuffleNetv2 network for recognition. During convolution operations, left-right and top-bottom convolutions can be introduced into some convolutional layers of the recognition network. The proportions of convolutional operations for the kernels used to implement standard convolution, left-right convolution, and top-bottom convolution are set to 1 / 2, 1 / 4, and 1 / 4 respectively. The specific proportions can be set according to actual needs. Introducing top-bottom and left-right convolutions improves the generalization ability and image detection capability of the recognition network. In practical applications, other convolutional networks can also be used, and the proportions of top-bottom and left-right convolutions can be adjusted according to the application scenario. Introducing top-bottom and left-right convolutions helps the network extract edge features of the image effectively, improving the recognition network's recognition capability for accurate image classification.

[0218] S705, Dual-network classification result verification processing:

[0219] For the second category output by the identification network and the first category output by the detection network, if the page number corresponding to the second category is the same as that of the first category and the confidence level of the second category is greater than the set confidence level threshold, the page number corresponding to the second category is determined as the corresponding page number of the page image obtained in step S701 in the boot guide manual; otherwise, the text and image search processing in step S706 is entered.

[0220] S706, Text and Image Search Processing:

[0221] The image feature vectors, page text, and corresponding page numbers for each page of the boot-up manual are pre-stored in a pre-defined database such as the Elasticsearch library.

[0222] When entering the text and image search processing, OCR technology can be used to extract the text content of the page image, and a text search can be performed in the preset database based on the text content to find the page text that is most similar to the text content. If the similarity is greater than the first threshold, the search result is determined to be valid, and the page number corresponding to the most similar page text is determined as the corresponding page number of the page image in the boot guide manual.

[0223] Alternatively, an image feature vector extraction algorithm can be used to extract a fixed-dimensional third feature vector from the page image, and a similar vector search can be performed in the preset database based on the third feature vector. The similarity of the feature vectors can be determined by calculating the vector distance, so as to find the image feature vector of the boot-up manual page that has a similarity greater than a second threshold with the third feature vector. The page number corresponding to the image feature vector of the most similar manual page is determined as the corresponding page number of the page image in the boot-up manual.

[0224] This embodiment uses the Mobilenetv3 small network to extract image feature vectors, but other networks can also be used in practical applications. When performing convolution operations, left-right and top-bottom convolution operations can be introduced into some convolutional layers of the recognition network. The proportions of convolution operations for the kernels used to implement standard convolution, left-right convolution, and top-bottom convolution in the convolutional layers can be set to 1 / 2, 1 / 4, and 1 / 4 respectively. The specific proportions can be set according to actual needs.

[0225] The text content search or third feature vector search described above can be used individually, meaning only one method needs to be employed. To improve the accuracy of image page recognition, when entering the text and image search process after the dual-network classification result verification, the third feature vector search method for page images is preferably used to determine the corresponding page number. When entering the text and image search process without accompanying images, both text content search and third feature vector search methods can be used simultaneously. The two page numbers determined by the two methods are compared. If the two page numbers are the same, that page number is taken as the corresponding page number; otherwise, an unrecognizable prompt result is output. Compared to related technologies that only use pure text recognition, this improves the accuracy and flexibility of recognition.

[0226] Through the overall processing flow of the above image recognition method and the introduction of vertical or horizontal convolution operations for each network, the richness and effectiveness of extracted features are enhanced, effectively improving the accuracy of image page number recognition and the robustness of network processing. It can also provide effective feature extraction and image page number recognition for page number images with poor image quality due to poor lighting conditions or other conditions.

[0227] Based on this image recognition method, the response processing method can be applied to the learning machine in this application scenario. This method may include the following steps:

[0228] If an image of one page of the boot-up manual is obtained, the image recognition method described above is used to determine the target page number in the boot-up manual corresponding to that page image. Based on the content mapping table of boot-up manual pages and guided demonstration operations preset in the learning machine, the learning machine is triggered to execute the guided demonstration operation associated with the boot-up manual page corresponding to the target page number.

[0229] For example, the backend of the boot-up guide application creates a content mapping table that associates each page of the boot-up guide with a specific demo video, animation, or a specific function interface of a particular application. For instance, page 5 of the guide might be associated with the "Mental Arithmetic Correction" function. When a user places a page of the guide in front of or behind the learning machine's camera, the machine captures an input image of that page. Using the image recognition method described above, it identifies the corresponding page number in the boot-up guide, queries the content mapping table, determines the associated operation for that page number, and initiates it. Assuming the identified page number is page 5, the boot-up guide application attempts to launch the pre-installed mental arithmetic app and guides the user through a series of interactive steps ("Step 1-Step 2") to access a function interface, such as the correction mode. The user is then guided to click on the correction mode, and the current page is used to demonstrate the correction of mental arithmetic problems displayed in the input image.

[0230] Corresponding to the embodiments of the aforementioned image recognition method, see [link to relevant documentation]. Figure 8 As shown, this application also provides an embodiment of an image recognition device, the device comprising:

[0231] The page to be identified module 801 is used to acquire page images of paper documents as pages to be identified;

[0232] The image detection and recognition module 802 is used to detect whether the page to be recognized includes an image using a pre-trained detection network, and if the page to be recognized includes an image, determine a first classification result for the image; crop the image from the page to be recognized, and input the image into a pre-trained recognition network to obtain a second classification result for the image; wherein, both the detection network and the recognition network include multiple classification categories, and each classification category corresponds to all images on a single page in the paper document;

[0233] The target page determination module 803 is used to determine the corresponding page number of the page to be identified in the paper document when the second classification result matches the first classification result.

[0234] In some embodiments, the target network includes the detection network or the recognition network; each convolutional layer of the target network includes at least one of the custom convolutional operations, namely, vertical convolution and horizontal convolution; the vertical convolution includes horizontally dividing the first convolutional kernel into two equal-sized sub-convolutional kernels and then calculating the difference between the vertical and horizontal convolutional results; the horizontal convolution includes vertically dividing the second convolutional kernel into two equal-sized sub-convolutional kernels and then calculating the difference between the horizontal and horizontal convolutional results.

[0235] In some embodiments, based on the custom convolution operation, the image detection and recognition module is specifically used for:

[0236] The detection network is used to determine the bounding box of the image, and the region of interest in the page to be identified is determined based on the bounding box.

[0237] The first image feature of the region of interest is extracted through the custom convolution operation in the detection network, and the first classification result of the image is predicted based on the first image feature.

[0238] In some embodiments, based on the custom convolution operation, the image detection and recognition module is specifically used for:

[0239] The page to be identified is cropped based on the bounding box of the image determined by the detection network to obtain the image and input into the recognition network;

[0240] The second image feature of the image is extracted through the custom convolution operation in the recognition network, and the second classification result of the image is predicted based on the second image feature.

[0241] In some embodiments, the plurality of convolutional kernels used by each convolutional layer of the target network include a first convolutional kernel of a first proportion for implementing the vertical convolution, a second convolutional kernel of a second proportion for implementing the horizontal convolution, and a third convolutional kernel of a third proportion for implementing standard convolution.

[0242] The vertical convolution or the horizontal convolution includes: dividing the corresponding convolution kernel into two equal-sized sub-convolution kernels; calculating the convolution result obtained by weighted summation of each sub-convolution kernel, and obtaining the absolute value of the difference between the two convolution results; and obtaining the corresponding final convolution result based on the absolute value.

[0243] In some embodiments, the target page determination module is specifically used for:

[0244] Obtain the first predicted category with the highest confidence in the first classification result, and obtain the second predicted category with the highest confidence in the second classification result;

[0245] If the second predicted category meets the set conditions, it is determined that the second classification result matches the first classification result, and the page number of the page in the paper document corresponding to the second predicted category is obtained as the corresponding page number; the set conditions include: the confidence level of the second predicted category is greater than or equal to a preset threshold, and the second predicted category is the same as the first predicted category.

[0246] In some embodiments, the apparatus further includes:

[0247] If the second classification result does not match the first classification result, image feature vector extraction is performed on the page to be identified to obtain a first feature vector of fixed dimensions.

[0248] The first feature vector is matched with the feature vector of the first page containing the illustration in the pre-stored paper document, and the page number corresponding to the matched feature vector is determined as the corresponding page number.

[0249] In some embodiments, the target network further includes a pre-trained fuzzy judgment network; the apparatus further includes:

[0250] By using the custom convolution operation in the fuzzy judgment network, the edge features of the page to be identified are extracted and passed to the subsequent layers of the fuzzy judgment network to obtain the fuzzy classification result of the page to be identified.

[0251] If the fuzzy classification result indicates that the page to be identified is not blurry, the clarity of the page to be identified is determined to meet the set requirements; otherwise, the clarity is determined to not meet the set requirements.

[0252] If the clarity meets the set requirements, detect whether the page to be identified includes an image.

[0253] In some embodiments, the apparatus further includes:

[0254] If the page to be identified does not contain an image, the text content of the page to be identified is extracted and matched with the page text of each second page in the pre-stored paper document that does not contain an image, and the first page number corresponding to the matched page text is obtained.

[0255] The image feature vector of the page to be identified is extracted to obtain a second feature vector of fixed dimension, and then matched with the feature vector of the pre-stored second page to obtain the second page number of the second page corresponding to the matched feature vector;

[0256] If the first page number and the second page number are the same, the first page number is determined as the corresponding page number of the page to be identified in the paper document.

[0257] In some embodiments, the apparatus further includes:

[0258] If the first page number and the second page number are different, control the display of the second page corresponding to the first page number and the second page number on the interactive interface;

[0259] Receive the page number selected by the user from the second page displayed through the interactive interface, and use the selected page number as the corresponding page number.

[0260] Corresponding to the embodiments of the aforementioned response processing method, see [link to relevant documentation]. Figure 9 As shown, this application also provides an embodiment of a response processing apparatus, the apparatus comprising:

[0261] Page number determination module 901 is used to determine the target page number in the target paper document corresponding to the page image by using the image recognition method described in any of the above embodiments when a page image of the target paper document is obtained.

[0262] The trigger execution module 902 is used to trigger the device to execute the device operation corresponding to the target page number according to the preset mapping relationship between the pages of the target paper document and the device operation.

[0263] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0264] This application also provides an electronic device, the structural schematic diagram of which is shown below. Figure 10As shown, the electronic device 1000 includes at least one processor 1001, a memory 1002, and a bus 1003. The at least one processor 1001 is electrically connected to the memory 1002. The memory 1002 is configured to store at least one computer-executable instruction, and the processor 1001 is configured to execute the at least one computer-executable instruction to perform the steps of any image recognition method or response processing method provided in any embodiment or optional implementation of this application.

[0265] Furthermore, the processor 1001 can be an FPGA (Field-Programmable Gate Array) or other devices with logic processing capabilities, such as an MCU (Microcontroller Unit) or a CPU (Central Processing Unit).

[0266] This application also provides another readable storage medium storing a computer program that, when executed by a processor, implements the steps of any image recognition method or response processing method provided in any embodiment or optional implementation of this application.

[0267] The readable storage media provided in this application include, but are not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. In other words, the readable storage media includes any medium by which a device (e.g., a computer) stores or transmits information in a readable form.

[0268] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image recognition method, characterized in that, The method includes: Obtain page images of paper documents as the pages to be recognized; A pre-trained detection network is used to detect whether the page to be identified includes an image, and if the page to be identified includes an image, a first classification result for the image is determined. The image is extracted from the page to be identified and input into a pre-trained recognition network to obtain a second classification result of the image; wherein, both the detection network and the recognition network include multiple classification categories, and each classification category corresponds to all the images on a single page in the paper document; Obtain the first predicted category with the highest confidence in the first classification result, and obtain the second predicted category with the highest confidence in the second classification result; If the second predicted category meets the set conditions, it is determined that the second classification result matches the first classification result, and the page number of the page in the paper document corresponding to the second predicted category is obtained as the corresponding page number; the set conditions include: the confidence level of the second predicted category is greater than or equal to a preset threshold, and the second predicted category is the same as the first predicted category.

2. The method according to claim 1, characterized in that, The target network includes the detection network or the recognition network; each convolutional layer of the target network includes at least one of the custom convolutional operations, namely, vertical convolution and horizontal convolution; the vertical convolution includes horizontally dividing the first convolutional kernel into two equal-sized sub-convolutional kernels and then calculating the difference between the vertical and horizontal convolutional results; the horizontal convolution includes vertically dividing the second convolutional kernel into two equal-sized sub-convolutional kernels and then calculating the difference between the horizontal and horizontal convolutional results.

3. The method according to claim 2, characterized in that, When the page to be identified includes an image, determining the first classification result of the image includes: The detection network is used to determine the bounding box of the image, and the region of interest in the page to be identified is determined based on the bounding box. The first image feature of the region of interest is extracted through the custom convolution operation in the detection network, and the first classification result of the image is predicted based on the first image feature.

4. The method according to claim 2, characterized in that, The process of cropping the image from the page to be identified and inputting the image into a pre-trained recognition network to obtain a second classification result for the image includes: The page to be identified is cropped based on the bounding box of the image determined by the detection network to obtain the image and input into the recognition network; The second image feature of the image is extracted through the custom convolution operation in the recognition network, and the second classification result of the image is predicted based on the second image feature.

5. The method according to any one of claims 2-4, characterized in that, The multiple convolutional kernels used in each convolutional layer of the target network include a first convolutional kernel with a first ratio for implementing the vertical convolution, a second convolutional kernel with a second ratio for implementing the horizontal convolution, and a third convolutional kernel with a third ratio for implementing standard convolution. The vertical convolution or the horizontal convolution includes: dividing the corresponding convolution kernel into two equal-sized sub-convolution kernels; calculating the convolution result obtained by weighted summation of each sub-convolution kernel, and obtaining the absolute value of the difference between the two convolution results; and obtaining the corresponding final convolution result based on the absolute value.

6. The method according to claim 1, characterized in that, The method further includes: If the second classification result does not match the first classification result, image feature vector extraction is performed on the page to be identified to obtain a first feature vector of fixed dimensions. The first feature vector is matched with the feature vector of the first page containing the illustration in the pre-stored paper document, and the page number corresponding to the matched feature vector is determined as the corresponding page number.

7. The method according to claim 2, characterized in that, The target network further includes a pre-trained fuzzy judgment network; the method further includes: By using the custom convolution operation in the fuzzy judgment network, the edge features of the page to be identified are extracted and passed to the subsequent layers of the fuzzy judgment network to obtain the fuzzy classification result of the page to be identified. If the fuzzy classification result indicates that the page to be identified is not blurry, the clarity of the page to be identified is determined to meet the set requirements; otherwise, the clarity is determined to not meet the set requirements. If the clarity meets the set requirements, detect whether the page to be identified includes an image.

8. The method according to claim 1, characterized in that, The method further includes: If the page to be identified does not contain an image, the text content of the page to be identified is extracted and matched with the page text of each second page in the pre-stored paper document that does not contain an image, and the first page number corresponding to the matched page text is obtained. The image feature vector of the page to be identified is extracted to obtain a second feature vector of fixed dimension, and then matched with the feature vector of the pre-stored second page to obtain the second page number of the second page corresponding to the matched feature vector; If the first page number and the second page number are the same, the first page number is determined as the corresponding page number of the page to be identified in the paper document.

9. The method according to claim 8, characterized in that, The method further includes: If the first page number and the second page number are different, control the display of the second page corresponding to the first page number and the second page number on the interactive interface; Receive the page number selected by the user from the second page displayed through the interactive interface, and use the selected page number as the corresponding page number.

10. A response processing method, characterized in that, The method includes: When a page image of the target paper document is obtained, the image recognition method as described in any one of claims 1-9 is used to determine the target page number in the target paper document corresponding to the page image; Based on the preset mapping relationship between the target paper document's pages and device operations, the device is triggered to execute the device operation corresponding to the target page number.

11. An image recognition device, characterized in that, The device includes: The page to be identified module is used to acquire page images of paper documents as pages to be identified; The image detection and recognition module is used to detect whether the page to be recognized includes an image using a pre-trained detection network, and if the page to be recognized includes an image, determine a first classification result for the image; crop the image from the page to be recognized, and input the image into a pre-trained recognition network to obtain a second classification result for the image; wherein, both the detection network and the recognition network include multiple classification categories, and each classification category corresponds to all images on a single page in the paper document; The target page determination module is used to obtain the first predicted category with the highest confidence in the first classification result and the second predicted category with the highest confidence in the second classification result; when the second predicted category meets the set conditions, it determines that the second classification result matches the first classification result and obtains the page number of the page in the paper document corresponding to the second predicted category as the corresponding page number; the set conditions include: the confidence of the second predicted category is greater than or equal to a preset threshold, and the second predicted category is the same as the first predicted category.

12. A response processing apparatus, characterized in that, The device includes: The page number determination module is used to determine the target page number in the target paper document corresponding to the page image when the page image of the target paper document is obtained, using the image recognition method as described in any one of claims 1-9. The trigger execution module is used to trigger the device to execute the device operation corresponding to the target page number based on the preset mapping relationship between the pages of the target paper document and the device operation.

13. An electronic device, characterized in that, include: Memory, processor; The memory is used to store computer programs; The processor is configured to invoke the computer program to implement the method as described in any one of claims 1-10.

14. A readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-10.

Images