Check identification method, device and equipment suitable for mobile terminal and medium

Abstract

The present disclosure provides a check recognition method suitable for mobile terminals, relates to the field of artificial intelligence, and can be applied to the field of financial technology. The method comprises the following steps: extracting image features of a check image by using a text region detection model, detecting text regions in the check image based on the image features, and then identifying text information in the text regions by using a text recognition model. When extracting the image features of the check image, a plurality of network layers in a backbone feature extraction network are used to extract the image features of the check image in a serial manner. The first feature image output by a predetermined network layer in the plurality of network layers is first processed by a feature enhancement module to obtain a second feature image, and then the second feature image is transmitted back to the backbone feature extraction network for continuous processing. The present disclosure also provides a check recognition device suitable for mobile terminals, equipment, a storage medium and a program product.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and can be used in the field of financial technology, and more specifically to a check recognition method, device, equipment, medium and program product suitable for mobile devices. Background Technology

[0002] Check transactions are a common banking service. With the rapid development of mobile internet, mobile banking apps have also launched check deposit services. When using a check for transactions (e.g., check deposit) on a mobile banking app, the user typically takes a picture of the check with their phone, uploads it to a backend server for image processing, and then returns the result to the client to complete the check deposit process. However, when the check image is large—for example, with the improved performance of mobile phone cameras, the file size of captured images is increasingly large—uploading the check image to the backend server becomes highly dependent on network speed. Under weak network conditions, image uploads may take too long or even fail, impacting the user experience. Summary of the Invention

[0003] In view of the above problems, this disclosure provides a check recognition method, apparatus, device, medium and program product suitable for mobile devices.

[0004] According to a first aspect of this disclosure, a check recognition method suitable for mobile devices is provided. The method includes: acquiring a check image to be recognized; extracting image features of the check image using a text region detection model, and detecting text regions in the check image based on the image features; recognizing text information in the text regions using a text recognition model; and uploading the text information to a server. The step of extracting image features of the check image using a text region detection model includes: extracting image features of the check image using multiple network layers in a backbone feature extraction network in a concatenated manner, wherein a first feature image output by a predetermined network layer in the multiple network layers is first enhanced by a feature enhancement module to obtain a second feature image, and then the second feature image is transmitted back to the backbone feature extraction network for further processing.

[0005] According to embodiments of this disclosure, enhancing the first feature image through the feature enhancement module includes: extracting features from the first feature image at n size levels to obtain n first feature sub-images, where n is an integer greater than or equal to 2, and the n size levels include a global size level for feature extraction based on a 1x1 grid and at least one local size level for feature extraction based on a grid different from 1x1; fusing the n first feature sub-images to obtain a second feature sub-image; and processing the second feature sub-image according to the image parameters required for further processing by the backbone feature extraction network to obtain the second feature image.

[0006] According to an embodiment of this disclosure, the step of extracting features from the first feature image at n size levels to obtain n first feature sub-images includes: performing pooling processing on the first feature image using n pooling layers corresponding to the n size levels respectively; and performing convolution processing on the outputs of the n pooling layers using n first convolution layers to compress the number of channels of the image to obtain the n first feature sub-images.

[0007] According to an embodiment of this disclosure, the kernel size of the first convolutional layer is 1*1.

[0008] According to embodiments of this disclosure, the step of fusing the n first feature sub-images to obtain a second feature sub-image includes: using the first feature sub-image corresponding to the global size level as the initial sampling object, continuously updating the sampling object in the following manner until all n feature sub-images are fused and the sampling object is output: for the sampling object, upsampling is performed according to the image resolution corresponding to the size level adjacent to the size level of the sampling object to obtain a feature sub-image to be fused; the feature sub-image to be fused and the first feature sub-image corresponding to the size level adjacent to the size level of the sampling object are fused to obtain a new sampling object.

[0009] According to an embodiment of this disclosure, the step of fusing the features of the n first feature sub-images to obtain a second feature sub-image further includes: fusing the sampled object output after all n feature sub-images have been fused with the first feature image to obtain the second feature sub-image.

[0010] According to embodiments of this disclosure, processing the second feature sub-image according to the image parameters required for further processing by the backbone feature extraction network to obtain the second feature image includes: when the predetermined network layer is one of the multiple network layers, performing convolution processing on the second feature sub-image according to the image parameters input to the next network layer of the predetermined network layer to obtain the second feature image; and when the predetermined network layer is the last network layer among the multiple network layers, performing convolution processing on the second feature sub-image according to the image parameters output by the backbone feature extraction network to obtain the second feature image.

[0011] A second aspect of this disclosure provides a check recognition device suitable for mobile devices. The device includes an acquisition module, a text region detection model, a text recognition model, and an upload module. The acquisition module acquires an image of a check to be recognized. The text region detection model extracts image features from the check image and detects text regions in the check image based on the image features. The text recognition model recognizes text information within the text regions. The upload module uploads the text information to a server. The text region detection model includes a backbone feature extraction network and a feature enhancement module. The backbone feature extraction network extracts image features from the check image through multiple network layers in a concatenated manner. The feature enhancement module enhances a first feature image output from a predetermined network layer in the multiple network layers into a second feature image, and then transmits the second feature image back to the backbone feature extraction network for further processing.

[0012] According to embodiments of this disclosure, the feature enhancement module is specifically configured to: extract features from the first feature image at n size levels to obtain n first feature sub-images, where n is an integer greater than or equal to 2, wherein the n size levels include a global size level for feature extraction according to a 1x1 grid and at least one local size level for feature extraction according to a different 1x1 grid; fuse the n first feature sub-images to obtain a second feature sub-image; and process the second feature sub-image according to the image parameters required for further processing by the backbone feature extraction network to obtain the second feature image.

[0013] A third aspect of this disclosure provides an electronic device. The electronic device includes one or more processors and a memory. The memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0014] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods described above.

[0015] A fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0016] The above one or more embodiments have the following advantages or beneficial effects: they can at least partially avoid problems such as excessively long upload times or even upload failures when uploading check images from mobile devices to the server in weak network environments. By adding a feature enhancement module to the lightweight backbone feature extraction network used on the mobile device, the feature enhancement module can further extract and enhance the image features output by at least one network layer, thereby improving the accuracy of feature extraction and the accuracy of text region detection, which is suitable for the business needs of mobile devices. Attached Figure Description

[0017] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 The illustration schematically depicts an application scenario of a check recognition method, apparatus, device, medium, and program product suitable for mobile devices according to embodiments of the present disclosure.

[0019] Figure 2 A flowchart illustrating a check recognition method suitable for mobile devices according to an embodiment of the present disclosure is shown schematically.

[0020] Figure 3 The flowchart illustrating the detection of text regions in a check image using a text detection model according to an embodiment of the present disclosure is shown in the illustration.

[0021] Figure 4 The illustration schematically shows the enhancement process of the feature enhancement module on the first feature image in a check recognition method according to an embodiment of the present disclosure;

[0022] Figure 5 This illustration schematically shows the process of applying a backbone feature extraction network and a feature enhancement module to recognize check images in a check recognition method according to another embodiment of this disclosure;

[0023] Figure 6 A schematic block diagram of a check recognition device suitable for mobile devices according to embodiments of the present disclosure is shown; and

[0024] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a check recognition method according to embodiments of the present disclosure. Detailed Implementation

[0025] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0027] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0028] When expressions such as "at least one of A, B, and C" are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having only A, only B, only C, A and B, A and C, B and C, and / or systems having A, B, and C, etc.). The terms "first," "second," etc., used herein are for distinction only and have no limiting meaning, and the number of any elements in the accompanying drawings is for illustrative purposes only and not for limitation.

[0029] The check recognition process can be divided into two parts: text detection and text recognition. Text detection identifies the text regions in the check image and is a preliminary step for text recognition. The accuracy of text detection directly affects the accuracy of text recognition.

[0030] The models used on the server side for recognizing check images are typically quite large, primarily due to the large size of the text detection models. Mobile devices (such as smartphones and iPads) have limited data processing capabilities, making it difficult to directly port server-side models to mobile devices. To address this, the inventors proposed a lightweight network model based on a feature enhancement module. This network model, used as a text region detection model and combined with a corresponding text recognition model, can meet the check recognition requirements of mobile devices.

[0031] Specifically, the check recognition method, apparatus, device, medium, and program products suitable for mobile terminals provided in this disclosure can utilize a text region detection model including the aforementioned feature enhancement module installed on the mobile terminal to perform text detection, then utilize a text recognition model installed on the mobile terminal to perform text recognition, and finally upload the text recognition result to the server for check transactions. The feature enhancement module can further extract and enhance the image features output by at least one network layer in the backbone feature extraction network of the text region detection model, thereby improving the accuracy of feature extraction and text region detection. This allows the text region detection model to be designed as a lightweight model that does not lose recognition accuracy, suitable for the business needs of mobile terminals.

[0032] Figure 1 The illustration schematically depicts an application scenario of a check recognition method, apparatus, device, medium, and program product suitable for mobile devices according to embodiments of the present disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0033] like Figure 1 As shown, application scenario 100 according to this embodiment may include a mobile terminal 101, a network 102, and a server 103. The network 102 is used as a medium to provide a communication link between the mobile terminal 101 and the server 103. The network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0034] Users can use mobile terminal 101 to interact with server 103 via network 12 to receive or send messages, etc. Various communication client applications can be installed on mobile terminal 101, such as shopping applications, mobile banking applications, social media platform software, etc. (for example only). Mobile terminal 101 can be various electronic devices with a display screen and web browsing support, including but not limited to smartphones, iPads, etc.

[0035] Server 103 can be a server that provides various services, such as a backend management server that supports websites browsed by users using mobile terminal 101 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0036] It should be noted that the check recognition method for mobile terminals provided in this disclosure can be executed by the mobile terminal 101. Accordingly, the check recognition device, equipment, medium, and program products for mobile terminals provided in this disclosure can also be disposed in the mobile terminal 101.

[0037] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0038] The following will be based on Figure 1 The described scene, through Figures 2-5 A check recognition method applicable to mobile devices according to embodiments of this disclosure will be described in detail. It should be noted that the sequence numbers of each operation in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0039] Figure 2 A flowchart illustrating a check recognition method for mobile devices according to an embodiment of the present disclosure is shown schematically.

[0040] like Figure 2 As shown, the check recognition method according to this embodiment may include operations S210 to S240.

[0041] First, in operation S210, an image of the check to be recognized is acquired. For example, a user can use mobile terminal 101 to open a mobile banking client and trigger the camera to capture an image of the check. Alternatively, for another example, a user can use a communication client (such as WeChat or email) in mobile terminal 101 to receive check images sent by other users and then provide them to the mobile banking application in mobile terminal 101.

[0042] Then, in operation S220, the text region detection model is used to detect the text region in the check image.

[0043] The text region detection model includes a backbone feature extraction network and a feature enhancement module. The backbone feature extraction network may include multiple network layers (e.g., convolutional layers, pooling layers, etc.). The feature enhancement module may be placed between two predetermined network layers, or after the last layer of the backbone feature network. The text region detection model can have one or multiple feature enhancement modules. To adapt to the lightweight network construction of mobile devices, in one embodiment, the backbone feature extraction network may adopt the MobileNet network.

[0044] Figure 3The flowchart illustrating the detection of text regions in a check image using a text detection model in operation S220 according to an embodiment of the present disclosure is shown in the illustration.

[0045] Combination Figure 3 Operation S220 may include operations S221 to S222.

[0046] First, in operation S221, multiple network layers in the backbone feature extraction network are used to extract image features of the check image in a series manner. The first feature image output by a predetermined network layer in the multiple network layers is first enhanced by the feature enhancement module to become a second feature image, and then the second feature image is transmitted back to the backbone feature extraction network for further processing.

[0047] Then, in operation S222, text regions in the check image are detected based on image features. The check mainly contains table lines, logo, seal, and text. By training the text detection model, it can locate text regions by analyzing the coordinate relationships or distribution characteristics of pixels with different features based on the extracted image features.

[0048] Adding a feature extraction module to the backbone feature extraction network can increase the network depth, improve feature extraction accuracy, and enhance the accuracy of text region inspection.

[0049] Next, S230 is operated to recognize text information in the text region using a character recognition model. This text information can include text location information and semantic information. The character recognition model can be an artificial intelligence model based on Optical Character Recognition (OCR) technology.

[0050] Finally, in operation S240, the text information is uploaded to the server. For example, server 103 performs a check transaction based on the received text information.

[0051] The text detection and recognition models are pre-trained artificial intelligence models that require pre-training. For example, a large number of check images can be collected on server 103 or other servers (cluster) to train the text detection and recognition models. Once the recognition accuracy of the trained text detection and recognition models meets the requirements, they can be packaged together and integrated into the mobile banking client for users to download and use on their mobile terminals 101.

[0052] The training process for the text detection model and the text recognition model can be carried out in two ways: either they can be trained jointly, using the same set of training sample data and using the output of the text detection model as the input of the text recognition model for training; or they can be trained separately, with the collected check images being preprocessed and labeled according to the training requirements of each model to form training sample data for each model.

[0053] This disclosure allows for the migration of check recognition from the server-side to the mobile device, enabling effective processing of check images even under weak network conditions and improving user experience. Furthermore, it avoids the risk of user information leakage that may occur when mobile devices upload check images to the server, thus enhancing user information security.

[0054] Figure 4 The illustration schematically shows the enhancement process of the feature enhancement module on the first feature image in a check recognition method according to an embodiment of the present disclosure.

[0055] like Figure 4 As shown, the process of enhancing the first feature image output by a certain network layer in the backbone feature network according to the embodiment of this disclosure may include operations S401 to S403.

[0056] First, in operation S401, feature extraction is performed on the first feature image at n size levels to obtain n first feature sub-images, where n is an integer greater than or equal to 2.

[0057] The n size levels include a global size level for feature extraction using a 1x1 grid and at least one local size level for feature extraction using a grid different from 1x1 (such as 2x2, 4x4, 8x8). When extracting features using a certain grid, each time the pixel features within the region defined by that grid are weighted and averaged to obtain the feature of a pixel in the output image. Then, after one extraction is completed, the grid can be moved across the input image at a preset step size; feature extraction is iterated continuously in this manner.

[0058] Feature extraction from the first feature image using a 1x1 grid can extract global features. Conversely, feature extraction from the first feature image using a grid other than 1x1 can extract local features within the corresponding grid area.

[0059] In one embodiment, at least one local size level may include multiple size levels. In one embodiment, the largest grid may be half the grid of the first feature image extracted by a predetermined network layer. For example, when the first feature image is 16x16, the largest grid among the n size levels may be 8x8. In this way, the features extracted at n size levels include features at multiple levels, from global features to local features at multiple different scales, increasing the hierarchy and richness of features compared to the first feature image.

[0060] In a specific implementation, in one embodiment, the first feature image can be pooled using n pooling layers corresponding to n size levels respectively, and then the outputs of the n pooling layers can be convolved using n first convolutional layers to compress the number of channels of the image, resulting in n first feature sub-images.

[0061] Convolutional layers extract image features through convolution operations. Pooling layers operate similarly to convolutional layers, primarily for feature reduction, significantly decreasing the size of the image parameter matrix. Pooling layers typically do not affect the interaction between channels; they only perform dimensionality reduction within each channel. Convolutional layers, on the other hand, allow interaction between channels.

[0062] In one embodiment, to reduce model processing parameters, the kernel size of the n first convolutional layers can be selected as 1*1. During image processing, the convolution kernel is used to weight and average pixels within a small region of the input image to obtain the corresponding pixel in the output image. The weights are defined by a function called the convolution kernel. The parameters of the convolution kernel include kernel size, stride, and padding steps. Kernel size represents the size of the receptive field in the network. For example, a 1*1 convolution kernel means that each convolution only weights and averages pixels within a 1*1 region to obtain the corresponding pixel in the output image. The stride represents the extraction precision, i.e., the length spanned in each convolution.

[0063] As can be seen, the n first feature sub-images extracted by operation S401 include features from multiple levels from global to local, increasing the receptive field of feature extraction and enriching the features that can be extracted.

[0064] Next, in operation S402, feature fusion is performed on the n first feature sub-maps to obtain the second feature sub-map.

[0065] During feature fusion, the first feature sub-image obtained using a small grid from n first feature sub-images can be upsampled and then fused with the first feature sub-images of adjacent size levels. Specifically, upsampling ensures that the feature extraction scale of the image extracted using the small grid is consistent with that of the first feature images corresponding to adjacent size levels, thus enabling feature fusion.

[0066] Specifically, the first feature sub-image corresponding to the global size level can be used as the initial sampling object. The sampling object is continuously updated in the following manner until all n feature sub-images are fused and the sampling object is output: First, the sampling object is upsampled according to the image resolution corresponding to the size level adjacent to the size level of the sampling object to obtain the feature sub-image to be fused; then, the feature sub-image to be fused and the first feature sub-image corresponding to the size level adjacent to the size level of the sampling object are fused to obtain a new sampling object.

[0067] In one embodiment, the sampled object output after the n first feature sub-images are fused step-by-step as described above can be used as the second feature sub-image. In another embodiment, the sampled object output after the n feature sub-images are all fused can also be fused with the first feature image to obtain the second feature sub-image. This way, the features of the original first feature image are not lost, ensuring that the features of the second feature sub-image are richer than those of the first feature image passed to the feature enhancement module. The embodiments of this disclosure use a feature pyramid pattern to perform feature fusion iteratively step by step, which can improve the robustness of the text detection model to text at different scales.

[0068] Then, in operation S403, the second feature sub-image is processed according to the image parameters required for further processing by the backbone feature extraction network to obtain the second feature image. This method ensures the smooth execution of the backbone feature extraction network and avoids excessive intervention or modification to the backbone feature extraction network.

[0069] Specifically, when the predetermined network layer is one of multiple network layers, the second feature sub-image is convolved according to the image parameters input to the next network layer of the predetermined network layer to obtain the second feature image. Alternatively, when the predetermined network layer is the last network layer of multiple network layers, the second feature sub-image is convolved according to the image parameters output by the backbone feature extraction network to obtain the second feature image.

[0070] In this embodiment, a feature enhancement module can be added to the lightweight backbone feature extraction network used on mobile devices, which increases the network depth and can improve the speed and accuracy of text detection by reducing the number of parameters, making it suitable for relevant business needs on mobile devices.

[0071] Figure 5 This illustration schematically depicts the process of applying a backbone feature extraction network and a feature enhancement module to recognize a check image in another embodiment of the check recognition method of this disclosure. Those skilled in the art will understand that... Figure 5 The illustrations shown are for illustrative purposes only and do not constitute a limitation of this disclosure.

[0072] like Figure 5As shown, in this embodiment, when recognizing a check image, the process of extracting features from the check image may include the following steps 1 to 7.

[0073] Step 1: The obtained check image is used to perform bottom-up feature extraction using a backbone feature extraction network to extract the feature layer (i.e., the first feature image mentioned above), and then input into the feature enhancement module.

[0074] Step 2: The feature layer input to the feature enhancement module is processed by a pooling layer. Specifically, assuming the first feature image size is 16×16×32, four independent pooling layers can be used, performing average pooling according to a grid size of 1×1, 2×2, 4×4, and 8×8 respectively. These 1×1, 2×2, 4×4, and 8×8 grid sizes correspond to four size levels.

[0075] In one embodiment, pooling can yield pooling layer outputs of 1×1×32, 2×2×32, 4×4×32, and 8×8×32.

[0076] Step 3: The outputs of the above four pooling layers are passed through four first convolutional layers to reduce the number of channels in each first feature map to one-quarter of its original size. The kernel size of these four first convolutional layers is 1×1. The outputs of the four first convolutional layers are first feature maps of 1×1×8, 2×2×8, 4×4×8, and 8×8×8, respectively.

[0077] Step 4: Proceed to the feature fusion stage. For every two adjacent size levels, the first feature sub-image obtained using the smaller grid is upsampled and then fused with the first feature sub-image corresponding to the other size level.

[0078] Specifically, the 1×1×8 first feature sub-image is upsampled to obtain a 2×2×8 image, which is then fused with the 2×2×8 first feature sub-image. The fused image is then a 2×2×16 image, which is further upsampled to obtain a 4×4×16 image. This 4×4×16 image is then fused with the 4×4×8 first feature sub-image to obtain a 4×4×24 image. Next, the 4×4×24 image is upsampled to obtain an 8×8×24 image, which is then fused with the 8×8×8 first feature sub-image to output an 8×8×32 image.

[0079] Step 5: Upsample the 8×8×32 output image to obtain 16×16×32, and fuse it with the 16×16×32 first feature image input from the backbone feature extraction network to the feature enhancement module to obtain a 16×16×64 image.

[0080] Step 6: The 16×16×64 image output from Step 5 is first processed through a convolutional layer with a 1×1 kernel to reduce the number of channels, and then fused using a convolutional layer with a 3×3 kernel to obtain the second feature image. Using a 1×1 kernel can reduce the number of parameters for feature extraction and reduce the computational load of the model.

[0081] Step 7: The second feature image is transmitted to the backbone feature extraction network. After further top-down feature fusion and multiple feature fusion layers, the backbone feature extraction network detects text regions. Then, the information of the detected text regions is transmitted to the character recognition model, which identifies text information such as text position and semantic information.

[0082] This embodiment of the disclosure can use MobileNet as the backbone feature extraction network. A feature enhancement module is added to the backbone feature extraction network, where the convolutional layers employ depthwise separable convolutional blocks, increasing the network depth and reducing the model parameters, thereby expanding the receptive field. In the feature fusion stage, a feature pyramid model is used to improve robustness to text at different scales. The detected text regions are input into the text recognition module to obtain the recognition results. This model can improve the speed of text detection by reducing the number of parameters, while increasing the network depth also improves the detection accuracy, making it suitable for mobile application requirements.

[0083] Based on the check recognition methods for mobile devices described in the above embodiments, this disclosure also provides a check recognition device for mobile devices. The following will be combined with... Figure 6 The device is described in detail.

[0084] Figure 6 A schematic block diagram of a check recognition device 600 suitable for mobile devices according to an embodiment of the present disclosure is shown.

[0085] like Figure 6 As shown, the check recognition device 600 may include an acquisition module 610, a text region detection model 620, a text recognition model 630, and an upload module 640. The text region detection model 620 includes a backbone feature extraction network 621 and a feature enhancement module 622. The check recognition device 600 can be installed in a mobile terminal 101 and can perform the functions described above. Figures 2-5 The method described.

[0086] The acquisition module 610 is used to acquire an image of the check to be identified. In one embodiment, the acquisition module 610 may perform the aforementioned operation S210.

[0087] The text region detection model 620 is used to extract image features from the check image and detect text regions in the check image based on the image features. In one embodiment, the text region detection model 620 can perform the aforementioned operation S220.

[0088] Specifically, the backbone feature extraction network 621 is used to extract image features of the check image through multiple network layers in a concatenated manner. The feature enhancement module 622 is used to enhance the first feature image output by a predetermined network layer in the multiple network layers into a second feature image, and then transmit the second feature image back to the backbone feature extraction network for further processing.

[0089] In some embodiments, the feature enhancement module 622 is specifically used to: firstly, extract features from the first feature image at n size levels to obtain n first feature sub-images, where n is an integer greater than or equal to 2, and the n size levels include a global size level for feature extraction according to a 1x1 grid and at least one local size level for feature extraction according to a different 1x1 grid; then, fuse the n first feature sub-images to obtain a second feature sub-image; next, process the second feature sub-image according to the image parameters required for further processing by the backbone feature extraction network to obtain a second feature image.

[0090] The character recognition model 630 is used to recognize text information in a text region. In one embodiment, the character recognition model 630 can perform the aforementioned operation S230.

[0091] The upload module 640 is used to upload text information to the server. In one embodiment, the upload module 640 can perform the aforementioned operation S240.

[0092] According to embodiments of this disclosure, any multiple modules among the acquisition module 610, text region detection model 620, text recognition model 630, upload module 640, backbone feature extraction network 621, and feature enhancement module 622 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the acquisition module 610, text region detection model 620, text recognition model 630, upload module 640, backbone feature extraction network 621, and feature enhancement module 622 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the following modules can be implemented, at least partially, as a computer program module, which can perform corresponding functions when the computer program module is run.

[0093] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a check recognition method according to embodiments of the present disclosure.

[0094] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0095] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0096] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0097] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0098] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0099] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0100] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0101] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0102] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0103] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0104] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0105] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0106] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A check recognition method suitable for mobile devices, comprising: Obtain the image of the check to be identified; The text region detection model is used to extract image features from the check image, and the text regions in the check image are detected based on the image features. The text information in the text region is identified using a character recognition model; as well as Upload the text information to the server; The step of extracting image features from the check image using a text region detection model includes: extracting image features from the check image using multiple network layers in a backbone feature extraction network in a cascaded manner. Specifically, the first feature image output by a predetermined network layer in the multiple network layers is first enhanced by a feature enhancement module to become a second feature image, and then the second feature image is transmitted back to the backbone feature extraction network for further processing. The predetermined network layer is either a network layer in the middle of the multiple network layers or the last network layer in the multiple network layers. The process of enhancing the first feature image into a second feature image by the feature enhancement module includes: For the first feature image, feature extraction is performed at n size levels to obtain n first feature sub-images, where n is an integer greater than or equal to 2. The n size levels include a global size level for feature extraction based on a 1x1 grid and at least one local size level for feature extraction based on a grid different from 1x1. Feature fusion is performed on the n first feature sub-maps to obtain a second feature sub-map; and The second feature sub-image is processed according to the image parameters required for further processing by the backbone feature extraction network to obtain the second feature image; The step of extracting features from the first feature image at n size levels to obtain n first feature sub-images includes: The first feature image is pooled using n pooling layers corresponding to the n size levels, respectively; and Using n first convolutional layers, the outputs of the n pooling layers are convolutionally processed to compress the number of channels in the image, resulting in the n first feature sub-images.

2. The method according to claim 1, wherein, The kernel size of the first convolutional layer is 1*1.

3. The method according to claim 1, wherein, The step of fusing the n first feature sub-maps to obtain the second feature sub-map includes: Using the first feature sub-image corresponding to the global size level as the initial sampling object, the sampling object is continuously updated in the following manner until all n feature sub-images are fused and the sampling object is output: For the sampled object, upsampling is performed according to the image resolution corresponding to the size level adjacent to the size level of the sampled object to obtain the feature sub-image to be fused; The feature sub-image to be fused and the first feature sub-image corresponding to the size level adjacent to the size level of the sampled object are fused to obtain a new sampled object.

4. The method according to claim 3, wherein, The step of fusing the n first feature sub-maps to obtain the second feature sub-map further includes: The sampled object, after all n feature sub-images have been fused, is then fused with the first feature image to obtain the second feature sub-image.

5. The method according to claim 1, wherein, The second feature sub-image is processed according to the image parameters required for further processing by the backbone feature extraction network to obtain the second feature image, including: When the predetermined network layer is one of the multiple network layers, the second feature sub-image is convolved according to the image parameters input to the next network layer of the predetermined network layer to obtain the second feature image; and When the predetermined network layer is the last network layer among the multiple network layers, the second feature sub-image is convolved according to the image parameters output by the backbone feature extraction network to obtain the second feature image.

6. A check recognition device suitable for mobile devices, wherein, The device includes: The acquisition module is used to acquire the image of the check to be identified; A text region detection model is used to extract image features from the check image and detect text regions in the check image based on the image features; and A text recognition model is used to recognize text information within the text region; and The upload module is used to upload the text information to the server. The text region detection model includes a backbone feature extraction network and a feature enhancement module. The backbone feature extraction network comprises multiple network layers, and the feature enhancement module is positioned between two network layers or after the last network layer. The backbone feature extraction network is used to extract image features of the check image through multiple network layers in a concatenated manner. The feature enhancement module is used to: enhance the first feature image output by the predetermined network layer located upstream of the feature enhancement module into a second feature image, and then transmit the second feature image back to the backbone feature extraction network for further processing. Specifically, the feature enhancement module is used for: For the first feature image, feature extraction is performed at n size levels to obtain n first feature sub-images, where n is an integer greater than or equal to 2. The n size levels include a global size level for feature extraction based on a 1x1 grid and at least one local size level for feature extraction based on a grid different from 1x1. Feature fusion is performed on the n first feature sub-maps to obtain a second feature sub-map; and The second feature sub-image is processed according to the image parameters required for further processing by the backbone feature extraction network to obtain the second feature image; The step of extracting features from the first feature image at n size levels to obtain n first feature sub-images includes: The first feature image is pooled using n pooling layers corresponding to the n size levels, respectively; and Using n first convolutional layers, the outputs of the n pooling layers are convolutionally processed to compress the number of channels in the image, resulting in the n first feature sub-images.

7. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method of any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method of any one of claims 1 to 5.

9. A computer program product comprising computer program instructions that, when executed by a processor, implement the method of any one of claims 1 to 5.

Images