Picture recognition method and device, equipment, storage medium and program product

Abstract

The invention provides a picture recognition method which can be applied to the technical field of big data and the technical field of artificial intelligence. The image recognition method comprises the steps that based on a first image recognition model, multiple visual features of a target image are extracted, and the target image is an image of a business bill; based on a second image recognition model, extracting a plurality of layout features used for representing character layout information of the business bill in the target picture; performing feature fusion processing on the plurality of visual features and the plurality of layout features by using a pre-constructed feature fusion network to obtain a target fusion feature; and performing specific information extraction processing on the target fusion feature by using the region generation network, and outputting a picture recognition result. The invention further provides a picture recognition device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the fields of big data technology and artificial intelligence technology, and more specifically to an image recognition method, device, equipment, storage medium and program product. Background Art

[0002] The processing and management of financial bills is an important part of financial business. Since manual bill processing is inefficient and prone to errors, it cannot meet the growing business needs. Therefore, in order to improve the efficiency and accuracy of bill processing, related methods use computer vision and machine learning technology to achieve automated bill detection and recognition.

[0003] In the process of realizing the concept of the present disclosure, the inventors found that there are at least the following problems in the related technology: the related methods of automatic bill detection and recognition using computer vision and machine learning technology have the problems of low accuracy and low recognition efficiency. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a method, apparatus, device, storage medium and program product for image recognition.

[0005] According to a first aspect of the present disclosure, a method for image recognition is provided, comprising: extracting multiple visual features of a target image based on a first image recognition model, wherein the target image is an image of a business bill; extracting multiple layout features of text layout information in the target image for characterizing the business bill based on a second image recognition model; performing feature fusion processing on multiple visual features and multiple layout features using a pre-constructed feature fusion network to obtain target fusion features; performing specific information extraction processing on the target fusion features using a region generation network, and outputting an image recognition result.

[0006] According to an embodiment of the present disclosure, the feature fusion network includes: a low-level feature fusion layer, a mid-level feature fusion layer, and a high-level feature fusion layer; the multiple visual features include: low-level visual features, mid-level visual features, and high-level visual features; the multiple layout features include: low-level layout features, mid-level layout features, and high-level layout features.

[0007] According to an embodiment of the present disclosure, a pre-constructed feature fusion network is used to perform feature fusion processing on multiple visual features and multiple layout features to obtain target fusion features, including: based on a low-level feature fusion layer, a first feature fusion processing is performed on low-level visual features and low-level layout features to obtain low-level fusion features; based on a mid-level feature fusion layer, a second feature fusion processing is performed on mid-level visual features and mid-level layout features to obtain mid-level fusion features; based on a high-level feature fusion layer, a third feature fusion processing is performed on high-level visual features and high-level layout features to obtain high-level fusion features; based on low-level fusion features, mid-level fusion features, and high-level fusion features, a target fusion feature is generated.

[0008] According to an embodiment of the present disclosure, generating a target fusion feature based on low-level fusion features, intermediate fusion features, and high-level fusion features includes: determining a first weight corresponding to the low-level fusion feature, a second weight corresponding to the intermediate fusion feature, and a third weight corresponding to the high-level fusion feature; based on the first weight, the second weight, and the third weight, performing weighted sum processing on the low-level fusion feature, the intermediate fusion feature, and the high-level fusion feature to obtain an initial fusion feature; inputting the initial fusion feature into a pre-constructed feature pyramid network to output a target fusion feature.

[0009] According to an embodiment of the present disclosure, the initial fusion feature is input into a pre-constructed feature pyramid network, and the target fusion feature is output, including: performing layout feature fusion processing on the layout feature and the initial fusion feature based on the feature pyramid network, and outputting the target fusion feature.

[0010] According to an embodiment of the present disclosure, a feature pyramid network includes multiple levels; based on the feature pyramid network, layout feature fusion processing is performed on layout features and initial fusion features, and outputting target fusion features includes: inputting the initial fusion features into a pre-constructed feature pyramid network, and outputting multiple initial level features corresponding to the multiple levels; performing layout feature fusion processing on the multiple initial level features and the multiple layout features to obtain the target fusion features.

[0011] According to an embodiment of the present disclosure, a region generation network is used to perform specific information extraction processing on the target fusion feature, and outputting an image recognition result includes: generating a candidate box corresponding to the target fusion feature based on the region generation network; and generating an image recognition result based on the candidate box.

[0012] The second aspect of the present disclosure provides an image recognition device, including: a first extraction module is used to extract multiple visual features of a target image based on a first image recognition model, wherein the target image is an image of a business document; a second extraction module is used to extract multiple layout features in the target image for characterizing text layout information of the business document based on a second image recognition model; a feature fusion module is used to perform feature fusion processing on multiple visual features and multiple layout features using a pre-constructed feature fusion network to obtain a target fusion feature; a specific information extraction module is used to perform specific information extraction processing on the target fusion feature using a region generation network, and output an image recognition result.

[0013] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0014] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0015] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0016] According to the embodiments of the present disclosure, on the one hand, by extracting the visual features of the target image based on the first image recognition model, extracting the layout features based on the second image recognition model, and performing feature fusion processing on the visual features and the layout features, the text visual features and the text layout information of the target image can be comprehensively focused on, so as to more comprehensively understand the image information and realize accurate positioning and extraction of key information fields on the target image. On the other hand, the visual features and the layout features can be combined with each other through feature fusion processing, which can provide a more accurate and representative feature representation. In addition, by intelligently generating candidate frames through the region generation network, the computational overhead can be reduced, and the speed and accuracy of image recognition can be improved. By combining the feature fusion network and the region generation network, the accuracy of the image recognition results can be improved, and the efficiency of the image recognition method can be improved, so as to realize the automated process of image recognition. Therefore, the technical problems of low accuracy and low recognition efficiency of the relevant image recognition methods are at least partially solved, thereby achieving the technical effect of improving the speed and accuracy of image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0018] Figure 1 The application scenario diagram of the image recognition method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown;

[0019] Figure 2 The flowchart of the picture recognition method according to the embodiment of the present disclosure is schematically shown;

[0020] Figure 3 A schematic diagram schematically shows a method for image recognition according to an embodiment of the present disclosure;

[0021] Figure 4 A flowchart of a picture recognition method according to another embodiment of the present disclosure is schematically shown;

[0022] Figure 5 A structural block diagram of a picture recognition device according to an embodiment of the present disclosure is schematically shown; and

[0023] Figure 6 A block diagram of an electronic device suitable for implementing a picture recognition method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0025] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0026] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0027] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0028] It should be noted that the image recognition method and device disclosed herein can be used in the fields of big data technology and artificial intelligence technology, and can also be used in any field other than the fields of big data technology and artificial intelligence technology. The application field of the image recognition method and device disclosed herein is not limited.

[0029] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0030] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0031] When handling financial business, it may be necessary to read key information such as the amount, issue date, and expiration date on the bill, but the efficiency and accuracy of manually reading key information in the bill are low. With the rapid development of deep learning technology, the efficiency and accuracy of image recognition and target detection have been significantly improved. In the process of realizing the concept of the present disclosure, the inventors found that there are at least the following problems in the related technology: For the related methods of processing the bill images of financial institutions through deep learning technology, they usually only rely on the extracted image visual features for image recognition and target detection. However, due to the different layouts and structures of bills, relying solely on visual features may not accurately judge the spatial relationship and arrangement between fields, resulting in low accuracy of bill image recognition results, which may not fully meet the requirements of financial institutions for identifying bill images.

[0032] In view of this, an embodiment of the present disclosure provides an image recognition method, comprising: based on a first image recognition model, extracting multiple visual features of a target image, wherein the target image is an image of a business bill; based on a second image recognition model, extracting multiple layout features in the target image for characterizing text layout information of the business bill; using a pre-constructed feature fusion network to perform feature fusion processing on multiple visual features and multiple layout features to obtain target fusion features; using a region generation network to perform specific information extraction processing on the target fusion features, and outputting an image recognition result.

[0033] Figure 1 The application scenario diagram of the image recognition method according to an embodiment of the present disclosure is schematically shown.

[0034] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0035] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0036] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0037] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0038] It should be noted that the image recognition method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the image recognition device provided in the embodiment of the present disclosure can generally be set in the server 105. The image recognition method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the image recognition device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0039] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0040] The following will be based on Figure 1 The scene described by Figure 2~Figure 4 The image recognition method of the embodiment of the present disclosure is described in detail.

[0041] Figure 2 The flowchart of the picture recognition method according to the embodiment of the present disclosure is schematically shown.

[0042] like Figure 2 As shown, the picture recognition method of this embodiment includes operations S210 to S240.

[0043] In operation S210, a plurality of visual features of a target image are extracted based on a first image recognition model, wherein the target image is an image of a business document.

[0044] In operation S220, a plurality of layout features for representing text layout information of the business document in the target image are extracted based on the second image recognition model.

[0045] In operation S230, a pre-built feature fusion network is used to perform feature fusion processing on the multiple visual features and the multiple layout features to obtain a target fusion feature.

[0046] In operation S240, a region generation network is used to perform specific information extraction processing on the target fusion feature, and an image recognition result is output.

[0047] According to an embodiment of the present disclosure, the above-mentioned image recognition method can be applied in the scenario of identifying different types of financial bill images. Due to business needs, financial institutions may need to accurately identify key text information in financial bills, as well as the layout of key text information, wherein the layout of key text information may include the location of the key text information in the bill, for example: in the case where the financial bill is a transfer check, the key text information may include the issue date, the issuer's signature, the payee's name, etc. Among them, the layout of the key text information may include the issue date being located at the upper left of the financial bill, the issuer's signature being located at the lower left of the financial bill, and the payee's name being located at the upper right of the financial bill.

[0048] However, the key text information in different types of financial bills may be different, and the location of the key text information may also be different. Related methods usually only perform image recognition based on the text features such as the edge, texture, and shape of financial bills. However, it is difficult to accurately judge the layout relationship between the texts based on the above text features, resulting in low accuracy in identifying different types of financial bill images. Therefore, it is necessary to comprehensively consider the key text information in financial bills and the layout of the key text information to improve the accuracy of bill image recognition.

[0049] According to an embodiment of the present disclosure, in operation S210, the target image may be an image of a financial institution bill or an image of another type of bill. The target image includes at least one text area and a background area other than the text area.

[0050] According to the embodiments of the present disclosure, the bill can be photographed or scanned to obtain the original image corresponding to the bill, or the original image can be obtained by other means, which are not limited here. The original image can be preprocessed to obtain the target image. The preprocessing may include: first, reducing the noise in the original image through a denoising operation to improve the effect of the subsequent algorithm. Then adjust the original image to a suitable size to retain the important information on the bill. Then grayscale the original image to make the information in the original image clearer and facilitate subsequent processing. Finally, the original image is converted into a black and white binary image through a binarization operation to obtain a target image. The binarization operation can facilitate subsequent processing and analysis.

[0051] According to the embodiments of the present disclosure, through preprocessing, the feature information in the bill image can be made more prominent, which is helpful for performing subsequent feature extraction operations.

[0052] According to an embodiment of the present disclosure, in order to identify the text area and text features in the target image, multiple visual features of the target image can be extracted by a first image recognition model. The visual features may include at least one of the following: edge features, texture features, shape features, and character features of the target image. Specifically, the edge features may include the border of the target image, the edge between the text area and the background area, etc. The texture features may include the texture of the financial bill, such as the watermark of the financial bill. The shape features may include the shape of the text in the financial bill, the shape of the stamp, etc. The character features may include the stroke structure of the text in the financial bill, etc.

[0053] According to an embodiment of the present disclosure, the first image recognition model may include, for example, a convolutional neural network. The convolutional neural network can learn and extract visual features in an image, such as edges, textures, and shapes, by training a large amount of image data, thereby helping to automatically locate and extract key information on the bill.

[0054] Specifically, for the first image recognition model, different convolution kernels can be used for feature extraction to improve the recognition performance of the first image recognition model. A higher level of feature representation can be gradually extracted through a combination of multiple convolutional layers and pooling layers. In order to improve the expressive power of the entire convolutional neural network, an activation function can be added after the convolutional layer and the pooling layer. In addition, in the process of extracting visual features, the structure of a feature pyramid network can be used to extract visual features, specifically including: fusing visual feature maps of different resolutions to generate a feature pyramid with different size information. The feature map in this structure contains target representations of different sizes, which can better support the detection tasks of targets of different sizes. Therefore, these features can effectively express the visual information in the target image and help subsequent information fusion.

[0055] In an embodiment of the present disclosure, before obtaining the user's information, the user's consent or authorization may be obtained. For example, before operation S210, a request to obtain the user's information may be issued to the user. When the user agrees or authorizes that the user's information may be obtained, operation S210 is performed.

[0056] According to an embodiment of the present disclosure, since the layout and structure of different types of bills are different, it may not be possible to accurately determine the spatial relationship and arrangement between the fields in the bill by relying solely on visual features, resulting in inaccurate image recognition results. Therefore, in operation S220, multiple layout features of text layout information for characterizing the business bill in the target image may be extracted based on the second image recognition model. The text layout information may include the spatial relationship and arrangement between the fields in the target image.

[0057] According to an embodiment of the present disclosure, the second image recognition model may include an encoder-decoder structure based on a self-attention mechanism, which is mainly used for sequence-to-sequence learning tasks such as machine translation, speech recognition, and text generation. As a powerful sequence modeling tool widely used in natural language processing and computer vision, the second image recognition model can effectively capture long-distance dependencies in sequence data through a self-attention mechanism and encode them in an orderly manner. When processing the target image, the second image recognition model can also capture the relative position relationship between fields through a special encoding method, and can capture the relationship and arrangement between fields, and perform effective feature encoding on them, thereby accurately extracting the layout information of the target image.

[0058] According to an embodiment of the present disclosure, when extracting multiple layout features in the target image, the encoding process of the second image recognition model is similar to a natural language processing task. The second image recognition model can convert each field into an embedding vector and encode it through position encoding. Then, multiple encoder layers are used to gradually improve the model's ability to model the relative position relationship between fields, thereby helping the second image recognition model to recognize the spatial relationship and arrangement between different fields on the bill.

[0059] Specifically, the encoding process of the second image recognition model includes operations 11 to 15.

[0060] Operation 11 converts the input sequence (i.e. all fields in the target image) into an embedding vector. Each field is mapped into a d-dimensional vector space, where d is the embedding dimension. This can be achieved through an embedding layer, which can obtain the embedding vector of each word or token from a pre-trained embedding table.

[0061] Operation 12: embed the position information into the input vector to achieve position encoding. This can be achieved by adding a position encoding vector, which can represent the relative position relationship of each position in the input sequence. The position encoding vector is calculated based on the sine and cosine functions, and the relative distance between different positions can be learned.

[0062] Operation 13. The second image recognition model can be composed of multiple encoder layers, each of which contains a self-attention mechanism and a feedforward neural network. In the self-attention mechanism, each input vector calculates similarity with other input vectors and obtains weights, and then their weighted average is used as the output vector, which can help the model capture the dependencies and interactions between different tags in the input sequence.

[0063] Operation 14. By stacking multiple encoder layers together, the modeling ability of the model can be gradually improved. The output of each encoder layer can be used as the input of the next encoder layer, thereby enhancing the model's understanding and representation ability of the input sequence.

[0064] Operation 15: The output of the last encoder layer is used as the encoder output of the second image recognition model. This is a vector sequence containing all input vectors, which can be used for subsequent tasks, such as classification or generation.

[0065] According to an embodiment of the present disclosure, if an image recognition result is generated based only on visual features or layout features, only the influence of text layout information or text feature information will be considered, resulting in inaccurate image recognition results. In order to improve the accuracy of the image recognition result, in operation S230, a pre-constructed feature fusion network can be used to perform feature fusion processing on multiple visual features and multiple layout features to obtain target fusion features.

[0066] According to an embodiment of the present disclosure, in order to further improve the accuracy of the image recognition result, in operation S240, the target fusion feature can be input into the region generation network. The region generation network can generate a candidate frame based on the target fusion feature, and learn the relationship between the target fusion feature and the candidate frame to generate a more accurate candidate frame, thereby outputting a more accurate image recognition result.

[0067] According to the embodiments of the present disclosure, on the one hand, by extracting the visual features of the target image based on the first image recognition model, extracting the layout features based on the second image recognition model, and performing feature fusion processing on the visual features and the layout features, it is possible to comprehensively focus on the text visual features and text layout information of the target image, thereby more comprehensively understanding the image information and achieving accurate positioning and extraction of key information fields on the target image. On the other hand, through feature fusion processing, the visual features and layout features can be combined with each other, which can provide a more accurate and representative feature representation. In addition, by intelligently generating candidate frames through a region generation network, computational overhead can be reduced and the speed and accuracy of image recognition can be improved. Through the combination of a feature fusion network and a region generation network, the accuracy of the image recognition results can be improved, and the efficiency of the image recognition method can be improved. At the same time, the robustness of the image recognition method can be improved, and an automated process for image recognition can be achieved.

[0068] Figure 3 The following is a schematic diagram of a method for image recognition according to an embodiment of the present disclosure.

[0069] like Figure 3 As shown, the financial institution bill can be scanned or photographed to obtain the original image corresponding to the financial institution bill. The original image is preprocessed to obtain the target image, wherein the preprocessing may include denoising, resizing, graying, binarization, etc. Multiple visual features of the target image are extracted based on the first image recognition model, and multiple layout features in the target image are extracted based on the second image recognition model, and the multiple visual features and multiple layout features are subjected to feature fusion processing using a pre-constructed feature fusion network to obtain the target fusion feature. The target fusion feature is input into the region generation network, and the image recognition result is output.

[0070] According to an embodiment of the present disclosure, the feature fusion network includes: a low-level feature fusion layer, a mid-level feature fusion layer, and a high-level feature fusion layer; the multiple visual features include: low-level visual features, mid-level visual features, and high-level visual features; the multiple layout features include: low-level layout features, mid-level layout features, and high-level layout features.

[0071] According to the embodiments of the present disclosure, for visual features, when the first image recognition model is a convolutional neural network, low-level visual features can be extracted through the shallow convolution layer of the convolutional neural network, and the low-level visual features include basic visual information such as edges, textures, corners, etc. of the image. Since the shallow convolution layer is more sensitive to small-scale structures and detail changes in the image, the shallow convolution layer can extract more detailed basic visual information in the target image.

[0072] Intermediate visual features can be extracted through the intermediate layers of the convolutional neural network. Intermediate visual features can represent local information in the target image, such as small shapes or local structures in the target image.

[0073] High-level visual features can be extracted through deep convolution kernels of convolutional neural networks. High-level visual features can represent more abstract information in the target image, such as the global semantic information of the target image.

[0074] According to an embodiment of the present disclosure, with respect to layout features, preliminary, less detailed position and layout information in a target image can be extracted through a second image recognition model to obtain low-level layout features; the relative positions between fields and more complex, relatively detailed layout information can also be extracted to obtain intermediate layout features; the global layout information of the target image can also be extracted, and the global layout information may include the overall structure of the target image and the precise position of each field.

[0075] Figure 4 The following schematically shows a flowchart of a method for image recognition according to another embodiment of the present disclosure.

[0076] like Figure 4 As shown, the picture recognition method of this embodiment includes operations S410 to S440.

[0077] In operation S410, based on the low-level feature fusion layer, a first feature fusion process is performed on the low-level visual features and the low-level layout features to obtain low-level fused features.

[0078] In operation S420, based on the intermediate feature fusion layer, a second feature fusion process is performed on the intermediate visual features and the intermediate layout features to obtain intermediate fusion features.

[0079] In operation S430, based on the high-level feature fusion layer, a third feature fusion process is performed on the high-level visual features and the high-level layout features to obtain high-level fused features.

[0080] In operation S440 , a target fusion feature is generated based on the low-level fusion feature, the mid-level fusion feature, and the high-level fusion feature.

[0081] According to an embodiment of the present disclosure, in order to closely combine visual features with layout features, thereby improving the accuracy of the image recognition method, a pre-constructed feature fusion network can be used to perform feature fusion processing on multiple visual features and multiple layout features. Specifically, in operation S410, the first feature fusion processing may include splicing low-level visual features and low-level layout features to obtain low-level fusion features. Low-level visual features can characterize the basic visual information in the target image, and low-level layout features can characterize the preliminary layout information of the target image. The low-level fusion features obtained by fusing the low-level visual features and the low-level layout features can characterize the detail information of the target image.

[0082] According to an embodiment of the present disclosure, in operation S420, the intermediate visual features can represent local information of the target image, the intermediate layout features can represent preliminary position and layout information, and the intermediate fused features obtained by fusing the intermediate visual features with the intermediate layout features can represent more detailed information of the target image.

[0083] According to an embodiment of the present disclosure, in operation S430, the high-level visual features can represent more abstract information in the target image, and the high-level layout features can represent global layout information of the target image. By integrating the high-level visual features with the high-level layout features, global information of the target image can be obtained.

[0084] According to an embodiment of the present disclosure, in operation S440, a weighted sum may be performed on the low-level fusion features, the intermediate-level fusion features, and the high-level fusion features, and a target fusion feature may be obtained based on the result of the weighted sum.

[0085] According to the embodiments of the present disclosure, by fusing low-level visual features with low-level layout features, more detailed and accurate information of the target image can be obtained; by fusing intermediate visual features with intermediate layout features, more extensive information of the target image than low-level fusion features can be obtained; by fusing high-level visual features with high-level layout features, the global information of the target image can be understood, thereby achieving different levels of fusion processing from local to global, from details to the whole, so that the relevant information of the target image can be fully extracted and a more accurate image recognition result can be obtained.

[0086] According to an embodiment of the present disclosure, generating a target fusion feature based on low-level fusion features, intermediate fusion features, and high-level fusion features includes: determining a first weight corresponding to the low-level fusion feature, a second weight corresponding to the intermediate fusion feature, and a third weight corresponding to the high-level fusion feature; based on the first weight, the second weight, and the third weight, performing weighted sum processing on the low-level fusion feature, the intermediate fusion feature, and the high-level fusion feature to obtain an initial fusion feature; inputting the initial fusion feature into a pre-constructed feature pyramid network to output a target fusion feature.

[0087] According to the embodiments of the present disclosure, the low-level fusion features, the intermediate fusion features, and the high-level fusion features can be weighted and summed, and the target fusion features can be obtained based on the weighted summation result. Specifically, the first weight, the second weight, and the third weight can be respectively assigned to the low-level fusion features, the intermediate fusion features, and the high-level fusion features in advance, and the weighted summation is performed based on the first weight, the second weight, and the third weight to obtain the initial fusion feature.

[0088] According to an embodiment of the present disclosure, pre-allocating weights may be: assigning a weight to each feature channel, wherein the weight is obtained through learning to reflect the importance of each feature in the detection and recognition task. Thus, the contribution of different features can be adaptively adjusted to achieve more effective fusion.

[0089] According to the embodiments of the present disclosure, an adaptive fusion method can also be used to dynamically adjust the fusion method according to the needs and feature importance. For example, in the detection of target images, more attention is paid to the fusion of layout information; while in the recognition of target images, more attention is paid to the fusion of visual information. In this way, the flexibility and robustness of the image recognition method can be improved.

[0090] According to an embodiment of the present disclosure, in order to further capture multi-scale information in the target image, the initial fusion feature can be input into a pre-constructed feature pyramid network. Specifically, the initial fusion feature is input into the pre-constructed feature pyramid network, and the target fusion feature is output, including: based on the feature pyramid network, performing layout feature fusion processing on the layout feature and the initial fusion feature, and outputting the target fusion feature.

[0091] According to an embodiment of the present disclosure, based on a feature pyramid network, layout features and initial fusion features are subjected to layout feature fusion processing, and features at different levels can be further fused. The feature pyramid network can generate a feature representation with rich multi-scale information by fusing features of different scales through top-down paths and lateral connections. The feature pyramid network includes multiple levels, and layout features can be added to each level to achieve cross-scale feature fusion.

[0092] According to an embodiment of the present disclosure, specifically, based on a feature pyramid network, layout feature fusion processing is performed on layout features and initial fusion features, and outputting target fusion features includes: inputting the initial fusion features into a pre-constructed feature pyramid network, and outputting multiple initial level features corresponding to multiple levels; performing layout feature fusion processing on multiple initial level features and multiple layout features to obtain target fusion features.

[0093] According to an embodiment of the present disclosure, a feature pyramid network includes a bottom-up path, a top-down path, and a lateral connection. For the bottom-up path, it can extract fusion features layer by layer to obtain a series of multi-scale feature maps at different levels. For the top-down path, it adjusts the resolution of the high-level feature map by upsampling, and fuses the high-level feature map with the low-level feature map. For each level of the feature pyramid network, each level can fuse the feature map extracted by the bottom-up path with the feature map transmitted by the top-down path through a lateral connection, and at the same time, multiple layout features can also be fused with the feature map transmitted by the top-down path. Specifically, the size of multiple layout features can be adjusted by upsampling to obtain an initial layout information feature, so that the size of the initial layout feature information matches the size of the feature map generated from the bottom to the top. Then adjust the number of channels of multiple layout features to obtain a target layout information feature, so that the number of channels of the target layout information feature matches the number of channels of the feature map generated from the bottom to the top. The target layout information feature can be added element by element to the feature map generated from the bottom up, and the upsampled high-level feature map can also be added element by element to the feature map generated from the bottom up to output the target fusion feature.

[0094] According to an embodiment of the present disclosure, based on a feature pyramid network, layout feature fusion processing is performed on multiple initial level features and multiple layout features to obtain target fusion features, which can capture multi-scale information in the target image and improve the efficiency and accuracy of image recognition.

[0095] According to an embodiment of the present disclosure, a region generation network is used to perform specific information extraction processing on the target fusion feature, and outputting an image recognition result includes: generating a candidate box corresponding to the target fusion feature based on the region generation network; and generating an image recognition result based on the candidate box.

[0096] According to an embodiment of the present disclosure, the target fusion feature may include a target fusion feature map. After the target fusion feature map is input into the region generation network, the region generation network may generate multiple candidate boxes of different sizes and proportions at each position on the target fusion feature map, and each candidate box is assigned a confidence score. Candidate boxes with confidence scores greater than a preset threshold may be screened out to obtain target candidate boxes, wherein the target candidate boxes contain a target object. The target object may be text information to be extracted from the target image, such as the withdrawal date, amount, etc.

[0097] According to the embodiments of the present disclosure, compared with the related rule-based or manually defined method of generating candidate frames, the region generation network is used to intelligently generate candidate frames, which can reduce unnecessary computational overhead and improve the speed and accuracy of detection. In addition, the region generation network can better adapt to image scenes of different scales and complexities by learning the relationship between image features and candidate frames, and also generate more accurate candidate frames. By introducing the region generation network, the efficiency of image recognition can be improved, and it can better cope with bills of different types and styles.

[0098] According to an embodiment of the present disclosure, the target fusion feature is first output based on the feature fusion network, and then the target fusion feature is input into the region generation network. Through the combination of the feature fusion network and the region generation network, the multimodal features can be better fused, and a more accurate candidate frame can be generated. In addition, the efficiency of image recognition can be improved by first outputting the target fusion feature based on the feature fusion network and then inputting the target fusion feature into the region generation network. Specifically, by outputting the target fusion feature map through the feature fusion network, the subsequent generation of candidate frames can be performed directly on the target fusion feature map without having to be performed on the entire target image, thereby reducing the amount of calculation for candidate frame generation and improving the efficiency of image recognition.

[0099] According to an embodiment of the present disclosure, in order to avoid feature redundancy and computational burden, a dimensionality reduction strategy can be used to reduce the dimension of the feature during the feature fusion process. Specifically, a convolution kernel of a predetermined size can be used to reduce the number of channels of the feature, such as a 1x1 convolution kernel, and a pooling operation can be used to further reduce the spatial resolution of the feature. Through the dimensionality reduction operation, a more compact and efficient feature representation can be obtained.

[0100] According to the embodiments of the present disclosure, in order to improve the real-time processing capabilities of the first image recognition model, the second image recognition model, the feature fusion network, and the region generation network, the above models can be optimized. The optimization process can include using network pruning, quantization, and acceleration techniques to reduce the amount of computation and memory usage of the model, thereby improving the processing speed of the algorithm. Special acceleration processors such as GPUs or FPGAs can also be used in hardware to effectively improve the real-time performance of the above models.

[0101] Based on the above-mentioned image recognition method, the present disclosure also provides an image recognition device. Figure 5 The device is described in detail.

[0102] Figure 5 The structural block diagram of the image recognition device according to an embodiment of the present disclosure is schematically shown.

[0103] like Figure 5As shown, the image recognition device 500 of this embodiment includes a first extraction module 510 , a second extraction module 520 , a feature fusion module 530 , and a specific information extraction module 540 .

[0104] The first extraction module 510 is used to extract multiple visual features of a target image based on a first image recognition model, wherein the target image is an image of a business bill. In one embodiment, the first extraction module 510 can be used to perform the operation S210 described above, which will not be described in detail here.

[0105] The second extraction module 520 is used to extract multiple layout features for representing text layout information of the business bill in the target image based on the second image recognition model. In one embodiment, the second extraction module 520 can be used to perform the operation S220 described above, which will not be repeated here.

[0106] The feature fusion module 530 is used to perform feature fusion processing on multiple visual features and multiple layout features using a pre-built feature fusion network to obtain a target fusion feature. In one embodiment, the feature fusion module 530 can be used to perform the operation S230 described above, which will not be repeated here.

[0107] The specific information extraction module 540 is used to perform specific information extraction processing on the target fusion feature using the region generation network and output the image recognition result. In one embodiment, the specific information extraction module 540 can be used to perform the operation S240 described above, which will not be repeated here.

[0108] According to an embodiment of the present disclosure, the feature fusion module includes a first feature fusion submodule, a second feature fusion submodule, a third feature fusion submodule and a first generation submodule.

[0109] The first feature fusion submodule is used to perform a first feature fusion process on low-level visual features and low-level layout features based on the low-level feature fusion layer to obtain low-level fusion features; the second feature fusion submodule is used to perform a second feature fusion process on intermediate visual features and intermediate layout features based on the intermediate feature fusion layer to obtain intermediate fusion features; the third feature fusion submodule is used to perform a third feature fusion process on high-level visual features and high-level layout features based on the high-level feature fusion layer to obtain high-level fusion features; the first generation submodule is used to generate target fusion features based on low-level fusion features, intermediate fusion features, and high-level fusion features.

[0110] According to an embodiment of the present disclosure, the first generating submodule includes a determining unit, a weighted summing unit and an input unit.

[0111] The determination unit is used to determine a first weight corresponding to a low-level fusion feature, a second weight corresponding to an intermediate fusion feature, and a third weight corresponding to a high-level fusion feature; the weighted summation unit is used to perform weighted summation processing on the low-level fusion feature, the intermediate fusion feature, and the high-level fusion feature based on the first weight, the second weight, and the third weight to obtain an initial fusion feature; the input unit is used to input the initial fusion feature into a pre-constructed feature pyramid network and output a target fusion feature.

[0112] According to an embodiment of the present disclosure, the input unit includes a layout feature fusion subunit. The layout feature fusion subunit is used to perform layout feature fusion processing on the layout feature and the initial fusion feature based on the feature pyramid network, and output the target fusion feature.

[0113] According to an embodiment of the present disclosure, the layout feature fusion subunit includes an output component and a layout fusion component. The output component is used to input the initial fusion feature into a pre-built feature pyramid network and output a plurality of initial level features corresponding to a plurality of levels; the layout fusion component is used to perform layout feature fusion processing on the plurality of initial level features and the plurality of layout features to obtain the target fusion feature.

[0114] According to an embodiment of the present disclosure, the specific information extraction module includes a second generation submodule and a third generation submodule.

[0115] The second generation submodule is used to generate a candidate box corresponding to the target fusion feature based on the region generation network; the third generation submodule is used to generate an image recognition result based on the candidate box.

[0116] According to an embodiment of the present disclosure, any multiple modules of the first extraction module 510, the second extraction module 520, the feature fusion module 530, and the specific information extraction module 540 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first extraction module 510, the second extraction module 520, the feature fusion module 530, and the specific information extraction module 540 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the first extraction module 510, the second extraction module 520, the feature fusion module 530, and the specific information extraction module 540 can be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function can be performed.

[0117] Figure 6 A block diagram of an electronic device suitable for implementing a picture recognition method according to an embodiment of the present disclosure is schematically shown.

[0118] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0119] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0120] According to an embodiment of the present disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the input / output (I / O) interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0121] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0122] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0123] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program includes a program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the image recognition method provided by the embodiment of the present disclosure.

[0124] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 601. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0125] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0126] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0127] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0128] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0129] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0130] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for image recognition, characterized in that: The method comprises: Extracting multiple visual features of a target image based on a first image recognition model, wherein the target image is an image of a business bill; Extracting multiple layout features for representing text layout information of the business bill from the target image based on the second image recognition model; Using a pre-built feature fusion network to perform feature fusion processing on the multiple visual features and the multiple layout features to obtain a target fusion feature; The region generation network is used to extract specific information from the target fusion features and output the image recognition result.

2. The method according to claim 1, characterized in that: The feature fusion network includes: a low-level feature fusion layer, a mid-level feature fusion layer, and a high-level feature fusion layer; The multiple visual features include: low-level visual features, mid-level visual features and high-level visual features; The plurality of layout features include: low-level layout features, mid-level layout features and high-level layout features.

3. The method according to claim 2, characterized in that The using of the pre-built feature fusion network to perform feature fusion processing on the multiple visual features and the multiple layout features to obtain the target fusion feature includes: Based on the low-level feature fusion layer, performing a first feature fusion process on the low-level visual features and the low-level layout features to obtain low-level fusion features; Based on the intermediate feature fusion layer, performing a second feature fusion process on the intermediate visual features and the intermediate layout features to obtain intermediate fusion features; Based on the high-level feature fusion layer, performing a third feature fusion process on the high-level visual features and the high-level layout features to obtain high-level fusion features; The target fusion feature is generated based on the low-level fusion feature, the mid-level fusion feature, and the high-level fusion feature.

4. The method according to claim 3, characterized in that The generating the target fusion feature based on the low-level fusion feature, the intermediate-level fusion feature, and the high-level fusion feature comprises: Determine a first weight corresponding to the low-level fusion feature, a second weight corresponding to the mid-level fusion feature, and a third weight corresponding to the high-level fusion feature; Based on the first weight, the second weight, and the third weight, weighted summing is performed on the low-level fusion feature, the intermediate-level fusion feature, and the high-level fusion feature to obtain an initial fusion feature; The initial fusion features are input into a pre-constructed feature pyramid network, and the target fusion features are output.

5. The method according to claim 4, characterized in that Inputting the initial fusion features into a pre-constructed feature pyramid network and outputting the target fusion features comprises: Based on the feature pyramid network, layout feature fusion processing is performed on the layout features and the initial fusion features, and the target fusion features are output.

6. The method according to claim 5, characterized in that The feature pyramid network includes multiple levels; The performing layout feature fusion processing on the layout feature and the initial fusion feature based on the feature pyramid network, and outputting the target fusion feature comprises: Inputting the initial fusion features into the pre-constructed feature pyramid network, and outputting a plurality of initial level features corresponding to the plurality of levels; Performing layout feature fusion processing on the multiple initial level features and the multiple layout features to obtain the target fusion feature.

7. The method according to claim 1, characterized in that The specific information extraction process of the target fusion feature using the region generation network and the output of the image recognition result include: Based on the region generation network, generating a candidate box corresponding to the target fusion feature; Based on the candidate box, the image recognition result is generated.

8. An image recognition device, characterized in that: The device comprises: A first extraction module, configured to extract a plurality of visual features of a target image based on a first image recognition model, wherein the target image is an image of a business bill; A second extraction module, configured to extract, based on a second image recognition model, a plurality of layout features in the target image for representing text layout information of the business bill; A feature fusion module, used to perform feature fusion processing on the multiple visual features and the multiple layout features using a pre-built feature fusion network to obtain a target fusion feature; The specific information extraction module is used to use the region generation network to perform specific information extraction processing on the target fusion feature and output the image recognition result.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images