Image processing apparatus, image processing method, and computer-readable storage medium

Abstract

An image processing apparatus, an image processing method, and a computer-readable storage medium are disclosed. The image processing apparatus includes a feature extraction unit configured to extract features of an input image; a text detection unit configured to detect text in the input image based on the features extracted by the feature extraction unit; a language identification unit configured to identify a language of the text detected by the text detection unit; a text recognition unit configured to recognize the detected text based on a recognition result of the language identification unit to obtain at least one character string set; and a first classification unit configured to classify the input image by matching the at least one character string set with a predetermined character string set to obtain a first classification result representing a category of an object involved in the input image for obtaining a final classification result of the category of the object.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of image processing, and in particular, to an image processing apparatus, an image processing method, and a computer-readable storage medium. BACKGROUND

[0002] Automatic classification of images involving objects such as certificates, documents, etc. can help people to quickly extract effective information from such images, thereby saving manpower and improving efficiency. However, due to the diversity of objects such as certificates, documents, etc., it is quite challenging to automatically classify images including such objects. Therefore, there is a need for a technique that can effectively classify an input image to obtain a class of an object involved in the input image. SUMMARY

[0003] A brief summary of the present disclosure is presented in the following to provide a basic understanding of some aspects of the present disclosure. However, it should be understood that this summary is not an extensive overview of the present disclosure. It is not intended to identify key or critical elements of the present disclosure or to delineate the scope of the present disclosure. Its sole purpose is to present some concepts of the present disclosure in a simplified form as a prelude to the more detailed description that is presented later.

[0004] In view of the above problems, an object of the present disclosure is to provide an image processing apparatus, an image processing method, and a computer-readable storage medium, which at least enable effective classification of an input image to obtain a final classification result representing a class of an object involved in the input image.

[0005] According to an aspect of the present disclosure, there is provided an image processing apparatus, including: a feature extraction unit configured to extract features of an input image; a text detection unit configured to detect text in the input image based on the features extracted by the feature extraction unit; a language identification unit configured to identify a language of the text detected by the text detection unit; a text recognition unit configured to recognize the detected text based on a recognition result of the language identification unit to obtain at least one character string set; and a first classification unit configured to classify the input image by matching the at least one character string set and a predetermined character string set to obtain a first classification result representing a class of an object involved in the input image, for obtaining a final classification result of the class of the object.

[0006] According to another aspect of the present disclosure, there is provided an image processing method including: a feature extraction step for extracting features of an input image; a text detection step for detecting text in the input image based on the features extracted by the feature extraction step; a language identification step for identifying a language of the text detected by the text detection step; a text recognition step for recognizing the detected text based on a recognition result obtained by the language identification step to obtain at least one character string set; and a first classification step for classifying the input image based on the recognition result and the at least one character string set to obtain a first classification result representing a category of an object to which the input image relates, for obtaining a final classification result of the category of the object.

[0007] According to other aspects of the present disclosure, there are also provided computer program codes and computer program products for implementing the above-described methods according to the present disclosure, and a computer-readable storage medium having the computer program codes for implementing the above-described methods according to the present disclosure recorded thereon.

[0008] Other aspects of embodiments of the present disclosure are provided in the following description and related drawings wherein a preferred embodiment of a method for fully and effectively disclosing the present disclosure is set forth and described, without imposing restrictions thereof. BRIEF DESCRIPTION OF DRAWINGS

[0009] The present disclosure can be better understood with reference to the following detailed description when considered in connection with the following drawings, in which like elements are numbered alike, in which:

[0010] Figure 1 is a block diagram illustrating an example of a functional configuration of an image processing apparatus according to an embodiment of the present disclosure;

[0011] Figure 2 is a block diagram illustrating an example of a functional configuration of a language identification unit according to an embodiment of the present disclosure;

[0012] Figure 3 is a flowchart illustrating an example of a character string matching process performed by a first classification unit according to an embodiment of the present disclosure;

[0013] Figure 4 is a block diagram illustrating an example of a functional configuration of an image processing apparatus according to another embodiment of the present disclosure;

[0014] Figure 5is a block diagram showing an example of an architecture of one specific implementation of an image processing apparatus according to a further embodiment of the present disclosure;

[0015] Figures 6A to 6C is a schematic diagram showing a principle of fusion processing by a classification model based on a Graph Attention Network (GAT);

[0016] Figure 7 is a block diagram showing an example of an architecture of one specific implementation of an image processing apparatus according to a further embodiment of the present disclosure;

[0017] Figure 8 is a flowchart showing an example of a flow of an image processing method according to an embodiment of the present disclosure;

[0018] Figure 9 is a flowchart showing an example of a flow of an image processing method according to a further embodiment of the present disclosure; and

[0019] Figure 10 is a block diagram showing an example structure of a personal computer which can be employed in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] Exemplary embodiments of the present disclosure will be described hereinafter with reference to the accompanying drawings. In the description, all features that are possible to be implemented will not be described in order to make the present disclosure clear and brief. It should be appreciated, however, that many implementation-specific decisions must be made to develop any such actual implementation, to achieve the developer's specific goals, such as compliance with system- and business-related constraints, which will vary from one implementation to another. Moreover, it should be appreciated that such a development effort might be very complex and time-consuming, but would nevertheless be a routine undertaking of design, fabrication, and manufacture for those of ordinary skill having the benefit of this disclosure.

[0021] It is also to be noted that, in order to avoid obscuring the present disclosure with unnecessary detail, only the structures of the apparatus and / or the processing steps that are closely related to the solution according to the present disclosure are shown in the drawings, while other details that are less relevant to the present disclosure are omitted.

[0022] Embodiments according to the present disclosure will be described in detail below with reference to the accompanying drawings.

[0023] First, an implementation example of an image processing apparatus 100 according to an embodiment of the present disclosure will be described with reference to Figure 1 Figure 1 is a block diagram showing an example of a functional configuration of the image processing apparatus 100 according to an embodiment of the present disclosure.

[0024] As​Figure 1 As shown, the image processing apparatus 100 according to the embodiments of the present disclosure can include a feature extraction unit 102, a text detection unit 104, a language recognition unit 106, a text recognition unit 108, and a first classification unit 110.

[0025] The feature extraction unit 102 can be configured to extract features of an input image. For example, the feature extraction unit 102 can employ ResNet-50 as a feature extraction network to extract features of an input image, however the feature extraction unit 102 can also employ other suitable feature extraction networks to extract features of an input image, which will not be elaborated here.

[0026] The text detection unit 104 can be configured to detect text in an input image based on the features extracted by the feature extraction unit 102. For example, the text detection unit 104 can detect the presence and / or the location of text in an input image. For example, the text detection unit 104 can employ a text detection method based on Differentiable Binarization (e.g., see AAAI-2020, Real-time Scene Text Detection with Differentiable Binarization) to detect the location of text in an input image. The location of text can be represented by the position coordinates of a corresponding text box.

[0027] The language recognition unit 106 can be configured to recognize the language of the text detected by the text detection unit 104. For example, the language recognition unit 106 can employ a patch-based CNN method, divide each text box detected by the text detection unit 104 into multiple image patches, then respectively recognize the language of each image patch, and finally vote to obtain the final language recognition result. That is, for a certain text box, the language with the most number of corresponding image patches can be taken as the language recognition result of the text box. For example, in a certain text box is divided into 3 image patches, and 2 of the image patches are of a first language (e.g., Chinese) and 1 of the image patches is of a second language (e.g., English), the first language can be taken as the language recognition result of the text box.

[0028] The following will be described with reference to Figure 2 The functional configuration example of the language recognition unit 106 according to the embodiments of the present disclosure will be described. Figure 2 is a block diagram illustrating a functional configuration example of the language recognition unit 106 according to the embodiments of the present disclosure.

[0029] For example, as shown in Figure 2As shown, the language recognition unit 106 can include a text box dividing subunit 1062, a feature extracting subunit 1064, and a classifying subunit 1068.

[0030] The text box dividing subunit 1062 can be configured to divide each text box into a plurality of image blocks. The feature extracting subunit 1064 can be configured to extract features of each image block. The classifying subunit 1068 can be configured to classify each image block based on the features of the image block to obtain a language corresponding to each image block, thereby obtaining a language recognition result of each text box.

[0031] For example, as shown in FIG. 1, the language recognition unit 106 can be configured to recognize a language of each text line of the input image, and obtain a language recognition result of each text line. Figure 2 As shown, the language recognition unit 106 can further include a feature processing subunit 1066 configured to process the features of each image block extracted by the feature extracting subunit 1064 to obtain processed features. In this case, the classifying subunit 1068 can be configured to classify each image block based on the processed features of the image block obtained via the feature processing subunit 1066.

[0032] The text recognition unit 108 can be configured to recognize the detected text based on the recognition result of the language recognition unit 106 to obtain at least one string set. For example, the text recognition unit 108 can select, for each text line, a corresponding text recognition engine to recognize the text line based on a language involved in the text line recognized by the language recognition unit 106, to obtain a recognized string of each text line. For example, one string set can include a string obtained by recognizing one text line. In addition, for example, one string set can include strings obtained by recognizing one or more text lines in one text box.

[0033] A string can include any character, such as a Chinese character, a Japanese character, a Latin character, a symbol, etc.

[0034] The first classifying unit 110 can be configured to classify the input image by matching the at least one string set obtained via the text recognition unit 108 with a predetermined string set, to obtain a first classification result representing a category of an object involved in the input image, for obtaining a final classification result of the category of the object. For example, the object can be a document, an identification, etc., however the object is not limited thereto. In the following, the present disclosure will be mainly described in combination with a case where the object is an identification.

[0035] Identity cards such as identity documents are indispensable in human life. Identity documents have various types, such as personal identity cards, driver's licenses, passports, residence cards, and the like. Personal identity documents are required in multiple industries such as hotels, security, banks, car rentals, and the like to confirm identity and provide services. Automatic classification of identity documents can help people quickly extract valid information from images including identity documents, thereby saving manpower and improving efficiency. However, due to the inconsistency and diversity of identity documents in different countries and regions, it is very challenging to automatically classify these identity documents.

[0036] Existing identity document classification methods can be generally divided into two categories: visual-based methods and text-based methods. The visual-based method mainly extracts visual features from the identity document image and classifies them. This method can lead to misclassification due to the similarity of some identity documents. In addition, if there are multilingual texts in the identity document image, the existing text-based classification method is not applicable because the identity documents come from different countries and / or regions.

[0037] As described above, the image processing apparatus 100 according to the embodiments of the present disclosure identifies the text in the input image based on the language recognition result to obtain at least one string set, and further classifies the input image by matching the obtained at least one string set with a predetermined string set. Thus, the image processing apparatus 100 according to the embodiments of the present disclosure can classify the input image (e.g., an identity document image, a document image, and the like) containing text in multiple languages.

[0038] In addition, the units included in the image processing apparatus 100 according to the embodiments of the present disclosure are decoupled, thus facilitating the debugging of the performance of each unit. Furthermore, the image processing apparatus 100 according to the embodiments of the present disclosure is easy to extend the types of objects (e.g., documents, identity documents, and the like) related to images. In addition, the image processing apparatus 100 according to the embodiments of the present disclosure can be applied to various scenarios of images including text.

[0039] In addition, the image processing apparatus 100 according to the embodiments of the present disclosure classifies the input image based on string matching, thus it can classify the object based only on the strings in the input image, and is easy to extend the object types (e.g., identity document types (such as personal identity cards, driver's licenses, passports, residence cards, and the like), document types).

[0040] For example, according to the embodiments of the present disclosure, the predetermined string set can be selected based on the recognition result of the language recognition unit 106. For example, the strings of one or more languages recognized by the language recognition unit 106 can be selected from the string set representing the category of the object as the predetermined string set, so as to reduce the time required for string matching.

[0041] For example, according to an embodiment of the present disclosure, the first classification unit 110 can be configured to classify the input image by matching a subset of the at least one set of strings corresponding to a predetermined range of positions in the input image and the predetermined set of strings, so that the time taken for string matching can be further reduced. For example, in the case where the input image is a document image, considering that the keywords in the document are mainly concentrated in the positions of the document close to the upper and lower edges of the document, the first classification unit 110 can match only a subset of the at least one set of strings corresponding to a range of positions close to the upper and lower edges of the image and the predetermined set of strings. For example, in the case where the input image is a document image, the first classification unit 110 can match only a subset of the at least one set of strings corresponding to a range of positions of 0≤y

[0042] For example, according to an embodiment of the present disclosure, the first classification unit 110 can be configured to match the set of strings to be matched with a first subset of the predetermined set of strings to obtain a first matching result, match the set of strings to be matched with a second subset of the predetermined set of strings based on the first matching result to obtain a second matching result, and classify the input image based on the first matching result and the second matching result, so that the time required for string matching can be further reduced. The set of strings to be matched can be the at least one set of strings obtained by the text recognition unit 108 or a subset of the at least one set of strings corresponding to a predetermined range of positions.

[0043] As an example, the category of the object to which the input image relates can be represented by the country or region to which the object corresponds and the object type of the object. In this case, the first subset can include strings representing the names of a plurality of countries and / or regions and the second subset can include strings representing a plurality of object types, or the first subset can include strings representing a plurality of object types and the second subset can include strings representing the names of a plurality of countries and / or regions.

[0044] For example, the region can be a region included in some countries, such as different states of the United States. In addition, the region can be a region including more than one country, such as Europe.

[0045] As another example, the category of the object can be represented only by the object type of the object. For example, in the case where the country or region to which the object to be classified relates is known, the category of the object can be represented only by the object type of the object, such as a document type.

[0046] The string matching processing performed by the first classification unit will be described below with reference to a specific example in which the input image is a certificate image. Figure 3 The string matching processing performed by the first classification unit will be described below with reference to a specific example in which the input image is a certificate image. Figure 3 is a flowchart showing a flow example of the string matching processing performed by the first classification unit 110 according to an embodiment of the present disclosure.

[0047] As shown in Figure 3 , the string matching processing starts at step S301. In step S302, the first classification unit 110 can match the set of strings to be matched with a first subset by a method such as dynamic programming to obtain a recall score. For example, in the example shown in Figure 3 , the first subset can include keywords of names of all countries and / or regions. For example, the first subset can include keywords of names of all countries and / or regions expressed by various languages, or the first subset can include only keywords of names of all countries and / or regions expressed by the language identified by the language identification unit 106.

[0048] In step S304, the first classification unit 110 can select a country or region corresponding to the k1 (k1 is a natural number and greater than 0) largest recall scores as a country / region candidate, and then the processing proceeds to step S306.

[0049] In step S306, the first classification unit 110 can select, as a second subset, keywords of a certificate type included in the country / region candidate obtained in step S304 from among a predetermined set of strings, and then the processing proceeds to step S308.

[0050] In step S308, the first classification unit 110 can match the set of strings to be matched with the second subset obtained in step S306 by a method such as dynamic programming to obtain a recall score, and then the processing proceeds to step S310.

[0051] In step S310, the first classification unit 110 can select a certificate type corresponding to the k2 (k2 is a natural number and greater than 0) largest recall scores as an object type candidate, and then the processing proceeds to step S312.

[0052] In step S312, the country / region candidates obtained in step S304 and the object type candidates obtained in step S310 can be combined. For each combination, the recall score of the combination is obtained based on the recall score of the country / region candidates (the recall score obtained in step S302) and the recall score of the object type candidates (the recall score obtained in step S308). The combinations corresponding to the top k3 (k3 is a natural number and greater than 0) largest recall scores are selected as combination candidates. Then, the process can end (step S313).

[0053] For example, the candidate combination with the highest recall score can be used as the first classification result.

[0054] For example, for each combination, the average of the recall scores of the country / region candidates (the recall score obtained in step S302) and the recall scores of the object type candidates (the recall score obtained in step S308) in that combination can be used as the recall score of that combination.

[0055] In addition, for example, thresholds for recall scores can be set for country / region candidates and object type candidates respectively, and only country / region candidates and object type candidates with recall scores greater than the corresponding thresholds can be selected.

[0056] As described above, the first classification unit 110 according to the embodiments of this disclosure can select a second subset based on a first matching result obtained by matching the set of strings to be matched with a first subset, thereby further reducing the time required for string matching.

[0057] Note that, although the above references Figure 3 Examples are described where the first subset includes strings representing names of multiple countries and / or regions, and the second subset includes strings representing multiple object types. However, the first subset may include strings representing multiple object types, and the second subset may include strings representing names of multiple countries and / or regions. In this case, the first classification unit 110 may select keywords including the corresponding object types' countries and / or regions as the second subset based on the first matching result.

[0058] The above has already referred to Figures 1 to 3 An implementation example of the image processing apparatus 100 according to an embodiment of the present disclosure is described below, with reference to... Figure 4 Figures 6 to 6 illustrate implementation examples of an image processing apparatus 400 according to an embodiment of the present disclosure.

[0059] Figure 4 This is a block diagram illustrating an example of the functional configuration of an image processing apparatus 400 according to another embodiment of the present disclosure. Figure 5is a block diagram showing an architecture example of one specific implementation of an image processing apparatus according to a further embodiment of the present disclosure.

[0060] As Figure 4 and Figure 5 shown, the image processing apparatus 400 according to an embodiment of the present disclosure can include a feature extraction unit 402, a text detection unit 404, a language identification unit 406, a text recognition unit 408, a first classification unit 410, a feature conversion unit 412, a visual feature extraction unit 414, a text feature extraction unit 416, and a second classification unit 418. The specific configurations of the feature extraction unit 402, the text detection unit 404, the language identification unit 406, and the text recognition unit 408 can be similar to those of the feature extraction unit 102, the text detection unit 104, the language identification unit 106, and the text recognition unit 108 of the image processing apparatus 100 described above with reference to FIG. 1, and thus will not be repeated here. Figures 1 to 3

[0061] The feature conversion unit 412 can be configured to perform conversion processing on the features extracted by the feature extraction unit 402 to obtain converted features. For example, the feature conversion unit 412 can perform image embedding processing.

[0062] The visual feature extraction unit 414 can be configured to extract, based on the features extracted by the feature extraction unit 402, visual features (which can also be referred to as ROI features) of a region of the input image corresponding to the text detected by the text detection unit 404.

[0063] The text feature extraction unit 416 can be configured to extract, based on the at least one set of strings obtained by the text recognition unit 408, text features of the input image in a case where the at least one set of strings only involves characters of a predetermined language. For example, the text feature extraction unit 416 can encode the at least one set of strings according to a predetermined encoding rule, and the obtained result is used as the text features of the input image. For example, the characters of the predetermined language can be Latin characters.

[0064] Note that, in this context, the at least one set of strings only involving characters of a predetermined language does not mean that the at least one set of strings only involves the predetermined language, but means that the at least one set of strings can involve one or more languages that the characters of the predetermined language can represent. For example, in the example case where the characters of the predetermined language are Latin characters, the at least one set of strings can involve one or more languages represented by the Latin characters, such as Latin, English, etc.

[0065] ​The second classification unit 418 can be configured to, in a case where the at least one set of strings obtained by the text recognition unit 408 involves only characters of a predetermined language, classify the input image based on the converted features obtained by the feature conversion unit 412, the visual features extracted by the visual feature extraction unit 414, and the text features extracted by the text feature extraction unit 416, using a pre-trained classification model, to obtain a second classification result representing a class of an object involved in the input image.

[0066] For example, the pre-trained classification model can be a classification model based on a Graph Attention Network (GAT).

[0067] For example, in a case where the second classification result satisfies a predetermined condition, the second classification result can be used as a final classification result of the class of the object involved in the input image. For example, the predetermined condition can be that a classification accuracy of the second classification unit 418 is greater than or equal to a predetermined threshold. The classification accuracy can be represented by, for example, a classification score.

[0068] For example, the first classification unit 410 can be configured to, in a case where the at least one set of strings involves only characters of the predetermined language, classify the input image only in a case where the second classification result does not satisfy the predetermined condition, to obtain the first classification result as the final classification result of the type of the object involved in the input image. For example, in a case where the at least one set of strings involves only characters of the predetermined language, the first classification unit 410 can classify the input image only in a case where the classification accuracy of the second classification unit 418 is less than the predetermined threshold or the second classification result indicates that the input image involves a negative sample, to obtain the first classification result as the final classification result.

[0069] As described above, the image processing apparatus 400 according to the further embodiment of the present disclosure can obtain the second classification result by the second classification unit 418 in a case where the at least one set of strings involves only characters of the predetermined language, and in a case where the second classification result satisfies the predetermined condition, the second classification result can be used as the final classification result. Further, in a case where the second classification result does not satisfy the predetermined condition, the first classification result can be obtained by the first classification unit 410 as the final classification result. Since the first classification unit 410 is easy to expand in the type of the object and the classification speed of the second classification unit 418 is fast, the image processing apparatus 400 including both the first classification unit 410 and the second classification unit 418 can support more types of objects while ensuring the classification speed and the classification accuracy.

[0070] For example, according to an embodiment of the present disclosure, as Figure 5As shown, the second classification unit 418 can include a feature fusion subunit 4182 and a classification subunit 4184.

[0071] The feature fusion subunit 4182 can be configured to fuse the visual features extracted by the visual feature extraction unit 414 and the text features extracted by the text feature extraction unit 416 by using a feature fusion layer of a pre-trained classification model, to obtain a global graph network feature of the input image.

[0072] The classification subunit 4184 can be configured to classify the input image by using the converted features obtained by the feature conversion unit 412 and the global graph network feature obtained by the feature fusion subunit 4182, to obtain a second classification result representing a category of an object involved in the input image.

[0073] Reference will be made below to Figures 6A to 6C The principle of the fusion processing performed by the feature fusion subunit 4182 by using the pre-trained classification model (i.e., the GAT-based classification model) will be described. Figures 6A to 6C is a schematic diagram illustrating the principle of the fusion processing performed by using the GAT-based classification model.

[0074] As shown in Figure 6B , the GAT can include a plurality of nodes. In the present embodiment, the feature of each node (also referred to as “node feature”) represents the feature of each text block, which includes two parts, i.e., a region of interest (ROI) feature and a corresponding text feature, as shown in Figure 6A . For example, a vector obtained by combining a vector r representing the ROI feature and a vector t representing the corresponding text feature (i.e., (r, t) or (t, r)) can be taken as the feature vector of the corresponding node. That is, the number of elements contained in the feature vector of the node is the sum of the number of elements of the vector of the corresponding ROI feature and the vector of the corresponding text feature.

[0075] According to the structure of the GAT, each node can be updated by adding the information of the neighbor nodes. Specifically, the node h i to be updated can be updated to h i .

[0076]

[0077] where N i is the number of neighbor nodes h i of the node h j to be updated. α ij represents the number of neighbor nodes h i and h jThe correlation between them. W is the weight of GAT, which reflects the linear transformation in GAT, and σ is the activation function. During training, α ij Both W and W will change.

[0078] like Figure 6C As shown, the node h1 to be updated has five neighboring nodes h2 to h6, and the correlation between them and node h1 is given by α. 12 To α 16 This indicates that the autocorrelation of node h1 is determined by α. 11 The updated node h1' can be obtained through equation (1) above. GAT can reach a stable state after several iterations. For example, the features of each node after reaching a stable state can be used as global graph network features.

[0079] In a typical GAT, α ij The value is either 0 or 1. α ij A value of 0 indicates that node h i and h j Not connected, and α ij A value of 1 indicates that node h i and h j Interconnected. To consider the characteristics of edges in GAT, nodes can be updated using EGNN-based GAT (i.e., EGNN(A)). In this case, α in equation (1) can be... ij e is expressed by the following equation (2) ij Replacement.

[0080]

[0081] In equation (2), x ij Represents node h i and h j The distance of the corresponding text block in the x-direction (i.e., the width direction of the input image), y ij Represents node h i and h j The distance of the corresponding text block in the y-direction (i.e., the height direction of the input image), where L and G represent the width and height of the input image, respectively. i and g i Representing node h respectively i The width and height of the corresponding text block, l j and g j Representing node h respectively j The width and height of the corresponding text block.

[0082] The EGNN(A) aggregates the features of the nodes and the features of the corresponding edges, and thus the global graph network features obtained therefrom help to further improve the classification accuracy. For details of the principle of the EGNN, please refer to Exploiting Edge Features for Graph Neural Networks, CVPR 2019.

[0083] As described above, the image processing apparatus according to the embodiments of the present disclosure fuses the visual features and the text features of the input image to obtain the global network features, and further classifies the input image based on the converted features and the global network features, so that the classification accuracy can be further improved.

[0084] For example, according to the embodiments of the present disclosure, as shown in Figure 7 The classification subunit 4184 can include a first classification part 4184a and a second classification part 4184b.

[0085] The first classification part 4184a can be configured to classify the input image by using the converted features obtained by the feature conversion subunit 412 and the global graph network features obtained by the feature fusion subunit 4182, to obtain the first classification sub-result.

[0086] The second classification part 4184b can be configured to classify the input image by using the converted features obtained by the feature conversion subunit 412 and the global graph network features obtained by the feature fusion subunit 4182, to obtain the second classification sub-result.

[0087] For example, the first classification part 4184a and the second classification part 4184b can share the remaining layers of the pre-trained classification model except for the last layer (e.g., the fully connected layer).

[0088] As an example, the first classification sub-result can be related to the country or region corresponding to the object involved in the input image, and the second classification sub-result can be related to the object type of the object. For example, in the example shown in Figure 7 , the first classification sub-result shows that the country corresponding to the object is Bangladesh, and the second classification sub-result shows that the object type is an ID card (ID Card).

[0089] Note that in Figure 7 , only part of the functional blocks of the image processing apparatus 400 are shown for convenience of illustration. In addition, as can be understood by those skilled in the art, the extracted features and the like shown in Figure 7 are only examples and are not limiting.

[0090] In addition, Figure 7The English words "USA", "Italy", "Bangladesh", "DE", "ID Card", "DL", and "PERMISO" in the above description represent the United States, Italy, Bangladesh, Germany, an ID card, a driver's license, and a permit, respectively.

[0091] As another example, the first classification sub-result can relate to an object type of the object, and the second classification sub-result can relate to a country or region to which the object involved in the input image corresponds.

[0092] As described above, the image processing apparatus 400 according to the embodiment of the present disclosure can include the first classification section 4184a to acquire the first classification sub-result and the second classification section 4184b to acquire the second classification sub-result, so that the classification accuracy can be further improved.

[0093] For example, according to the embodiment of the present disclosure, the pre-trained classification model is obtained by training an initial classification model using a set of sample images in the following manner: extracting features of the sample images; performing conversion processing on the extracted features to acquire converted features; detecting text in the sample images based on the extracted features; identifying a language of the detected text; identifying the detected text based on a result of the language identification to obtain at least one string set; extracting visual features of a region of the sample images corresponding to the detected text based on the extracted features; extracting text features of the sample images based on the at least one string set; fusing the visual features of the sample images and the text features of the sample images using a feature fusion layer of the initial classification model to obtain global graph network features of the sample images; and training the initial classification model based on a first loss sub-function and a second loss sub-function for the initial classification model using the converted features of the sample images and the global graph network features of the sample images to satisfy a predetermined convergence condition, thereby obtaining the pre-trained classification model. For example, the initial classification model can be trained by a multi-branch training method similar to Faster R-CNN (Faster Region-based Convolutional Neural Network) (see, for example, Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, 2017).

[0094] For example, the first loss sub-function can relate to the object type, and the second loss sub-function can relate to the country and / or the region.

[0095] For example, the predetermined convergence condition can be that the training reaches a predetermined number of times, the function values of the first loss sub-function and the second loss sub-function are respectively less than corresponding predetermined threshold values, or a weighted sum of the value of the first loss function and the value of the second loss function is less than a predetermined threshold value.

[0096] The image processing apparatus 100 according to the embodiments of the present disclosure has been described above, and the present disclosure also provides the following embodiments of the image processing method corresponding to the above-mentioned embodiments of the image processing apparatus 100.

[0097] Figure 8 is a flowchart showing a flow example of an image processing method 800 according to an embodiment of the present disclosure. As shown, the image processing method 800 according to an embodiment of the present disclosure can start at a start step S802 and end at an end step S826. The image processing method 800 can include a feature extraction step S804, a text detection step S806, a language identification step S808, a text recognition step S810, and a first classification step S814. Figure 8

[0098] In the feature extraction step S804, the features of the input image can be extracted. For example, the feature extraction step S804 can be implemented by the feature extraction unit 102 described above with reference to FIG. 1, and thus specific details can be referred to the description of the feature extraction unit 102 above, which will not be repeated here. Figures 1 to 3

[0099] In the text detection step S806, the text in the input image can be detected based on the features extracted by the feature extraction step S804. For example, in the text detection step S806, the presence or absence of text and / or the position of the text in the input image can be detected. For example, the text detection step S806 can be implemented by the text detection unit 104 described above with reference to FIG. 1, and thus specific details can be referred to the description of the text detection unit 104 above, which will not be repeated here. Figures 1 to 3

[0100] In the language identification step S808, the language of the text detected by the text detection step S806 can be identified. For example, the language identification step S808 can be implemented by the language identification unit 106 described above with reference to FIG. 1, and thus specific details can be referred to the description of the language identification unit 106 above, which will not be repeated here. Figures 1 to 3

[0101] In the text recognition step S810, the detected text can be recognized based on the recognition result of the language identification step S808 to obtain at least one character string set. For example, the text recognition step S810 can be implemented by the text recognition unit 108 described above with reference to FIG. 1, and thus specific details can be referred to the description of the text recognition unit 108 above, which will not be repeated here. Figures 1 to 3 ​​​​The described text recognition unit 108 is implemented as such, and specific details can be referred to the description of the text recognition unit 108 above, which will not be repeated here.

[0102] In the first classification step S814, the input image can be classified by matching the at least one string set obtained via the text recognition step S810 and the predetermined string set to obtain a first classification result representing a class of the object involved in the input image for obtaining the final classification result of the class of the object. For example, the first classification step S814 can be implemented by the first classification unit 110 described above with reference to FIG. 1, and specific details can be referred to the description of the first classification unit 110 above, which will not be repeated here. Figures 1 to 3 The described first classification unit 110 is implemented as such, and specific details can be referred to the description of the first classification unit 110 above, which will not be repeated here.

[0103] As described above, similarly to the image processing apparatus 100 according to the embodiments of the present disclosure, the image processing method 800 according to the embodiments of the present disclosure can recognize text in an input image based on the language recognition result to obtain at least one string set, and further classify the input image by matching the obtained at least one string set and a predetermined string set, and thus can classify an input image (e.g., a certificate image, a document image, etc.) containing text in multiple languages.

[0104] In addition, the image processing method 800 according to the embodiments of the present disclosure is easy to expand the types of objects (e.g., documents, certificates, etc.) involved in the image. In addition, the image processing method 800 according to the embodiments of the present disclosure can be applied to various scenarios of images including text.

[0105] In addition, the image processing method 800 according to the embodiments of the present disclosure classifies the input image based on string matching, and thus can classify the object based on only the strings in the input image and is easy to expand the object type.

[0106] For example, according to the embodiments of the present disclosure, the predetermined string set can be selected based on the recognition result of the language recognition step S808. For example, the strings of one or more languages recognized in the language recognition step S808 can be selected from the string set representing the class of the object as the predetermined string set, so that the time required for string matching can be reduced.

[0107] For example, according to the embodiments of the present disclosure, in the first classification step S814, the input image can be configured to be classified by matching the subset of strings in the at least one string set described above corresponding to the predetermined position range in the input image and the predetermined string set, so that the time spent on character matching can be further reduced.

[0108] For example, according to an embodiment of this disclosure, in the first classification step S814, the set of strings to be matched can be matched with a first subset of a predetermined set of strings to obtain a first matching result; the set of strings to be matched can be matched with a second subset of the predetermined set of strings based on the first matching result to obtain a second matching result; and the input image can be classified based on the first matching result and the second matching result, thereby further reducing the time required for string matching. The set of strings to be matched can be at least one of the above-mentioned string sets obtained through the text recognition step S810, or a set of strings from the at least one string set that corresponds to a predetermined position range.

[0109] As an example, the category of objects involved in the input image can be represented by the country or region corresponding to the object and the object type of the object. In this case, the first subset may include strings representing the names of multiple countries and / or regions and the second subset may include strings representing multiple object types, or the first subset may include strings representing multiple object types and the second subset may include strings representing the names of multiple countries and / or regions.

[0110] As another example, the category of an object can be represented solely by its object type. For instance, if the country or region corresponding to the objects in the input image to be classified is known, the category of the object can be represented solely by its object type.

[0111] The above has already referred to Figure 8 An implementation example of the image processing method 800 according to an embodiment of the present disclosure is described below, with reference to... Figure 9 An implementation example of an image processing method 900 according to another embodiment of the present disclosure is described.

[0112] Figure 9 This is a flowchart illustrating a process example of an image processing method 900 according to another embodiment of the present disclosure.

[0113] like Figure 9 As shown, the image processing method 900 according to an embodiment of the present disclosure may begin at a start step S902 and end at an end step S926. The image processing method 900 may include a feature extraction step S904, a text detection step S906, a language recognition step S908, a text recognition step S910, a string set determination step S912, a first classification step S914, a feature transformation step S916, a visual feature extraction step S918, a text feature extraction step S920, a second classification step S922, and a second classification result determination step S924. The feature extraction step S904, text detection step S906, language recognition step S908, and text recognition step S910 are consistent with those described above.Figure 8 The described feature extraction step S804, text detection step S806, language recognition step S808, and text recognition step S810 are similar, and thus will not be repeated here.

[0114] In the character string set determination step S912, it can be determined whether the at least one character string set obtained by the text recognition step S910 involves only characters of the predetermined language. In a case where it is determined that the at least one character string set does not involve only characters of the predetermined language (step S912, No), the process proceeds to a first classification step S914.

[0115] In the first classification step S914, the input image can be classified by matching the at least one character string set obtained by the text recognition step S910 with a predetermined character string set, to obtain a first classification result indicating a class of an object involved in the input image as a final classification result. For example, the first classification step S914 can be implemented by the first classification unit 410 described above with reference to FIG. 4, and thus specific details can be found in the description of the first classification unit 410 above, which will not be repeated here. Figures 4 to 7

[0116] On the other hand, in a case where it is determined that the at least one character string set involves only characters of the predetermined language (step S912, Yes), the process proceeds to a feature conversion step S916.

[0117] In the feature conversion step S916, a conversion process can be performed on the features extracted by the feature extraction step S904 to obtain converted features. For example, the feature conversion step S916 can be implemented by the feature conversion unit 412 described above with reference to FIG. 4, and thus specific details can be found in the description of the feature conversion unit 412 above, which will not be repeated here. Figures 4 to 7

[0118] In the visual feature extraction step S918, visual features (which can also be referred to as ROI features) of a region of the input image corresponding to the text detected by the text detection step S906 can be extracted based on the features extracted by the feature extraction step S904. For example, the visual feature extraction step S918 can be implemented by the visual feature extraction unit 414 described above with reference to FIG. 4, and thus specific details can be found in the description of the visual feature extraction unit 414 above, which will not be repeated here. Figures 4 to 7

[0119] In the text feature extraction step S920, text features of the input image can be extracted based on the at least one character string set obtained by the text recognition step S810. The text feature extraction step S920 can be implemented by the text feature extraction unit 416 described above with reference to FIG. 4, and thus specific details can be found in the description of the text feature extraction unit 416 above, which will not be repeated here. Figures 4 to 7 ​​​The described text feature extraction unit 416 is implemented, and thus specific details can be referred to the description of the text feature extraction unit 416 above, which will not be repeated here.

[0120] In the second classification step S922, the input image can be classified based on the converted features obtained through the feature conversion step S916, the visual features extracted through the visual feature extraction step S918, and the text features extracted through the text feature extraction step S920, by using the pre-trained classification model, to obtain a second classification result representing the class of the object involved in the input image. The second classification step S922 can be implemented by the second classification unit 418 described above, and thus specific details can be referred to the description of the second classification unit 418 above, which will not be repeated here. Figures 4 to 7 The described second classification unit 418 is implemented, and thus specific details can be referred to the description of the second classification unit 418 above, which will not be repeated here.

[0121] In the second classification result determination step S924, it can be determined whether the second classification result satisfies a predetermined condition. In the case where it is determined that the second classification result satisfies the predetermined condition (step S924, “Yes”), the second classification result can be output as a final classification result, and the processing ends.

[0122] On the other hand, in the case where it is determined that the second classification result does not satisfy the predetermined condition (step S924, “No”), the processing proceeds to the first classification step S914.

[0123] As described above, similarly to the image processing apparatus 400 according to the further embodiment of the present disclosure, the image processing method 900 according to the further embodiment of the present disclosure can obtain the second classification result through the second classification step S922 in the case where the above-described at least one set of strings involves only characters of a predetermined language, and in the case where the second classification result satisfies the predetermined condition, the second classification result can be used as the final classification result. Further, in the case where the second classification result does not satisfy the predetermined condition, the first classification result can be obtained as the final classification result through the first classification step S914. Thus, the image processing method 900 according to the further embodiment of the present disclosure can support more object types while ensuring the classification speed and the classification accuracy.

[0124] For example, according to the embodiment of the present disclosure, the second classification unit 418 can include a feature fusion sub-step and a classification sub-step Figure 9 (not shown).

[0125] In the feature fusion sub-step, the visual features extracted through the visual feature extraction step S918 and the text features extracted through the text feature extraction step S920 can be fused by using a feature fusion layer of the pre-trained classification model, to obtain a global graph network feature of the input image. For example, the feature fusion sub-step can be implemented by the feature fusion unit 412 described above with reference toFigure 5 The described feature fusion sub-unit 4182 is implemented, and thus specific details can be referred to the above description of the feature fusion sub-unit 4182, which will not be repeated here.

[0126] In the classification sub-step, the input image can be classified by using the converted features obtained through the feature conversion step S916 and the global graph network features obtained through the feature fusion sub-step, to obtain a second classification result representing a class of the object involved in the input image. For example, the classification sub-step can be implemented by the described classification sub-unit 4184, and thus specific details can be referred to the above description of the classification sub-unit 4184, which will not be repeated here. Figure 5 The described classification sub-unit 4184 is implemented, and thus specific details can be referred to the above description of the classification sub-unit 4184, which will not be repeated here.

[0127] As described above, the image processing method 900 according to the embodiments of the present disclosure fuses the visual features and the text features of the input image to obtain the global network features, and further classifies the input image based on the converted features and the global network features, so that the classification accuracy can be further improved.

[0128] For example, according to the embodiments of the present disclosure, the classification sub-step can include a first classification sub-step and a second classification sub-step Figure 9 (not shown).

[0129] In the first classification sub-step, the input image can be classified by using the converted features obtained through the feature conversion step S916 and the global graph network features obtained through the feature fusion sub-step, to obtain the first classification sub-result.

[0130] In the second classification sub-step, the input image can be classified by using the converted features obtained through the feature conversion step S916 and the global graph network features obtained through the feature fusion sub-step, to obtain the second classification sub-result.

[0131] As described above, the image processing method 900 according to the embodiments of the present disclosure can include the first classification sub-step to obtain the first classification sub-result and the second classification sub-step to obtain the second classification sub-result, so that the classification accuracy can be further improved.

[0132] It should be noted that although the functional configuration and operation of the image processing apparatus and the image processing method according to the embodiments of the present disclosure are described above, this is only an example and not a limitation, and those skilled in the art can modify the above embodiments according to the principles of the present disclosure, for example, the functional modules and operations in each embodiment can be added, deleted or combined, etc., and such modifications all fall within the scope of the present disclosure.

[0133] It should also be noted that the method embodiments herein are corresponding to the above-described device embodiments, and thus the contents not described in detail in the method embodiments can be referred to the description of the corresponding part in the device embodiments, which will not be repeated here.

[0134] In addition, the present disclosure also provides a storage medium and a program product. It should be understood that the machine-executable instructions in the storage medium and the program product according to the embodiments of the present disclosure can also be configured to execute the above-mentioned image processing method, and thus the contents not described in detail herein can be referred to the description of the corresponding part previously, which will not be repeated here.

[0135] Accordingly, the storage medium for carrying the above-mentioned program product including machine-executable instructions is also included in the disclosure of the present application. The storage medium includes but is not limited to floppy disks, optical disks, magneto-optical disks, memory cards, memory sticks, and the like.

[0136] In addition, it should also be noted that the above-mentioned series of processes and devices can also be implemented by software and / or firmware. In the case of implementation by software and / or firmware, the computer having a dedicated hardware structure is loaded with the program constituting the software from the storage medium or the network, and the computer is capable of performing various functions and the like when various programs are installed. Figure 10 The general-purpose personal computer 700 shown is installed with programs constituting the software, and the computer is capable of performing various functions and the like when various programs are installed.

[0137] In Figure 10 the central processing unit (CPU) 701 performs various processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage section 708 to the random access memory (RAM) 703. In the RAM 703, data required when the CPU 701 performs various processes and the like is also stored as necessary.

[0138] The CPU 701, the ROM 702, and the RAM 703 are connected to each other via the bus 704. The input / output interface 705 is also connected to the bus 704.

[0139] The following components are connected to the input / output interface 705: the input section 706 including a keyboard, a mouse, and the like; the output section 707 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker and the like; the storage section 708 including a hard disk and the like; and the communication section 709 including a network interface card such as a LAN card, a modem, and the like. The communication section 709 performs communication processing via a network such as the Internet.

[0140] The drive 710 is also connected to the input / output interface 705 as necessary. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is installed in the drive 710 as necessary, and a computer program read therefrom is installed in the storage section 708 as necessary.

[0141] In a case where the above series of processes are implemented by software, a program constituting the software is installed from a network such as the Internet or a storage medium such as the detachable medium 711.

[0142] Those skilled in the art will appreciate that such storage media are not limited to the above examples. Figure 10 The detachable medium 711 shown is one in which a program is stored, and is distributed separately from the apparatus to provide the program to the user. Examples of the detachable medium 711 include a magnetic disk (including a floppy disk (registered trademark)), a magneto-optical disk (including a mini disk (MD) (registered trademark)), and a semiconductor memory. Alternatively, the storage medium can be the ROM 702, a hard disk included in the storage section 708, or the like, in which a program is stored, and which is distributed to the user together with the apparatus including them.

[0143] The preferred embodiments of the present disclosure have been described above with reference to the accompanying drawings, but the present disclosure is of course not limited to the above examples. Those skilled in the art can make various changes and modifications within the scope of the appended claims, and it is to be understood that such changes and modifications naturally fall within the technical scope of the present disclosure.

[0144] For example, a plurality of functions included in one unit in the above embodiments can be implemented by separate apparatuses. Alternatively, a plurality of functions implemented by a plurality of units in the above embodiments can be implemented by separate apparatuses, respectively. In addition, one of the above functions can be implemented by a plurality of units. Needless to say, such a configuration is included in the technical scope of the present disclosure.

[0145] In this specification, the steps described in the flowcharts described in the flowcharts include not only the processing performed in the order shown as a time series, but also the processing performed in parallel or individually rather than necessarily in the order shown as a time series. Furthermore, even in the steps that are processed in a time series, needless to say, the order can be changed as appropriate.

[0146] In addition, the technology according to the present disclosure can also be configured as follows.

[0147] Note 1. An image processing apparatus comprising:

[0148] a feature extraction unit configured to extract a feature of an input image;

[0149] a text detection unit configured to detect text in the input image based on the feature extracted by the feature extraction unit;

[0150] a language identification unit configured to identify a language of the text detected by the text detection unit;

[0151] a text recognition unit configured to recognize the detected text based on a recognition result of the language identification unit to obtain at least one character string set; and

[0152] a first classification unit configured to classify the input image by matching the at least one character string set and a predetermined character string set to obtain a first classification result representing a category of an object involved in the input image for obtaining a final classification result of the category of the object.

[0153] Paragraph 2. The image processing apparatus according to Paragraph 1, wherein the predetermined character string set is selected based on the recognition result of the language identification unit.

[0154] Paragraph 3. The image processing apparatus according to Paragraph 2, wherein the first classification unit is configured to classify the input image by matching a character string set corresponding to a predetermined position range in the input image among the at least one character string set and the predetermined character string set.

[0155] Paragraph 4. The image processing apparatus according to Paragraph 3, wherein the first classification unit is configured to:

[0156] match the character string set corresponding to the predetermined position range and a first subset of the predetermined character string set to obtain a first matching result;

[0157] match the character string set corresponding to the predetermined position range and a second subset of the predetermined character string set based on the first matching result to obtain a second matching result; and

[0158] classify the input image based on the first matching result and the second matching result.

[0159] Paragraph 5. The image processing apparatus according to Paragraph 4, wherein the category of the object involved in the input image is represented by a country or a region to which the object corresponds and an object type of the object, and

[0160] wherein the first subset includes character strings representing names of a plurality of countries and / or regions and the second subset includes character strings representing a plurality of object types, or the first subset includes character strings representing a plurality of object types and the second subset includes character strings representing names of a plurality of countries and / or regions.

[0161] Paragraph 6. The image processing apparatus according to any one of Paragraphs 1 to 5, further comprising:

[0162] a feature conversion unit configured to perform conversion processing on the features extracted by the feature extraction unit to obtain converted features;

[0163] a visual feature extraction unit configured to extract, based on the features extracted by the feature extraction unit, visual features of a region of the input image corresponding to the text detected by the text detection unit;

[0164] a text feature extraction unit configured to extract, based on the at least one string set, text features of the input image in a case where the at least one string set only involves characters of a predetermined language; and

[0165] a second classification unit configured to, in a case where the at least one string set only involves characters of the predetermined language, classify the input image based on the converted features obtained by the feature conversion unit, the visual features extracted by the visual feature extraction unit, and the text features extracted by the text feature extraction unit, by using a pre-trained classification model, to obtain a second classification result representing a category of an object involved in the input image,

[0166] wherein, in a case where the second classification result satisfies a predetermined condition, the second classification result is used as a final classification result of the category of the object involved in the input image,

[0167] wherein the first classification unit is further configured to, in a case where the at least one string set only involves characters of a predetermined language, classify the input image only in a case where the second classification result does not satisfy a predetermined condition, to obtain the first classification result as a final classification result of the type of the object involved in the input image, and

[0168] wherein the pre-trained classification model is a classification model based on a graph attention network.

[0169] Paragraph 7. The image processing apparatus according to Paragraph 6, wherein the second classification unit includes:

[0170] a feature fusion sub-unit configured to fuse the visual features extracted by the visual feature extraction unit and the text features extracted by the text feature extraction unit by using a feature fusion layer of the pre-trained classification model, to obtain a global graph network feature of the input image; and

[0171] A classification subunit configured to classify the input image by using the converted feature acquired by the feature conversion unit and the global graph network feature acquired by the feature fusion subunit to obtain a second classification result representing a category of an object involved in the input image.

[0172] Clause 8. The image processing apparatus according to clause 7, wherein

[0173] The category of the object involved in the input image is represented by a country or region corresponding to the object and an object type of the object,

[0174] The second classification result includes a first classification sub-result representing the country or region corresponding to the object and a second classification sub-result representing the object type of the object,

[0175] The classification subunit includes a first classification part and a second classification part,

[0176] The first classification part is configured to classify the input image by using the converted feature acquired by the feature conversion unit and the global graph network feature acquired by the feature fusion subunit to obtain the first classification sub-result, and

[0177] The second classification part is configured to classify the input image by using the converted feature acquired by the feature conversion unit and the global graph network feature acquired by the feature fusion subunit to obtain the second classification sub-result.

[0178] Clause 9. The image processing apparatus according to clause 8, wherein the pre-trained classification model is obtained by training an initial classification model by using a set of sample images in the following manner:

[0179] extracting features of the sample images;

[0180] performing conversion processing on the extracted features to acquire converted features;

[0181] detecting text in the sample images based on the extracted features;

[0182] identifying a language of the detected text;

[0183] identifying the detected text based on a result of the language identification to obtain at least one set of character strings;

[0184] extracting visual features of regions of the sample images corresponding to the detected text based on the extracted features;

[0185] extracting a text feature of the sample image based on the at least one character string set;

[0186] fusing the visual feature of the sample image and the text feature of the sample image by using a feature fusion layer of the initial classification model to obtain a global graph network feature of the sample image; and

[0187] training the initial classification model based on the first loss sub-function and the second loss sub-function for the initial classification model by using the converted feature and the global graph network feature of the sample image to meet a predetermined convergence condition, thereby obtaining the pre-trained classification model.

[0188] Addendum 10. An image processing method, comprising:

[0189] a feature extraction step for extracting a feature of an input image;

[0190] a text detection step for detecting text in the input image based on the feature extracted by the feature extraction step;

[0191] a language recognition step for recognizing a language of the text detected by the text detection step;

[0192] a text recognition step for recognizing the detected text based on a recognition result obtained by the language recognition step to obtain at least one character string set; and

[0193] a first classification step for classifying the input image based on the recognition result and the at least one character string set to obtain a first classification result representing a class of an object involved in the input image for obtaining a final classification result of the class of the object.

[0194] Addendum 11. The image processing method according to Addendum 10, wherein the predetermined character string set is selected based on a recognition result of the language recognition step.

[0195] Addendum 12. The image processing method according to Addendum 11, wherein in the first classification step, the input image is classified by matching a character string set corresponding to a predetermined position range in the input image in the at least one character string set and the predetermined character string set.

[0196] Addendum 13. The image processing method according to Addendum 12, wherein in the first classification step:

[0197] the character string set corresponding to the predetermined position range is matched with a first subset in the predetermined character string set to obtain a first matching result;

[0198] matching a second subset of the predetermined set of strings based on the first matching result with a set of strings corresponding to the predetermined location range to obtain a second matching result; and

[0199] classifying the input image based on the first matching result and the second matching result.

[0200] Note 14. The image processing method according to Note 13, wherein the category of the object involved in the input image is represented by a country or region corresponding to the object and an object type of the object, and

[0201] wherein the first subset includes strings representing names of a plurality of countries and / or regions and the second subset includes strings representing a plurality of object types, or the first subset includes strings representing a plurality of object types and the second subset includes strings representing names of a plurality of countries and / or regions.

[0202] Note 15. The image processing method according to any one of Notes 10 to 14, wherein in a case where the at least one set of strings only involves characters of a predetermined language, the method further comprises:

[0203] a feature conversion step for performing conversion processing on the features extracted by the feature extraction step to obtain converted features;

[0204] a visual feature extraction step for extracting visual features of a region of the input image corresponding to the detected text based on the features extracted by the feature extraction step;

[0205] a text feature extraction step for extracting text features of the input image based on the at least one set of strings; and

[0206] a second classification step for classifying the input image based on the converted features obtained by the feature conversion step, the visual features extracted by the visual feature extraction step, and the text features extracted by the text feature extraction step, using a pre-trained classification model, to obtain a second classification result representing a category of an object involved in the input image,

[0207] wherein in a case where the second classification result satisfies a predetermined condition, the second classification result is used as a final classification result of the category of the object involved in the input image,

[0208] wherein, in a case that the at least one character string set only involves characters of a predetermined language, the first classification step is only performed in a case that the second classification result does not satisfy a predetermined condition, to obtain the first classification result as a final classification result of a type of an object involved in the input image, and

[0209] wherein, the pre-trained classification model is a classification model based on a graph attention network.

[0210] Clause 16. The image processing method of clause 15, wherein the second classification step comprises:

[0211] a feature fusion sub-step for fusing the visual features extracted by the visual feature extraction step and the text features extracted by the text feature extraction step by using a feature fusion layer of the pre-trained classification model, to obtain a global graph network feature of the input image; and

[0212] a classification sub-step for classifying the input image by using the converted features obtained by the feature conversion step and the global graph network feature obtained by the feature fusion sub-step, to obtain a second classification result representing a category of an object involved in the input image.

[0213] Clause 17. The image processing method of clause 16, wherein,

[0214] the category of the object involved in the input image is represented by a country or region corresponding to the object and an object type of the object,

[0215] the second classification result comprises a first classification sub-result representing the country or region corresponding to the object and a second classification sub-result representing the object type of the object,

[0216] the classification sub-step comprises a first classification sub-step and a second sub-step,

[0217] in the first classification sub-step, for classifying the input image by using the converted features obtained by the feature conversion step and the global graph network feature obtained by the feature fusion sub-step, to obtain the first classification sub-result, and

[0218] in the second classification sub-step, for classifying the input image by using the converted features obtained by the feature conversion step and the global graph network feature obtained by the feature fusion sub-step, to obtain the second classification sub-result.

[0219] Clause 18. The image processing method of clause 17, wherein the pre-trained classification model is obtained by training an initial classification model using a set of sample images by:

[0220] extracting features of the sample images;

[0221] performing conversion processing on the extracted features to obtain converted features;

[0222] detecting text in the sample images based on the extracted features;

[0223] recognizing a language of the detected text;

[0224] recognizing the detected text based on a result of the language recognition to obtain at least one string set;

[0225] extracting visual features of regions of the sample images corresponding to the detected text based on the extracted features;

[0226] extracting text features of the sample images based on the at least one string set;

[0227] fusing the visual features of the sample images and the text features of the sample images using a feature fusion layer of the initial classification model to obtain global graph network features of the sample images; and

[0228] training the initial classification model based on a first loss sub-function and a second loss sub-function for the initial classification model using the converted features and the global graph network features of the sample images to satisfy a predetermined convergence condition to obtain the pre-trained classification model.

[0229] Clause 19. A computer-readable storage medium storing program instructions for execution by a computer to perform the method of any one of clauses 10 to 18.

Claims

1. An image processing apparatus, comprising: The feature extraction unit is configured to extract features from the input image; A text detection unit is configured to detect text in the input image based on features extracted by the feature extraction unit; The language identification unit is configured to identify the language of the text detected by the text detection unit; A text recognition unit is configured to recognize the detected text based on the recognition result of the language recognition unit, so as to obtain at least one string set; The first classification unit is configured to classify the input image by matching the at least one set of strings with a predetermined set of strings to obtain a first classification result representing the category of the object involved in the input image, so as to obtain a final classification result of the category of the object; The feature transformation unit is configured to transform the features extracted by the feature extraction unit to obtain transformed features; A visual feature extraction unit is configured to extract visual features of the region of the input image corresponding to the text detected by the text detection unit based on the features extracted by the feature extraction unit. The text feature extraction unit is configured to extract text features of the input image based on the at least one string set, provided that the at least one string set only involves characters of a predetermined language. as well as The second classification unit is configured to, when the at least one string set involves only characters of the predetermined language, use a pre-trained classification model to classify the input image based on the transformed features obtained by the feature transformation unit, the visual features extracted by the visual feature extraction unit, and the text features extracted by the text feature extraction unit, to obtain a second classification result representing the category of the objects involved in the input image. Where the second classification result meets predetermined conditions, the second classification result is used as the final classification result of the category of the object involved in the input image. The first classification unit is further configured to classify the input image only when the at least one string set involves characters of a predetermined language, and only if the second classification result does not meet a predetermined condition, so as to obtain the first classification result as the final classification result of the type of object involved in the input image. The first classification unit is configured to classify the input image by matching the set of strings in the at least one string set that corresponds to a predetermined position range in the input image with the predetermined string set.

2. The image processing apparatus according to claim 1, wherein, The first classification unit is configured as follows: The set of strings corresponding to the predetermined location range is matched with the first subset of the predetermined string set to obtain a first matching result; The set of strings corresponding to the predetermined location range is matched with the second subset of the predetermined string set based on the first matching result to obtain a second matching result; as well as The input image is classified based on the first matching result and the second matching result, wherein the category of the object involved in the input image is represented by the country or region corresponding to the object and the object type of the object. The first subset includes strings representing the names of multiple countries and / or regions, and the second subset includes strings representing multiple object types; or the first subset includes strings representing multiple object types, and the second subset includes strings representing the names of multiple countries and / or regions.

3. The image processing apparatus according to claim 1 or 2, wherein, The pre-trained classification model is a graph attention network-based classification model.

4. The image processing apparatus according to claim 3, wherein, The second classification unit includes: A feature fusion subunit is configured to utilize the feature fusion layer of the pre-trained classification model to fuse the visual features extracted by the visual feature extraction unit and the text features extracted by the text feature extraction unit to obtain global graph network features of the input image; and The classification subunit is configured to classify the input image using the transformed features obtained by the feature transformation unit and the global graph network features obtained by the feature fusion subunit, so as to obtain a second classification result representing the category of the objects involved in the input image.

5. The image processing apparatus according to claim 4, wherein, The category of the objects involved in the input image is represented by the country or region corresponding to the object and the object type. The second classification result includes a first classification sub-result representing the country or region corresponding to the object and a second classification sub-result representing the object type. The classification subunit includes a first classification section and a second classification section. The first classification unit is configured to classify the input image using transformed features obtained by the feature transformation unit and global graph network features obtained by the feature fusion subunit, to obtain a first classification sub-result. The second classification unit is configured to classify the input image using the transformed features obtained by the feature transformation unit and the global graph network features obtained by the feature fusion subunit, so as to obtain the second classification result.

6. The image processing apparatus according to claim 5, wherein, The pre-trained classification model is obtained by training the initial classification model using a sample image set in the following manner: Extract features from sample images; The extracted features are then transformed to obtain transformed features; Detect text in the sample image based on the extracted features; Identify the language of the detected text; The detected text is identified based on the language recognition results to obtain at least one set of strings; Based on the extracted features, visual features of the region in the sample image corresponding to the detected text are extracted; Based on the at least one set of strings, extract the text features of the sample image; The visual features and text features of the sample image are fused using the feature fusion layer of the initial classification model to obtain the global graph network features of the sample image. as well as By utilizing the transformed features of the sample images and global graph network features, and based on the first and second loss functions for the initial classification model, the initial classification model is trained to meet predetermined convergence conditions, thereby obtaining the pre-trained classification model.

7. An image processing method, comprising: The feature extraction step is used to extract features from the input image; A text detection step is used to detect text in the input image based on the features extracted through the feature extraction step; The language identification step is used to identify the language of the text detected by the text detection step. A text recognition step is used to recognize the detected text based on the recognition result obtained through the language recognition step, so as to obtain at least one string set; as well as The first classification step is used to classify the input image based on the recognition result and the at least one string set to obtain a first classification result representing the category of the object involved in the input image, so as to obtain the final classification result of the object category. Where the at least one string set involves only characters of a predetermined language, the method further includes: The feature transformation step is used to transform the features extracted by the feature extraction step to obtain transformed features; A visual feature extraction step is used to extract visual features of the region of the input image corresponding to the detected text based on the features extracted by the feature extraction step. A text feature extraction step is used to extract text features from the input image based on the at least one set of strings; and The second classification step utilizes a pre-trained classification model to classify the input image based on the transformed features obtained in the feature transformation step, the visual features extracted in the visual feature extraction step, and the text features extracted in the text feature extraction step, to obtain a second classification result representing the category of the objects involved in the input image. Where the second classification result meets predetermined conditions, the second classification result is used as the final classification result of the category of the object involved in the input image. Where the at least one string set involves only characters of a predetermined language, the first classification step is performed only if the second classification result does not meet the predetermined conditions, so as to obtain the first classification result as the final classification result of the type of object involved in the input image. In the first classification step, the input image is classified by matching the string set corresponding to a predetermined position range in the input image from the at least one string set with a predetermined string set.

8. The image processing method according to claim 7, wherein, The pre-trained classification model is a graph attention network-based classification model.

9. A computer-readable storage medium storing program instructions that, when executed by a computer, are used to perform the method according to claim 7 or 8.

Images