Information processing apparatus, information processing method, and storage medium

Abstract

To extract a region including a character string from an image in which a plurality of character strings written in a plurality of different directions are mixed with higher accuracy. An information processing apparatus includes an extraction unit, a determination unit, and a selection unit. The extraction unit extracts one or more character string regions including a plurality of character strings written in each of a plurality of directions from an image including the plurality of character strings. The determination unit determines, for one or more connection regions corresponding to regions in which a plurality of character string regions are connected, a direction in which a character string included in the connection region is written. The selection unit selects, for each of the one or more connection regions, one or more character string regions extracted in a direction identical to a direction determined for the connection region from among the one or more character string regions included in the connection region.

Description

Technical Field

[0001] The embodiments of the present invention relate to information processing apparatus, information processing method, and storage medium. Background Technology

[0002] There are known techniques for extracting regions containing strings (string regions) from an input image and for recognizing strings from the extracted string regions. Images sometimes contain multiple strings recorded in multiple different directions. When dealing with images where strings recorded in multiple directions coexist, it is required that the image be configured so as not to erroneously extract string regions as regions in directions different from the original recording directions. Summary of the Invention

[0003] The purpose of this invention is to provide an information processing apparatus, information processing method, and storage medium capable of extracting regions containing strings from an image in which multiple strings recorded in multiple different directions coexist with higher precision.

[0004] The information processing apparatus of this embodiment includes an extraction unit, a determination unit, and a selection unit. The extraction unit extracts one or more string regions from an image comprising multiple strings, according to each of multiple directions. The determination unit determines the direction in which the strings included in the connection region are recorded for each of one or more connection regions corresponding to regions that connect adjacent string regions. The selection unit, according to each of the one or more connection regions, selects one or more string regions extracted from the one or more string regions included in the connection region, for a direction consistent with the direction determined for the connection region. Attached Figure Description

[0005] Figure 1 This is a flowchart illustrating the process of extracting the string region for comparison examples.

[0006] Figure 2 This is a diagram used to illustrate an example of incorrectly extracting a string region.

[0007] Figure 3 This is a block diagram of the information processing apparatus according to the implementation method.

[0008] Figure 4 This is a flowchart illustrating an example of the extraction process in the implementation method.

[0009] Figure 5 This is a diagram representing a specific example of the extraction process.

[0010] Figure 6 This is a diagram used to illustrate the extraction of connected regions and the determination of directions.

[0011] Figure 7 This is a diagram used to illustrate the extraction example of the connection region in Variation Example 1.

[0012] Figure 8 This is a diagram used to illustrate the example of extracting the string region in variation example 2.

[0013] Figure 9 This is a diagram showing examples of display screens that can be applied to the implementation methods and various modifications.

[0014] Figure 10 This is a hardware configuration diagram of the information processing device in the implementation method.

[0015] Explanation of reference numerals in the attached figures: 100 Information Processing Device 101 obtained the department 102 Extraction Section 103 Judgment Department 104 Selection Department 105 Output Control Unit 121 Storage Department 122 Display Unit Detailed Implementation

[0016] Hereinafter, preferred embodiments of the information processing apparatus of the present invention will be described in detail with reference to the accompanying drawings.

[0017] The string extraction function, which extracts string regions from an image, is used as a pre-processing step in character recognition functions to identify strings from the extracted string regions. The extracted string regions are not limited to use in character recognition functions and can be used in any other function. For example, the string regions to be extracted can also be used in the following functions.

[0018] • Function to determine the language of strings within a string range • Functionality to determine the handwriting style of a string within a specified range (for identifying an individual). • Allocate multiple string regions to perform various processing functions As a technique for extracting string regions from an image, there are, for example, the following techniques.

[0019] Extraction of string regions based on semantic segmentation Extraction of string regions based on concentration histogram • String region extraction based on bounding box detection In any technique, for images containing multiple strings recorded in multiple different directions, it is sometimes impossible to extract the string regions in the correct orientation. The following are examples illustrating situations where string regions are not extracted correctly.

[0020] The following is an example (comparative example) illustrating string region extraction based on semantic segmentation. Furthermore, the following explanation uses the case where multiple directions are horizontal (left-right axis) and vertical (top-bottom axis). These multiple directions are not limited to horizontal and vertical; they can be any other direction. For example, multiple directions can also include tilting directions and directions along curves (circumferential directions, etc.).

[0021] Figure 1 This is a flowchart illustrating the process of extracting string regions based on semantic segmentation in the comparative example.

[0022] When image 201 is input, model 211 performs semantic segmentation on image 201 and outputs the semantic segmentation result 202. Model 211 is, for example, a model that learns by outputting classification information for each of the multiple partial regions included in the input image. The model can also be any constructed model, such as a neural network model.

[0023] A partial region may be a pixel or a region comprising multiple pixels (such as superpixels). Categorical information may indicate whether a partial region corresponds to or does not correspond to a string. For example, the categorical information may take values ​​like the following.

[0024] • The portion of the string that does not correspond to the string: 0 • The portion of the string corresponding to: 1 exist Figure 1 In the example, the rectangle within processing result 202 corresponds to a partial region (with a classification information value of 1) that is the same as the string. Next, among the multiple partial regions, the region connecting adjacent partial regions is extracted as the string region. For Figure 1 Image 201 outputs extraction results 203, which includes eight string regions 221-228. String regions 221-228 correspond to Japanese strings as follows.

[0025] • String range 221: This represents the first two characters ("Spring and Summer") of the four horizontally written Chinese characters representing the four seasons. • String range 222: This represents the remaining two characters ("autumn" and "winter") out of the four horizontally written Chinese characters representing spring, summer, autumn, and winter. • String range 223: This represents the first two characters ("Spring and Summer") of the four vertically written Chinese characters representing the four seasons. • String range 224: represents the remaining two of the four vertically written Chinese characters for spring, summer, autumn, and winter ("autumn and winter"). • String range 225: Five vertically written hiragana characters ("kakikukeko") ·String area 226: 5 vertically written hiragana characters ("あいうえお") • String range 227: Five horizontally written katakana characters ("アイウエオ") • String range 228: Five horizontally written katakana characters (“カキクケコ”) Next, determine the recording direction of the characters in each string region. The recording direction can be determined, for example, by the following method.

[0026] • Method for determining based on the aspect ratio of the string region • A method that performs character recognition in multiple recording directions (e.g., horizontal and vertical) and determines the recording direction based on the direction from which the most likely recognition result is obtained. • A method using a discriminator (discrimination model) learned by taking a string region as input and outputting the recorded direction. Figure 1 This example demonstrates how to obtain the ideal semantic segmentation result 202 and correctly extract the string region. Figure 2 This is a diagram used to illustrate an example of incorrectly extracting a string region.

[0027] exist Figure 2 The example shown is image 201b, where the input consists of four horizontally written Chinese characters representing the four seasons (spring, summer, autumn, and winter) divided into two lines. Processing results 202a-202e represent examples of possible semantic segmentation results for image 201b. Furthermore, in processing results 202a-202e, black areas represent regions that do not correspond to the string (classification information value = 0), and white areas represent regions that correspond to the string (classification information value = 1).

[0028] Processing result 202a corresponds to the result of correct semantic segmentation. For example, two horizontal string regions are extracted from processing result 202a. If character recognition is performed on the extracted two string regions, the correct recognition result is obtained, as shown below processing result 202a.

[0029] Processing result 202b is equivalent to a semantic segmentation result with classification information such as extracting string regions in the wrong direction. For example, two vertical string regions are extracted from processing result 202b. If character recognition is performed on the two extracted string regions, the correct recognition result string is obtained, but as shown below processing result 202b, the order of the obtained strings is incorrect for Japanese strings.

[0030] Processing results 202c~202e represent other examples of semantic segmentation results where classification information such as incorrectly extracted string regions was set. Processing results 202c~202e are equivalent to processing results where horizontal and vertical strings are mixed and presented as string regions. Even when performing character recognition on such string regions, correct recognition results are usually not obtained. The "?" in the recognition result indicates that although some characters are shown, the recognition result is incorrect.

[0031] Thus, in comparative examples, for images where strings from multiple recording directions coexist, it is sometimes impossible to properly extract the string regions.

[0032] The information processing apparatus of the embodiment has the function of eliminating the problems mentioned above that occurred in the comparative example. For example, the information processing apparatus of the embodiment extracts a string region including a string recorded in each of multiple directions, and determines the direction in which the string included in the link region corresponds to the region formed by linking multiple string regions. Furthermore, the information processing apparatus of the embodiment selects a string region extracted from the string regions included in the link region that corresponds to the determined direction. As a result, string regions can be extracted with higher precision from an image in which multiple strings recorded in multiple different directions coexist.

[0033] Figure 3 This is a block diagram illustrating an example of the configuration of the information processing apparatus 100 in an embodiment. For example... Figure 3 As shown, the information processing device 100 includes an acquisition unit 101, an extraction unit 102, a judgment unit 103, a selection unit 104, an output control unit 105, a storage unit 121, and a display unit 122.

[0034] The acquisition unit 101 acquires various types of information used in the information processing apparatus 100. For example, the acquisition unit 101 acquires an image that is the object of extraction for a string region. The image that is the object of processing is an image that includes multiple strings. The method by which the acquisition unit 101 acquires information can be any method, such as a method that can be used to receive information from an external device via a network or a method that can be used to read information from a storage medium.

[0035] Extraction unit 102 extracts one or more string regions from an image containing multiple strings, in each of multiple directions. For example, extraction unit 102 performs semantic segmentation of output classification information in each of the multiple directions and in each of the multiple partial regions (pixels, superpixels, etc.) included in the image, thereby extracting one or more string regions.

[0036] The following explanation uses the case where multiple directions are horizontal and vertical as examples. The same order can be applied even when other directions are included.

[0037] The extraction unit 102 performs semantic segmentation on the input image using, for example, a model learned by processing results of semantic segmentation with the recording direction of the output string horizontal and the recording direction of the string vertical. The extraction unit 102 extracts a region from multiple partial regions that connects adjacent partial regions as a string region.

[0038] The model can be divided into model M1, which outputs the semantic segmentation processing results with the horizontal recording direction, and model M2, which outputs the semantic segmentation processing results with the vertical recording direction. Alternatively, it can be a model that combines model M1 and model M2.

[0039] Model M1, for example, is a model that outputs classification information indicating whether a region corresponds to a horizontal string (e.g., value = 1) or a region other than that (e.g., value = 0). Model M2, for example, is a model that outputs classification information indicating whether a region corresponds to a vertical string (e.g., value = 1) or a region other than that (e.g., value = 0). That is, both Model M1 and Model M2 are models that classify into two classes.

[0040] Categorical information can also be represented as the certainty (probability) of each class. In this case, the class with the highest certainty is used as the classification result.

[0041] The extraction unit 102 is not limited to extracting the structure of string regions through semantic segmentation. For example, instance segmentation can be used instead of semantic segmentation. In addition, the extraction unit 102 can also extract more than one string region by applying either bounding box detection or a vision transformer. In either case, the extraction unit 102 only needs to be configured to extract more than one string region, including multiple strings recorded in each of multiple directions (e.g., horizontal and vertical), along each direction.

[0042] The extraction unit 102 can also deform the image in each of multiple directions and extract more than one string region using the deformed image. For example, the extraction unit 102 can also be configured to use a horizontally scaled-down image when extracting a string region that includes horizontally written strings. This allows for faster processing within the extraction unit 102. Furthermore, since the output is based on the processing result (string region) of the scaled-down image, subsequent processing using the processing result can be accelerated.

[0043] The determination unit 103 determines the direction of the string contained in one or more linked regions in the image. A linked region is a region included in the image, which is equivalent to a region that connects multiple adjacent string regions.

[0044] The judgment unit 103 performs semantic segmentation on the input image using a model M3 learned in a manner that outputs classification information, for example, according to each of the multiple partial regions included in the image. The classification information indicates which of a plurality of directions the direction of the string is recorded in the case of a region that does not correspond to the string or a region that does correspond to the string.

[0045] Model M3 is, for example, a model that outputs classification information with values ​​as follows: That is, model M3 classifies into 3 classes. Classification information can also represent the certainty (probability) of each class.

[0046] • The portion of the string that does not correspond to the string: 0 • The portion of the string corresponding to the horizontal line: 1 • The portion of the string corresponding to the vertical axis: 2 Model M3 can also be integrated with at least one of model M1 and model M2 used by extraction unit 102.

[0047] The determination unit 103 outputs classification information (value 1 or 2) indicating the direction of strings recorded in multiple partial regions, and connects multiple adjacent partial regions to generate one or more connected regions. Furthermore, for each of the generated connected regions, the determination unit 103 determines the direction of the strings recorded in that region. The determination unit 103 outputs the generated one or more connected regions and the determined direction for each region as the processing result.

[0048] The method by which the determination unit 103 determines the direction of the connected region can be any method. For example, the determination unit 103 determines the direction of the connected region based on the values ​​of the classification information of the multiple partial regions included in the connected region, and the direction represented by the value that the number of partial regions with that value is greater than other values.

[0049] The decision unit 103 may also use the priority assigned to each of the multiple directions to determine the direction in a way that gives priority to the direction assigned a higher priority than the other directions.

[0050] For example, the number of elements can be assigned a weight corresponding to the priority of each direction. Alternatively, if the horizontal priority is higher than the vertical priority, the number of regions with a set horizontal value can be multiplied by a weight greater than the vertical priority, and then compared with the number of regions with a set horizontal value. Priority can be specified, for example, through user interaction on the screen. That is, a setting section that allows users to set priorities can be provided. For example, when dealing with an image that is identified as containing a large number of vertical strings, the user can specify that vertical priority should be applied. This functionality improves the accuracy of direction determination.

[0051] It can also be configured to set priorities for each category of the image, with the user specifying the image category. For example, for categories with more vertically written strings, such as newspaper images, a priority indicating vertical orientation can be pre-set in the storage unit 121, etc., described later.

[0052] When using classification information that represents the certainty (probability) of each class, the classification result can also be determined based on the certainty assigned a weight corresponding to the priority. For example, suppose the certainty of a region corresponding to a horizontal string is 0.6, and the certainty of a region corresponding to a vertical string is 0.4. Furthermore, suppose a priority is assigned to prioritize the vertical direction, and the weight corresponding to this priority is 2. The determination unit 103 compares the value 0.8 (2 multiplied by 0.4) with 0.6 and determines that the classification result for this region is vertical.

[0053] The determination unit 103 may also use the character recognition results for each of the more than one linked regions to determine the direction. For example, the determination unit 103 performs character recognition assuming the linked region is horizontal and character recognition assuming the linked region is vertical. The determination unit 103 determines the direction in which the most likely recognition result is obtained as the recording direction of the string in the linked region.

[0054] The decision unit 103 is not limited to performing the extraction of connected regions and the determination of directions through semantic segmentation. For example, instance segmentation may be used instead of semantic segmentation. In addition, the decision unit 103 may also perform the extraction of connected regions and the determination of directions by applying either bounding box detection or a vision transformer.

[0055] The judgment unit 103 can also use the magnified or reduced image to determine the direction of the connected region. Furthermore, the judgment unit 103 can also output the magnified or reduced connected region. This enables improvements in judgment accuracy based on expanded field of view, as well as reductions in computational load (high-speed processing).

[0056] The selection unit 104 uses the processing result of the judgment unit 103 to select a string region of any direction from the multiple string regions extracted by the extraction unit 102. For example, the selection unit 104 selects one or more string regions from the string regions included in the linking region for each linking region, based on the direction determined for the linking region.

[0057] The output control unit 105 controls the output of various types of information used in the information processing device 100. For example, the output control unit 105 outputs information representing the string area selected by the selection unit 104. The method of outputting information can also be arbitrary, such as the method of displaying on the display unit 122, or the method of sending information to an external device via a network.

[0058] At least some of the aforementioned units (acquisition unit 101, extraction unit 102, judgment unit 103, selection unit 104, and output control unit 105) can be implemented by one or more processing units. For example, each of the aforementioned units can be implemented by one or more processors. For instance, each of the aforementioned units can also be implemented by having a processor such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) execute a program, i.e., by software implementation. Each of the aforementioned units can also be implemented by a dedicated IC (Integrated Circuit) processor, i.e., by hardware. Each of the aforementioned units can also be implemented using both software and hardware. When using multiple processors, each processor can implement one of the aforementioned units, or it can implement two or more of the aforementioned units.

[0059] Storage unit 121 stores various types of information used in the information processing device. For example, storage unit 121 stores images acquired by acquisition unit 101 and processing results of each unit. Storage unit 121 can be composed of commonly used storage media such as flash memory, memory card, RAM (Random Access Memory), HDD (Hard Disk Drive), and optical disc.

[0060] The display unit 122 is a display device for displaying various information, such as a liquid crystal display and a touch panel.

[0061] The information processing device 100 can be physically comprised of a single device or multiple devices. For example, the information processing device 100 can also be built in a cloud environment. Alternatively, the components within the information processing device 100 can be distributed across multiple devices.

[0062] Next, the string region extraction process of the information processing device 100 in the embodiment will be described. Figure 4 This is a flowchart illustrating an example of the extraction process in the implementation method. Figure 5 This is a diagram representing a specific example of the extraction process.

[0063] The acquisition unit 101 acquires the image as the extraction object for the string region (step S101). Figure 5 In the example, image 201 is obtained from part 101.

[0064] Extraction unit 102 extracts string regions from image 201 in each of multiple directions (step S102). Figure 5 In the example, the extraction unit 102 outputs the extraction result 502-1 of the horizontal string region based on model M1 and the extraction result 502-2 of the vertical string region based on model M2.

[0065] The determination unit 103 extracts the connected regions from the image 201 and determines the direction of the extracted connected regions (step S103). Figure 5 In the example, the judgment unit 103 outputs a processing result 502-3 including four connection regions 511-514. The two connection regions 512 and 513 enclosed by solid lines represent those judged as horizontal connection regions, and the remaining two connection regions 511 and 514 represent those judged as vertical connection regions. Furthermore, in Figure 5 In this context, connected regions are represented by elliptical shapes, but they are not limited to elliptical shapes and can also be represented by any other shape.

[0066] For each extracted link region, the selection unit 104 selects a string region whose direction matches the direction determined for that link region from the string regions included in that link region and extracted by the extraction unit 102 (step S104). Figure 5 Let's illustrate with an example. For instance, the linked region 511 is determined to be vertical. Therefore, the selection unit 104 selects the two string regions 521a and 521b included in the vertical string region extraction result 502-2 for the linked region 511. Similarly, the selection unit 104 extracts string regions for the remaining linked regions 512-514 and outputs the selection result 503 including the extracted string regions.

[0067] The output control unit 105 outputs the selected string region (step S105), ending the extraction process. The output string region can be used for character recognition processing, for example.

[0068] In the present embodiment, even for an image in which strings in multiple writing directions coexist, the writing direction can be determined in units of connection regions corresponding to regions formed by connecting multiple string regions. Thus, for example, as in Figure 2 the processing results 202c to 202e, it is possible to avoid extracting string regions connected by regions with mutually different writing directions.

[0069] Next, an example of the extraction of connection regions and the determination of directions performed in the extraction process will be described. Figure 6 is a diagram for explaining an example of the extraction of connection regions and the determination of directions. Figure 6 Similar to Figure 2 is an example of extracting a connection region for an image 201b including four horizontally written Chinese characters divided into two lines.

[0070] The determination unit 103 performs semantic segmentation on the image 201b based on the model M3 and outputs a processing result 601. The rectangles in the processing result 601 correspond to partial regions, and the numerical values represent the values of classification information. For the partial regions corresponding to the strings, it is preferable that all the values of the classification information are set to 1, which represents a horizontally written string. In the processing result 601, for the multiple partial regions corresponding to the second Chinese character ("Xia"), the value 2 representing a vertically written string is wrongly set.

[0071] The determination unit 103 outputs classification information (value 1 or value 2) indicating the direction of the written string, and generates a connection region by connecting multiple adjacent partial regions. In Figure 6 the example, one connection region 602 corresponding to the central part of the image 201b is generated.

[0072] The determination unit 103 determines the direction of the connection region. In Figure 6 the example, among the values of the classification information of the multiple partial regions included in the connection region, the number 52 of partial regions set to value 1 is greater than the number 12 of partial regions set to value 2. Therefore, the determination unit 103 determines the horizontal direction indicated by value 1 as the direction of the connection region.

[0073] (Variant Example 1) In the above-described embodiment, the generation (extraction) of the connection region by the determination unit 103 is performed independently of the extraction of the string region by the extraction unit 102 using models M1, M2, etc. In Variant Example 1, an example of generating (extracting) a connection region using the extraction result of the string region by the extraction unit 102 will be described.

[0074] In this modified example, the determination unit 103 integrates the multiple classification information for each direction output by the extraction unit 102 through semantic segmentation. Furthermore, the determination unit 103 uses the integrated classification information to generate one or more linked regions, each including multiple partial regions that have output classification information, wherein the classification information represents a region corresponding to a string.

[0075] Figure 7 This is a diagram illustrating an example of extracting the connected regions in this variation. Classification information 702-1 corresponds to the processing result of semantic segmentation based on model M1. Classification information 702-2 corresponds to the processing result of semantic segmentation based on model M2. Classification information 702-3 corresponds to the combination of classification information 702-1 and classification information 702-2. Combining, for example, involves calculating the sum of the two classification information sets.

[0076] The judgment unit 103 uses the classification information 702-3 to generate one or more connection regions. For example, the judgment unit 103 generates a region as a connection region that surrounds the portion of the classification information 702-3 whose value is other than 0. Figure 7 In the example, the decision unit 103 outputs the processing result 502-3b, which includes four connected regions.

[0077] (Variation Example 2) In the above implementation, models M1 and M2 are used, and models M1 and M2 output semantic segmentation results with horizontal and vertical recording directions, respectively. The learning of models M1 and M2 requires training data for the corresponding directions. Generally, providing training data for different recording directions is not easy. For example, training data representing horizontal strings needs to be prepared so that even strings that are actually vertical are judged as horizontal strings by model M1.

[0078] Therefore, in Variation 2, an example of constructing a model in a way that makes it easier to prepare learning data will be illustrated.

[0079] In this variant, the extraction unit 102 also uses a model MA that performs semantic segmentation on a character-by-character basis. The model MA is a model (first model) that takes an image as input and outputs regions in the image that include more than one character.

[0080] The extraction unit 102 extracts one or more string regions in each of multiple directions by taking into input a region of one or more characters extracted by model MA and outputting one or more string regions. Model MB includes, for example, model M1', which takes into input the output of model MA (the result of semantic segmentation at the character unit level) and outputs the processing result of semantic segmentation in the horizontal direction; and model M2', which takes into input the output of model MA (the result of semantic segmentation at the character unit level) and outputs the processing result of semantic segmentation in the vertical direction.

[0081] Figure 8 This is a diagram illustrating an example of string region extraction in this variation. Classification information 801 corresponds to the semantic segmentation result based on model MA. Classification information 802-1 corresponds to the semantic segmentation result based on model M1'. Classification information 802-2 corresponds to the semantic segmentation result based on model M2'.

[0082] When learning models M1', M2', and MA, learning data 811-1, 811-2, and 811-3 were used, respectively.

[0083] Learning data 811-1 is used to teach a horizontal string region. Learning data 811-2 is used to teach a vertical string region. Learning data 811-3 is used to teach a region of character units.

[0084] The gray areas within learning data 811-1 and 811-2 represent regions designated as areas where the difference with the correct answer data does not need to be calculated during learning. Thus, learning data 811-1 does not need to include learning data for identifying strings that would normally be included in a vertical string region as horizontal strings. Similarly, learning data 811-2 does not need to include learning data for identifying strings that would normally be included in a horizontal string region as vertical strings. Therefore, preparing learning data 811-1 and 811-2 is not difficult. Furthermore, preparing learning data for teaching character units, such as learning data 811-3, is also not difficult.

[0085] Models MA and MB (model M1' and model M2') learn in a way that reduces the difference between the output of model MA and the learning data 811-3, which is equivalent to the correct answer data (first correct answer data), and the difference between the output of model MB and the learning data 811-1 and learning data 811-2, which are equivalent to the correct answer data (second correct answer data).

[0086] (Example from the image) Figure 9This diagram illustrates an example of a display screen applicable to the implementation method and its variations. For example, the output control unit 105 outputs a display screen 901a to the display unit 122 for accepting changes in the direction of one or more selected string regions. The display screen 901a includes an input image and a reading result. The reading result indicates the result of character recognition performed on the string region extracted from the input image. Two rectangular frames representing the extracted string regions are shown on the input image. In the display screen 901a, horizontal string regions, including strings that were originally written vertically but were incorrectly identified as horizontal, are extracted.

[0087] Display screen 901b is a screen that indicates a change in the direction of a string region specified by a user. For example, the user selects region 902 to change the direction, specifies that the direction of the string region included in region 902 should be changed, and performs the recognition process again. For example, when a change in direction is accepted through display screen 901b, selection unit 104 selects one or more string regions extracted for the changed direction.

[0088] In the example of display screen 901b, it is specified that the horizontal orientation will be changed to the vertical orientation. When the determination result of the direction of the connection area is changed from horizontal to vertical, the selection unit 104 reselects one or more string areas extracted according to the vertical orientation shown in the changed determination result. Furthermore, character recognition is re-performed on the selected string areas. Display screen 901c is an example of a screen that includes the string areas whose orientation has been changed and the result of character recognition (reading result) for those string areas.

[0089] The vertical string region has already been extracted, so there is no need to extract the string region again if the direction is changed. That is, it can perform character recognition and other processing with changed direction at high speed.

[0090] Thus, in the information processing apparatus of the embodiment, it is possible to extract regions containing strings from an image in which multiple strings recorded in multiple different directions coexist with higher precision.

[0091] Next, use Figure 10 The hardware configuration of the information processing device in the implementation method will be described. Figure 10 This is an explanatory diagram showing an example of the hardware configuration of an information processing device for an implementation method.

[0092] The information processing device of the embodiment includes a control device such as a CPU (Central Processing Unit) 51, a storage device such as a ROM (Read Only Memory) 52 and a RAM (Random Access Memory) 53, a communication I / F 54 for communication with a network, and a bus 61 connecting the various parts.

[0093] The program executed by the information processing device of the embodiment is provided in advance by being pre-assembled into ROM 52, etc.

[0094] The program executed by the information processing device of the embodiment may also be provided as a computer program product as a file recorded in an installable or executable form on a computer-readable recording medium such as CD-ROM (Compact Disk Read Only Memory), floppy disk (FD), CD-R (Compact Disk Recordable), DVD (Digital Versatile Disk).

[0095] Furthermore, the program executed by the information processing device of the embodiment can be stored on a computer connected to a network such as the Internet, and provided by downloading it via the network. Alternatively, the program executed by the information processing device of the embodiment can be provided or distributed via a network such as the Internet.

[0096] The program executed by the information processing apparatus of the embodiment enables the computer to function as a component of the aforementioned information processing apparatus. The computer can read the program from a computer-readable storage medium into the main storage device and execute it via the CPU 51.

[0097] Several embodiments of the present invention have been described, but these embodiments are provided by way of example and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments or variations thereof are included in the scope or spirit of the invention, and are included within the scope of the invention as described in the claims and its equivalents.

Claims

1. An information processing device, wherein, have: The extraction unit extracts one or more string regions from an image containing multiple strings, in each of multiple directions; The determination unit determines the direction of the string included in the connection region for each of one or more connection regions that corresponds to a region formed by connecting multiple adjacent string regions. as well as The selection unit, according to each of the more than one connection regions, selects one or more string regions within the more than one string regions included in the connection region, for the direction consistent with the direction determined for the connection region.

2. The information processing apparatus according to claim 1, wherein, The judgment unit outputs classification information for each of the multiple partial regions included in the image. The classification information indicates whether the region does not correspond to the string, or, if it is a region corresponding to the string, which of the multiple directions the string's direction belongs to. The determination unit connects multiple adjacent partial regions that output classification information indicating the direction of the recorded string, thereby generating one or more connection regions. The determination unit determines the direction of the string included in the connection region for each of the generated one or more connection regions.

3. The information processing apparatus according to claim 1, wherein, The extraction unit outputs classification information indicating whether a region corresponds to a string recorded in a given direction, based on each of the multiple directions and each of the multiple partial regions included in the image, thereby extracting one or more string regions. The judgment unit integrates the multiple classification information output according to each of the multiple directions to generate one or more connection regions, each including multiple partial regions that output the classification information, where the classification information represents a region corresponding to a string. The determination unit determines the direction of the string included in the connection region for each of the generated one or more connection regions.

4. The information processing apparatus according to claim 1, wherein, The extraction unit extracts regions of one or more characters using a first model and extracts regions of one or more strings using a second model in each of multiple directions. The first model inputs the image and outputs regions of one or more characters included in the image, and the second model inputs the extracted regions of one or more characters and outputs regions of one or more strings.

5. The information processing apparatus according to claim 4, wherein, The first model and the second model learn in a way that reduces the difference between the output of the first model and the first correct answer data, and the difference between the output of the second model and the second correct answer data.

6. The information processing apparatus according to claim 1, wherein, The extraction unit applies one of semantic segmentation, instance segmentation, bounding box detection, and visual transformer in each of the plurality of directions to extract one or more of the string regions.

7. The information processing apparatus according to claim 1, wherein, The extraction unit deforms the image in each of the plurality of directions, and uses the deformed image to extract one or more string regions.

8. The information processing apparatus according to claim 1, wherein, The judgment unit applies any one of semantic segmentation, instance segmentation, bounding box detection, and visual transformer to determine the direction for one or more of the connected regions.

9. The information processing apparatus according to claim 1, wherein, The determination unit uses the magnified or reduced image to determine the direction for one or more of the connected regions.

10. The information processing apparatus according to claim 1, wherein, The determination unit uses the character recognition results for each of the more than one connected regions to determine the direction.

11. The information processing apparatus according to claim 1, wherein, The decision unit uses the priority of each of the multiple directions to determine the direction.

12. The information processing apparatus according to claim 1, further comprising: The output control unit outputs a display screen to the display unit for accepting changes in the direction of one or more selected string regions. When the change of direction is accepted through the display screen, the selection unit selects one or more of the string regions extracted for the changed direction.

13. An information processing method, executed by an information processing device, wherein, The information processing method includes: The extraction step involves extracting one or more string regions from an image containing multiple strings, in each of multiple directions; The determination step involves, for one or more linked regions that correspond to a region formed by connecting multiple adjacent string regions, determining the direction of the string included in the linked region; and In the selection step, for each of the more than one connection regions, in the more than one string regions included in the connection region, one or more string regions are selected for the direction that is consistent with the direction determined for the connection region.

14. A storage medium, wherein, It stores programs for causing the computer to perform the following steps: The extraction step involves extracting one or more string regions from an image containing multiple strings, in each of multiple directions; The determination step involves, for one or more linked regions that correspond to a region formed by connecting multiple adjacent string regions, determining the direction of the string included in the linked region; and In the selection step, for each of the more than one connection regions, in the more than one string regions included in the connection region, one or more string regions are selected for the direction that is consistent with the direction determined for the connection region.

Images