Information processing device, information processing method, and program

Abstract

To improve the accuracy of linking body texts recognized from an image.SOLUTION: An information processing device D includes a control unit 1, a communication unit 2, and a memory unit 3. The control unit 1 performs operations including acquiring an article image, acquiring one or more areas from the article image, classifying at least one of the one or more areas as a body text, recognizing characters in the body text, and concatenating multiple possibly consecutive body texts while allowing overlaps in accordance with predetermined reading order rules.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] A technique is known in the past that extracts a set of lines with a fixed direction from a set of basic elements that make up a document, and extracts columns by associating the lines with each other (for example, Patent Document 1). In this technique, linking of column elements is performed so as to satisfy the following three conditions: 1) Every row component belongs to one and only one column element. 2) The column areas of the column elements do not overlap each other. Here, a column area is an area defined by the circumscribing rectangles of the line components belonging to each column element. 3) The row elements belonging to a column element with a determined column direction are ordered in the column direction and are merged together so that no row elements overlap in the column direction orthogonally. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 11-2194072 Summary of the Invention [Problem to be solved by the invention]

[0004] The background art does not consider that an element may overlap with other elements as a candidate for following them, so there is room for improvement in the linking accuracy.

[0005] In view of the above circumstances, an object of the present disclosure is to improve the accuracy of concatenation of text recognized from an image. [Means for solving the problem]

[0006] In order to solve the above problems, the information processing device according to this embodiment is An information processing device including a control unit, a communication unit, and a storage unit, the control unit comprising: Obtaining an article image; obtaining one or more regions from the article image; classifying at least one of said one or more regions as text; recognizing characters of said body of text; Concatenating multiple possibly consecutive texts according to a predetermined reading order rule, while allowing overlaps; The operation includes:

[0007] In order to solve the above problem, the information processing method according to the present embodiment includes: An information processing method by an information processing device including a control unit, a communication unit, and a storage unit, Obtaining an article image; obtaining one or more regions from the article image; classifying at least one of said one or more regions as text; recognizing characters of said body of text; Concatenating multiple possibly consecutive texts according to a predetermined reading order rule, while allowing overlaps; Includes.

[0008] In order to solve the above problem, the program according to this embodiment is The computer functions as the information processing device. Effect of the Invention

[0009] According to an embodiment of the present disclosure, it is possible to improve the accuracy of concatenation of text recognized from an image. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a functional block diagram of an information processing device. [Diagram 2] FIG. 1 illustrates region acquisition and classification using an object detection model. [Diagram 3] FIG. [Figure 4] FIG. 4 is a diagram showing an article image after utilizing the results of FIGS. 2 and 3. [Diagram 5] FIG. 1 is a diagram illustrating classification of regions by an image classification model. [Figure 6] FIG. [Figure 7] FIG. [Figure 8] FIG. 13 is a diagram showing a sample result of region acquisition and classification. [Figure 9] FIG. 13 is a diagram showing vertical concatenation of the text. [Figure 10] FIG. 13 is a diagram showing types of articles. [Figure 11] FIG. [Figure 12] FIG. 13 is a diagram showing types of boxed articles. [Figure 13] FIG. 13 is a diagram showing grouping of boxed articles. [Figure 14] FIG. 13 shows alternative examples of boxed article grouping. [Figure 15] FIG. 1 shows the reading order of the text. [Figure 16] This is a diagram showing the reading order of the main text in a horizontally composed article. [Figure 17] FIG. 13 is a diagram showing the connection of the main text. [Figure 18] FIG. 13 is a diagram showing the result of deduplication. [Figure 19] FIG. 13 illustrates reclassification of subheadings. [Figure 20] FIG. 13 is a diagram showing the association between an image and an explanation. [Figure 21] FIG. 13 is a diagram showing the association of an image and pictorial explanation with an article. [Figure 22] FIG. 13 shows an image without a pictorial explanation. [Diagram 23] FIG. 13 is a diagram showing how to determine the similarity between a pictorial explanation and a caption. [Figure 24] FIG. 13 is a diagram showing a sample result of region connection. [Diagram 25] FIG. 2 is a diagram showing the data structure of an article DB (database). [Figure 26]FIG. 11 is a flowchart illustrating an information processing method executed by the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of the present invention will be described with reference to the drawings. 1 is a functional block diagram of an information processing device D. The information processing device D includes a control unit 1, a communication unit 2, and a storage unit 3.

[0012] The information processing device D may be a server. The information processing device D may be installed in, for example, a facility dedicated to a business operator or a shared facility including a data center. The process executed by the information processing device D may be executed by a plurality of information processing devices that are distributed.

[0013] The control unit 1 controls the communication unit 2 and the storage unit 3, and also controls all operations related to the entire operation of the information processing device D. The control unit 1 includes, for example, one or more general-purpose processors including a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The control unit 1 may include one or more dedicated processors specialized for specific processing. Instead of including a processor, the control unit 1 may include one or more dedicated circuits. The dedicated circuits may be, for example, a FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 1 may include an ECU (Electronic Control Unit). The control unit 1 transmits and receives any information via the communication unit 2.

[0014] The communication unit 2 includes a communication module compatible with one or more wired or wireless LAN (Local Area Network) standards for connecting to a network. The communication unit 2 may include a module compatible with one or more mobile communication standards including LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation). The communication unit 2 may include a communication module compatible with one or more short-range communication standards or specifications including Bluetooth (registered trademark), AirDrop (registered trademark), IrDA, ZigBee (registered trademark), Felica (registered trademark), or RFID. The communication unit 2 transmits or receives any information to another terminal via the network.

[0015] The memory unit 3 includes, for example, a semiconductor memory, a magnetic memory, an optical memory, or a combination of at least two of these, but is not limited to these. The semiconductor memory is, for example, a RAM or a ROM. The RAM is, for example, an SRAM or a DRAM. The ROM is, for example, an EEPROM. The memory unit 3 may function, for example, as a main memory device, an auxiliary memory device, or a cache memory. The memory unit 3 may store information resulting from analysis or processing by the control unit 1. The memory unit 3 may store various information related to the operation or control of the information processing device D, etc. The memory unit 3 may store system programs, application programs, embedded software, etc. The memory unit 3 may be provided outside the information processing device D and accessed from the information processing device D. The memory unit 3 includes an article DB31.

[0016] The information processing method executed by the information processing device D will be described in detail below.

[0017] [1. Acquisition and classification of territories] The memory unit 3 of the information processing device D stores an article image. The article image includes one or more articles. The article image may be obtained by capturing or scanning the page of a newspaper, magazine, or the like. The control unit 1 acquires the article image. As shown in FIG. 2, the control unit 1 acquires (extracts) one or more regions from the article image using an arbitrary object detection model. The control unit 1 classifies each acquired region into, for example, one of the following region types. Headline ·image Pictorial explanations (i.e. captions that explain the content of the corresponding images) ·table ·advertisement ·Face name field

[0018] The object detection model may be, for example, a fine-tuning of Faster R-CNN trained using HJDataset. The object detection model may be trained on a predetermined number of newspaper pages. The detection method for acquiring and classifying regions is arbitrary.

[0019] The control unit 1 performs image processing on the article image. For example, as shown in A of FIG. 3, the control unit 1 may perform noise processing and a process of setting the lower advertisement at the bottom of the article image as an exception to the RLSA (Run Length Smoothing Algorithm) and contour extraction described below. As an additional example or an alternative example, the control unit 1 performs segmentation by RLSA as shown in B of FIG. 3. Specifically, when the number of adjacent white pixels in the article image is equal to or less than a threshold, the control unit 1 replaces the white pixels with black pixels. Any OSS may be used for implementation. RLSA may be applied to an image that has been subjected to morphological transformation by OpenCV. As an additional example or an alternative example, the control unit 1 may perform contour extraction by OpenCV and obtain a rectangular region of the contour as shown in C of FIG. 3. Specifically, the control unit 1 obtains the boundary line of each region and obtains a rectangle circumscribing each boundary line for each region. The RLSA, OSS, and OpenCV used in the image processing here are examples, and any other algorithm or program may be used as an alternative.

[0020] When only the object detection model is applied to the article image, there is a possibility that an area may not be detected or may be misaligned. When only image processing is applied to the article image, one headline may be separated into multiple areas due to differences in the spacing between characters in the article image. Therefore, the control unit 1 may generate an article image that integrates the results of applying the object detection model and the results of image processing, as shown in FIG. 4. The integration method is arbitrary.

[0021] The control unit 1 classifies the region using an image classification model. The image classification model may be a fine-tuning of the Swin Transformer. In training the image classification model, a segmentation image may be used.

[0022] The results of using an image classification model in addition to the object detection model and image processing are shown in Figure 5. With this configuration, in addition to classifying regions into the region types mentioned above (e.g., headlines, images, etc.), we can also classify regions into body text (e.g., body text region 53).

[0023] Additionally or alternatively, the control unit 1 performs post-processing on the article image shown in Fig. 5. For example, the control unit 1 may combine the erroneously subdivided regions 51 into one region. The control unit 1 may delete the region 52 that is not required to be acquired. The article image after the post-processing is performed is shown in Fig. 6.

[0024] The control unit 1 may determine, as a heading region, an area acquired and classified as a heading by an object detection model or image processing. The control unit 1 identifies an area acquired as a heading region by an image classification model among areas not determined as heading regions. The control unit 1 may determine, among the identified areas, an area whose size exceeds a predetermined value as a heading, and may determine an area whose size is equal to or smaller than the predetermined value (e.g., areas 71 and 72 in FIG. 7) as a sub-heading.

[0025] The following regions obtained and classified by the above process are shown in FIG. Headings (e.g. area 81) Body text (e.g. area 82) Image (e.g. area 83) Picture explanation (for example, area 84) Table (e.g. area 85) Advertising (e.g. area 86) - Surface name field (e.g. area 87)

[0026] [2. Character recognition] The control unit 1 recognizes characters in the text for each acquired and classified region using OCR (Optical Character Recognition / Reader). For example, Google's API may be used as the OCR engine. The control unit 1 may remove noise or process the article image before character recognition to improve accuracy. For example, the control unit 1 may remove salt and pepper noise for each region using OpenCV.

[0027] The area to be subjected to character recognition may be limited to only the character area or the body area. This allows the character to be estimated from the context, thereby improving the accuracy of character recognition. Additionally or alternatively, the control unit 1 may use OCR to recognize characters in headings, pictorial explanations, or advertisements.

[0028] The control unit 1 stores the recognized characters in the storage unit 3 in association with each area.

[0029] As shown in FIG. 9, the control unit 1 connects characters recognized in the body region in the following procedure. Step 1. Get an image of the body area Step 2. Use the RLSA described above to join the images vertically for each row. Step 3. Get the borders for each row Step 4. Get the bounding rectangles for each border, one for each row Step 5. Arrange the rectangles from top to bottom, starting with the rectangle on the right. As an additional or alternative example, the control unit 1 may eliminate the influence of characters that are not the main text of the target region by whitening (i.e., masking) the heading region that overlaps the main text of the target region at a point between steps 1 and 2 above.

[0030] In the example of FIG. 9, a case where the main text is written vertically is described. However, the control unit 1 may determine whether the writing direction of the main text is vertical or horizontal using an existing machine learning model (e.g., CRAFT). Specifically, the control unit 1 estimates the area for each character and the area obtained by connecting them, and determines that the main text is written vertically if there are many vertically long areas in the multiple connected areas. The control unit 1 determines that the main text is written horizontally if there are many horizontally long areas in the multiple connected areas. If the control unit 1 determines that the main text is written horizontally, instead of the above step 5, it arranges the rectangles from the top in order from left to right.

[0031] [3. Linking Areas] The control unit 1 detects the type of article by the following method, and connects one or more texts according to the type of article. As shown in FIG. 10, for example, the types of articles are as follows: Regular article (A in Figure 10) Boxed article (Figure 10B) Horizontally-composed article (C in Fig. 10)

[0032] Specifically, the control unit 1 detects ruled lines (i.e., straight lines) from the article image using OpenCV etc. As an example, an article image before ruled line detection (A in FIG. 11) and an article image after ruled line detection (B in FIG. 11) are shown.

[0033] (1) Detecting the type of article and determining the reading order (1-1) Boxed article A boxed article is an article arranged in a box shape (i.e., a rectangular shape). Boxed articles are at least divided into a type in which the article is surrounded by a frame 121 (A in FIG. 12) and a type in which the article is not surrounded by a frame (B in FIG. 12).

[0034] As a first method for detecting a boxed article, the control unit 1 determines whether or not a frame exists in the article image shown in Fig. 13. For example, the control unit 1 may determine whether or not a frame exists by the following procedure. Step 1. Generate an image in which all the acquired areas, including the heading and body text, are masked (whited out) (in this case, an image is generated in which only the lines remain) Step 2. Extract contours and detect rectangles using OpenCV Step 3. Filtering according to the length and width of the rectangle When the control unit 1 detects a frame, it detects an area 131 surrounded by the frame as a box-based area. One or more areas in the box-based area are grouped and associated with one article. The associated one or more areas are stored in the storage unit 3 in association with each box-based article.

[0035] As a second method for detecting boxed articles, the control unit 1 judges whether or not there is a horizontal ruled line in the article image shown in FIG. 14. Here, a horizontal ruled line refers to a ruled line that separates multiple vertically arranged areas. If the control unit 1 judges that there is no horizontal ruled line between one body text area and another, it groups those body text areas and associates them with one boxed article. This method is effective when there is no area surrounded by a frame in the article image. Next, when the control unit 1 detects a ruled line surrounding those body text areas, it detects boxed areas with the ruled line as a boundary line and associates the boxed areas with boxed articles (here, reference numbers 141 and 142). The control unit 1 detects heading areas within each boxed area. The control unit 1 associates the detected headings with the boxed articles and stores them in the memory unit 3.

[0036] The control unit 1 determines the reading order of the body text area in each detected boxed article in accordance with a predetermined reading order rule. For example, the predetermined reading order rule is as follows. In vertical writing: Read to the left, and when you come across a line, an image (however, an image whose height is greater than the height of one column of the page), a heading, or an illustration, jump to the right edge of the lower area. In boxed articles, read over headings. When the text is written horizontally, it will read to the right and jump to the lower area when it hits a line.

[0037] For example, in the example of a boxed article shown in A of Fig. 15, when reading continues to the left 151 and encounters an explanation 152, the user jumps to the main text area 153 below. As shown in B of Fig. 15, the control unit 1 determines the reading order of the main text in the boxed article as follows. Area 0: First in reading order Area 1: Second in reading order Area 2: Reading order 3 Area 3: Reading order 4th Area 4: Reading order 5th Area 5: Reading order 6th Area 6: Reading order 7th

[0038] (1-2) Horizontally-composed article As a method of detecting horizontally set articles, when the control unit 1 detects one or more body texts that have been determined to be written horizontally through character recognition by OCR, it groups the one or more body texts and associates them with one horizontally set article using the same method as the method for detecting boxed articles.Furthermore, the control unit 1 determines the reading order of the body text in horizontally set articles using the same method as the method for determining the reading order in the case of horizontally set boxed articles.As shown in Figure 16, the control unit 1 determines the reading order of horizontally set articles as follows. Area A0: 1st in reading order Area A1: Second in reading order Area A2: 3rd in reading order

[0039] (1-3) Regular articles The control unit 1 detects an area in the article image excluding the boxed article area and the horizontally set article area as a candidate area for a regular article. The control unit 1 searches for the start position of the text in the candidate area for a regular article. The start position may be determined as follows. - If there is text to the left of the heading: The start position is the top right corner of the text above the left of the heading. If there is no text to the left of the heading, the start position is the top right corner of the text below the heading.

[0040] The control unit 1 links all the multiple body texts that may be consecutive for each article according to a predetermined reading order rule. In this case, the body text of a certain area is allowed to be linked to the body texts of multiple other areas in a overlapping manner. In the example of Figure 17, the control unit 1 links the body texts as follows. The text in areas A0 to A2 is linked to the text in areas X1 to X6. The text in area B0 is linked to the text in areas X1 to X6. The text in areas C0 to C2 is linked to the text in areas X4 to X6. · The text in areas D0 to D3 is linked to the text in areas X4 to X6. · The text of area E0 is linked to the text of area E1. · The body of area G0 is linked to the body of area X6.

[0041] As described above, the text of the regions X1 to X6 is overlapped and linked to the text of the other regions. When the control unit 1 determines that there is no other text region that may be adjacent to the region E1 and the region F0, it determines that the text of the article is complete for each of the regions E1 and F0.

[0042] The control unit 1 executes the process of eliminating overlaps. Specifically, the control unit 1 judges whether or not the main text of the immediately preceding candidate area for each of areas X1 to X6 ends with a period. For example, the control unit 1 refers to area A2 and area B0, which are the candidate areas immediately preceding area X1, and detects that the main text of area A2 ends with a period and that the main text of area B0 does not end with a period. The control unit 1 judges the main text of area B0 as the main text immediately preceding area X1. The control unit 1 repeats the process of eliminating overlaps until there are no more overlaps. The result of elimination of overlaps is shown in FIG. 18.

[0043] As an additional or alternative example to the duplication elimination process, the control unit 1 may determine the contextual connection with the immediately preceding text using, for example, Next Sentence Prediction (based on BERT) to determine the text immediately preceding the overlapped region.

[0044] As shown in A of FIG. 19, in a regular article, some of the subheadings (here, areas 191, 192, and 193) may be determined to be headlines, and one article may be separated into multiple articles. To resolve the separation, the control unit 1 may classify the subheadings and re-execute the processing performed on regular articles on the execution results. Specifically, the classification of the subheadings may be performed using a classification model. The classification model may be constructed by the following procedure. Step 1. Run the headline candidates through spaCy to break them down into parts of speech and count each part of speech. Step 2: Using the headline candidate and the following sentence, BERT performs binary classification into subheading and headline, and obtains the probability that the headline candidate is a subheading. Step 3. The "parts of speech contained in the subheading" and "probabilities output by BERT" obtained from steps 1 and 2 above are input into LightGBM, and it is determined whether the headline candidate is a subheading or a headline. The classification result of the subheadings is shown in Fig. 19B. As shown in Fig. 19B, the body text regions A0 to A7 are grouped and associated with one regular article, that is, the body text regions A0 to A7 belong to the same article.

[0045] The control unit 1 may acquire a text region that does not belong to any article and store it in the storage unit 3.

[0046] (2) Linking images and explanations The control unit 1 associates an image and a pictorial explanation for explaining the content of the image with each other as a pair and stores them. The association method is divided into a case where the article is a boxed article and a case where the article is not a boxed article. (2-1) Boxed article The control unit 1 detects an image and a pictorial annotation within a boxed region that has been determined to be a boxed article, and stores the image and the pictorial annotation in the storage unit 3 in association with the boxed article. (2-2) Other than boxed articles It may be difficult to determine a pair between an image and a corresponding explanation only from the distance between them. The control unit 1 executes the association by utilizing a rule that an image and an explanation are not paired across a ruled line. Specifically, the control unit 1 acquires the ruled lines on all four sides (i.e., top, bottom, left, and right) of the explanation area 201 shown in A of FIG. 20. The acquired ruled lines are indicated by arrows 201A, 201B, 201C, and 201D. The control unit 1 acquires a rectangular area 202 surrounded by the acquired ruled lines as shown in B of FIG. 20. The control unit 1 acquires an image area 203 that overlaps with the area 202. As shown in C of FIG. 20, the control unit 1 associates the explanation area 201 and the image area 203 as a pair.

[0047] After associating the pair, the control unit 1 acquires the ruled lines on all four sides of the image area 203, as shown in A of FIG. 21. The acquired ruled lines are indicated by arrows 211A, 211B, 211C, and 211D. As shown in B of FIG. 21, the control unit 1 acquires a rectangular area 212 surrounded by the acquired ruled lines. The control unit 1 sets one or more articles to which a heading area (e.g., heading area 213) or a body area (e.g., body area 214) overlapping with the area 212 is associated, as candidates for an article to which the pair of the explanation area 201 and the image area 203 is associated. When there is one candidate, the control unit 1 associates the pair of the explanation area 201 and the image area 203 with the candidate article and stores it in the storage unit 3.

[0048] As an additional or alternative example, when the control unit 1 detects a text area 221 that reads "=Photo=" or "=Photo" (Figure 22) in the text associated with the image, it may determine that there is no pictorial explanation associated with the image.

[0049] Additionally or alternatively, the control unit 1 may determine an article to which a pictorial explanation is associated based on the similarity between the pictorial explanation and the headline. For example, Sentence-BERT may be used to determine the similarity. In the example of FIG. 23, the control unit 1 determines the distance between the vectors of the pictorial explanation 231, "Fishermen busy with a bumper catch of bluefin tuna..." and the headline 232, "'Ashinaga-san' collected donations for sick orphans." The determined distance is 0.9, for example. The following technology may be used to implement the determination process. https: / / docs.scipy.org / doc / scipy / reference / generated / scipy.spatial.distance.cdist.html The control unit 1 determines that the distance between the vectors of the pictorial explanation 231, "Fishermen busy with bumper catch of bluefin tuna..." and the headline 233, "Record quantity of bluefin tuna, 35 tons caught in one net," is 0.4. The determined distance is 0.4, for example. The control unit 1 associates the headline 233, which has a small distance between its vectors and the pictorial explanation 231 (i.e., has a high similarity to the pictorial explanation 231), and the pictorial explanation 231 with the same article.

[0050] The result of linking regions executed in this manner is shown in Figure 24. Regions that are associated with the same article (i.e., belong to the same article) are surrounded by lines of the same type.

[0051] [4. Classification] The control unit 1 assigns one or more classifications to each of the articles created by linking regions based on the headline and the text using a classifier. For example, the classifier may be a fine-tuned version of BERT.

[0052] 25, the control unit 1 stores in the storage unit 3 a headline, a body text, an image, a position, a classification, etc., in association with each article. Additionally or alternatively, the control unit 1 may store in the storage unit 3 an explanation in association with each article. The control unit 1 can use the information stored in the storage unit 3 to electronically generate the article.

[0053] FIG. 26 shows a flowchart of an information processing method executed in the information processing device D.

[0054] In S1, the control unit 1 of the information processing device D acquires an article image. In S2, the control unit 1 acquires one or more regions from the article image and classifies the one or more regions into the main text, etc. In S3, the control unit 1 recognizes characters in the main text by OCR.

[0055] In S4, the control unit 1 links the regions and organizes them by article. S4 includes linking multiple bodies of text that may be continuous, while allowing overlaps, according to a predetermined reading order rule. In S5, the control unit 1 assigns a classification to each article.

[0056] As described above, according to this embodiment, the control unit 1 of the information processing device D includes acquiring an article image, acquiring one or more regions from the article image, classifying at least one of the one or more regions into the main text, recognizing characters in the main text, and linking multiple possibly consecutive main texts while allowing overlaps according to a predetermined reading order rule. With this configuration, the information processing device D can improve the linking accuracy of the main texts recognized from the image. Furthermore, for example, the information processing device D can extract article information from a non-digitized paper page, store it in a database, and enable reprinting for use.

[0057] According to this embodiment, the operation of the control unit 1 includes, when detecting a body text that is overlapped and connected to a plurality of other body texts, determining whether each of the other body texts ends with a period, and determining the body text that does not end with a period as the immediately preceding body text. With this configuration, the information processing device D can improve the accuracy of determining the immediately preceding body text.

[0058] According to the present embodiment, the operation of the control unit 1 includes, when detecting a body text that is overlapped and linked with a plurality of other bodies texts, determining the immediately preceding body text from the contextual connection. With this configuration, the information processing device D can improve the accuracy of determining the immediately preceding body text.

[0059] According to the present embodiment, the operation of the control unit 1 includes determining that the article body is complete when it is determined that there is no body text that is linked to multiple other bodies in a duplicated manner. With this configuration, the information processing device D can reliably perform duplicate elimination.

[0060] According to this embodiment, the operation of the control unit 1 includes acquiring and classifying at least one area of ​​a headline, an image, an illustration, a front page, an advertisement, and a page name column in the acquired article image using an object detection model, acquiring a body text area in the acquired article image using image processing, and classifying the body text area into the body text using an image classification model. With this configuration, the information processing device D can improve the accuracy of area acquisition and classification.

[0061] According to this embodiment, the operation of the control unit 1 includes obtaining ruled lines from an article image, and detecting whether the article is a boxed article, horizontally set article, or normal article from at least one of the following: the presence or absence of a frame, the presence or absence of horizontal ruled lines, and the writing direction of the body text in the article image. With this configuration, the information processing device D can accurately determine the type of article and execute subsequent processing according to the type.

[0062] According to this embodiment, the operation of the control unit 1 includes, when a normal article is detected, executing a linking process that allows overlapping in the normal article. With this configuration, the information processing device D executes the linking process only when the article type is a normal article, thereby reducing the processing load.

[0063] Although the present disclosure has been described based on the drawings and embodiments, it should be noted that those skilled in the art can make various modifications and corrections based on the present disclosure. Therefore, it should be noted that these modifications and corrections are included in the scope of the present disclosure. For example, the functions included in each means or step can be rearranged so as not to be logically inconsistent, and multiple means or steps can be combined or divided into one.

[0064] For example, in the above embodiment, a program for executing all or part of the functions or processes of the information processing device D can be recorded in a computer-readable recording medium. The computer-readable recording medium includes a non-transitory computer-readable medium, such as a magnetic recording device, an optical disk, a magneto-optical recording medium, or a semiconductor memory. The program is distributed by, for example, selling, transferring, or lending a portable recording medium such as a DVD (Digital Versatile Disc) or a CD-ROM (Compact Disc Read Only Memory) on which the program is recorded. The program may also be distributed by storing the program in a server storage and transmitting the program from the server to another computer. The program may also be provided as a program product. The present disclosure can also be realized as a program executable by a processor. [Explanation of symbols]

[0065] D. Information processing device

Claims

1. An information processing device including a control unit, a communication unit, and a storage unit, the control unit comprising: Obtaining an article image; obtaining one or more regions from the article image; classifying at least one of the one or more regions as body text; recognizing characters of said body of text; Concatenating multiple possibly consecutive texts according to a predetermined reading order rule, while allowing overlaps; An information processing device that performs operations including the steps of:

2. 2. The information processing device according to claim 1, wherein the operation is When a body text that is overlapping and linked to a plurality of other body texts is detected, the information processing device determines whether each of the other plurality of body texts ends with a period, and determines a body text that does not end with a period as the immediately preceding body text.

3. 2. The information processing device according to claim 1, wherein the operation is When detecting a body text that is linked to a plurality of other bodies, the information processing device determines the immediately preceding body text from a contextual connection.

4. 2. The information processing device according to claim 1, wherein the operation is and determining that the body of the article is complete when it is determined that there is no body text that is linked to multiple other bodies.

5. 2. The information processing device according to claim 1, wherein the operation is acquiring and classifying at least one region of a headline, an image, a pictorial description, a table, an advertisement, and a page name section in the acquired article image using an object detection model; acquiring a body text region from the acquired article image using image processing, and classifying the body text region into a body text region using an image classification model; An information processing device comprising:

6. 2. The information processing device according to claim 1, wherein the operation is Obtaining a ruled line from the article image; detecting whether the article image is a boxed article, a horizontally set article, or a normal article based on at least one of the presence or absence of a frame, the presence or absence of a horizontal ruled line, and the writing direction of the body text; An information processing device comprising:

7. 7. The information processing apparatus according to claim 6, wherein the operation is an information processing device that, when a regular article is detected, executes the linking with overlapping allowed for the regular article.

8. An information processing method by an information processing device including a control unit, a communication unit, and a storage unit, Obtaining an article image; obtaining one or more regions from the article image; classifying at least one of the one or more regions as body text; recognizing characters of said body of text; Concatenating multiple possibly consecutive texts according to a predetermined reading order rule, while allowing overlaps; An information processing method comprising:

9. A program for causing a computer to function as the information processing device according to claim 1.

Images