Information processing apparatus, information processing method, and program

The information processing device simplifies the determination of relationships between image regions by calculating self-feature and relational feature amounts, eliminating the need for character recognition and enhancing the efficiency of form reading services.

JP2024162848APending Publication Date: 2024-11-21KK TOSHIBA +1
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023078783
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-11
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Existing techniques for determining relationships between multiple regions in image data require complex character recognition and semantic tag generation, making the process cumbersome and inefficient.

Method used

An information processing device that utilizes a reception, extraction, calculation, and determination unit to analyze image data, calculating self-feature and relational feature amounts to determine specific relationships between regions without the need for character recognition, using a pre-trained model to simplify the process.

Benefits of technology

Enables quicker and simpler determination of relationships between image regions, reducing the need for manual setting and enabling efficient heading recognition in form reading services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024162848000001_ABST
    Figure 2024162848000001_ABST
Patent Text Reader

Abstract

To determine, with a simpler configuration, the relationship between a plurality of areas included in image data.SOLUTION: An information processing apparatus comprises a receiving unit, an extraction unit, a calculation unit, and a determination unit. The receiving unit receives input of image data including a plurality of image areas having a specific relationship, and area information indicating a first image area in the image data. The extraction unit extracts one or more second image areas from the image data. The calculation unit calculates, using the image data, a relationship feature quantity indicating the relationship with the other image areas in the image data for the first image area indicated by the area information and calculates a self feature quantity indicating the feature of an area for each of the second image areas. The determination unit determines the second image area in the specific relationship with the first image area on the basis of the similarity between the relationship feature quantity and the self feature quantity.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] An embodiment of the present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] A reading service that reads information from image data of a standard form requires prior processing such as setting the area from which information is to be read (reading area). In addition, multiple reading areas in the image data may have specific relationships (such as hierarchical relationships) with each other.

[0003] As a technique for extracting information required for converting the layout of a document that includes rectangles, a technique for determining a hierarchical structure of a plurality of regions within image data has been proposed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-056796 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the above-mentioned techniques require, for example, a process of character recognition of image data and a process of converting the character recognition data to generate information indicating meaning (semantic tag information).

[0006] An object of the present invention is to provide an information processing device, an information processing method, and a program that are capable of determining the relationship between a plurality of areas included in image data with a simpler configuration. [Means for solving the problem]

[0007] An information processing device according to an embodiment includes a receiving unit, an extracting unit, a calculating unit, and a determining unit. The receiving unit receives input of image data including a plurality of image regions having a specific relationship, and region information indicating a first image region in the image data. The extracting unit extracts one or more second image regions from the image data. The calculating unit uses the image data to calculate a relation feature amount indicating a relationship between the first image region indicated by the region information and other image regions in the image data, and calculates a self feature amount indicating a region feature for each of the second image regions. The determining unit determines a second image region having a specific relationship with the first image region based on a similarity between the relation feature amount and the self feature amount. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating an example of a configuration of an information processing apparatus according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of an image area in image data of a form. [Diagram 3] FIG. 3 is a diagram illustrating an example of the calculated feature amount. [Figure 4] FIG. 4 is a diagram illustrating an example of a determination made by the determining unit. [Diagram 5] FIG. 5 is a diagram illustrating an example of a determination made by the determining unit. [Figure 6] FIG. 6 is a flowchart illustrating an example of the determination process according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a model structure. [Figure 8] FIG. 8 is a diagram illustrating an example of a model structure. [Figure 9] FIG. 9 is a diagram illustrating an example of a model structure. [Figure 10] FIG. 10 is a diagram illustrating an example of a model structure. [Figure 11] FIG. 11 is a diagram for explaining an example of calculating a plurality of relation feature amounts. [Figure 12] FIG. 12 is a diagram for explaining an example of the configuration of input data to be input to the GNN. [Figure 13]FIG. 13 is an explanatory diagram illustrating an example of a hardware configuration of the information processing apparatus according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of an information processing apparatus according to the present invention will be described in detail with reference to the accompanying drawings.

[0010] In the following, an example in which the embodiment is applied to a reading service that reads information from image data of a form will be mainly described, but the technology to which the embodiment can be applied is not limited to the reading service. The following embodiment can be applied to other technologies, services, systems, devices, etc. that utilize a function to determine whether a specific relationship exists between multiple areas included in image data.

[0011] As described above, in the reading service, it is necessary to set in advance the reading area from which information is read. As part of the advance setting, it may be necessary to set a caption for the reading area. Such setting work is performed, for example, by the user. On the other hand, if a function for setting a caption by analyzing image data can be realized, the setting work by the user will not be necessary.

[0012] In the heading setting function, it is necessary to estimate headings suitable for each of a plurality of regions having a specific relationship. The specific relationship is, for example, a hierarchical relationship in which a plurality of regions have a hierarchical structure. For such a function, it is desirable to determine the relationship between a plurality of regions with a simpler configuration and at a higher speed.

[0013] The information processing device of this embodiment calculates, for each of a plurality of regions in the image data, a feature of the region itself (self feature) and a feature indicating a relationship with other regions in the image data (relation feature). Then, using these feature, it is determined whether the plurality of regions have a specific relationship (such as a hierarchical relationship). A model trained in advance is used to calculate the feature. This makes it possible to determine the presence or absence of a specific relationship more quickly with a simpler configuration, eliminating the need for, for example, character recognition processing and processing for generating information indicating meaning.

[0014] Fig. 1 is a block diagram showing an example of a configuration of an information processing device 100 according to an embodiment. As shown in Fig. 1, the information processing device 100 includes a receiving unit 101, an extracting unit 102, a calculating unit 103, a determining unit 104, an output control unit 111, and a storage unit 121.

[0015] The receiving unit 101 receives input of various information used in the information processing device 100. For example, the receiving unit 101 receives input of image data IDA including a plurality of image areas having a specific relationship, and area information indicating an image area IRA (first image area) in the image data IDA.

[0016] The image data IDA is, for example, data obtained by reading a document using a reading device such as a scanner. The image area IRA is, for example, an image area designated as an area for which a heading is to be set. The area information is information indicating the position (coordinates) of the image area IRA within the image data IDA. Hereinafter, the image area IRA may be referred to as an input area.

[0017] The input area (area information) may be specified by, for example, a user, or an area extracted from the image data IDA may be set as the input area. In the latter case, not only the task of setting a heading but also the task of setting an image area to which a heading is to be set is unnecessary.

[0018] The extraction unit 102 extracts one or more image regions IRB (second image regions) from the image data IDA. The extracted image regions IRB are regions that are candidates for regions that are in a specific relationship with the input region (image region IRA). Hereinafter, the image regions IRB may be referred to as candidate regions.

[0019] When configured to extract an input area from image data IDA as described above, for example, extraction unit 102 extracts one or more image areas from image data IDA, sets one of the extracted image areas as an input area, and sets information indicating the position of the input area as area information.

[0020] The method of extracting a region (candidate region, input region) from image data by the extraction unit 102 may be any conventionally used method, but for example, the following method can be applied. (M1) Random sampling from the neighborhood of the input domain (M2) Rule-based method using lines (M3)Semantic Segmentation using DNN (Deep Neural Network) (M4) Bounding box-based object detection using DNN (e.g., YOLO: You Only Look Once, SSD: Single Shot multibox Detector, etc.)

[0021] In (M3)~(M4), the DNN is pre-trained so that it can extract candidate regions.

[0022] Fig. 2 is a diagram showing an example of an image area in image data of a form. Fig. 2 shows an example of image data 210 including an entry area for entering a date of birth. The image data 210 may be data including two or more entry areas.

[0023] 2, image data 210 includes image regions 211 to 214. The image regions 211 to 214 correspond to the following regions, respectively. · Image area 211: An entry area for entering the day of the date of birth · Image area 212: An area where the character "Day", which corresponds to the heading of image area 211, is output · Image area 213: An entry area for entering the date of birth · Image area 214: An area where the character "Date of Birth", which corresponds to the heading of image area 213, is output

[0024] Image area 211 and image area 212 can be interpreted as having a specific relationship, namely, an entry area related to "Day" and an area where the heading is output. Note that by considering the entry area as the lower level and the area where the heading is output as the upper level, it can also be interpreted that the two are in a hierarchical relationship.

[0025] Similarly, image area 213 and image area 214 can be interpreted as having a specific relationship, namely, an entry area related to "Date of Birth" and an area where the heading is output.

[0026] Furthermore, image area 214 corresponds to an area that outputs the "Date of Birth" including the "Day" output to image area 212. Therefore, it can be interpreted that image area 214 is an upper-level area of image area 212. In other words, it can be interpreted that image area 214 and image area 212 are in a hierarchical relationship. In this embodiment, a plurality of image areas having such a specific relationship can be determined.

[0027] Note that image areas 211 to 214 may be input areas or candidate areas. For example, there may be a case where image area 211 is designated as the input area and image areas 212 to 214 are extracted as candidate areas. Also, for example, there may be a case where image area 213 is designated as the input area and image areas 211, 212, and 214 are extracted as candidate areas.

[0028] Returning to FIG. 1, the calculation unit 103 calculates the feature amount of the image area. For example, the calculation unit 103 calculates the self-feature amount and the relationship feature amount for each of the input area and the candidate area using the input image data IDA.

[0029] The self feature indicates the feature of the image region itself, and the relation feature indicates the relationship between the image region and other image regions in the image data IDA.

[0030] For example, the calculation unit 103 inputs the image data IDA and the region information, and calculates the self feature and the related feature using a model that outputs the self feature and the related feature. The model may have any structure, and is, for example, a neural network model constructed by machine learning.

[0031] The model is constructed by prior learning using learning data. For example, the model is trained so that the similarity between the self feature of an image region that has a specific relationship with the image region IRA indicated by the region information among image regions included in the image data and the related feature output from the model is greater than the similarity between the self feature of an image region that does not have a specific relationship with the image region IRA and the related feature output from the model.

[0032] In other words, the model is trained to output, for each image region, the feature amount (self feature amount) of the image region itself as well as the feature amount (self feature amount) of another image region that has a specific relationship with the image region as a related feature amount. Details of the model that can be used for the feature amount extraction process by the calculation unit 103 will be described later.

[0033] FIG. 3 is a diagram showing an example of features calculated for the image regions 211 to 214 in FIG. 2. As shown in FIG. 3, the calculation unit 103 calculates an self feature and a related feature for each of the image regions 211 to 214. Two feature values ​​with a common hatching indicate similarity or agreement. For example, the related feature value of the image region 211 has a common hatching with the self feature value of the image region 212. This means that the image regions 211 and 212 are in a specific relationship. In this embodiment, the specific relationship can be determined based on the correspondence (similarity or non-similarity) between the self feature value and the related feature value in this manner.

[0034] Returning to FIG. 1, the determination unit 104 determines an image region IRB that has a specific relationship with the image region IRA based on the similarity between the related feature and the self feature. The feature is expressed, for example, as a vector. In this case, the determination unit 104 can calculate the similarity between the related feature and the self feature by using the cosine similarity between the two vectors and the inverse of the distance between the two vectors.

[0035] When two vectors are denoted by u and v, the cosine similarity between the vectors u and v is expressed, for example, by the following equation (1).

[0036]

number

[0037] The reciprocal of the distance between vectors can be, for example, the reciprocal of the L2 distance, as shown in the following formula (2). Note that the distance is not limited to the L2 distance.

[0038]

number

[0039] The method of calculating the similarity is not limited to the above, and any conventionally used method may be used. For example, instead of the reciprocal of the distance, the determination unit 104 may calculate a value obtained by inverting the sign of the distance (e.g., multiplying it by minus 1) as the similarity.

[0040] Furthermore, when using a similarity measure in which a smaller value indicates greater similarity, the distance between vectors itself or the inverse of the cosine similarity may be used as the similarity measure.

[0041] The determination unit 104 uses the similarity calculated as described above to determine an image region IRB that has a specific relationship with the image region IRA from among the one or more extracted image regions IRB. For example, the determination unit 104 determines an image region IRB whose similarity is equal to or greater than a threshold (similarity threshold) as an image region IRB that has a specific relationship with the image region IRA.

[0042] Furthermore, the determination unit 104 may recursively search for other image areas IRB that are in a specific relationship with the image area IRA, for the image area IRB that is determined to be in a specific relationship with the image area IRA.

[0043] The calculation unit 103 calculates the self feature amount and the related feature amount for the input region (image region IRA) and all the extracted candidate regions (image regions IRB). Therefore, the determination unit 104 can use the calculated self feature amount and the related feature amount for a recursive search.

[0044] For example, the determination unit 104 further determines other image areas IRB that have a specific relationship with the image area IRB based on the similarity between the relational feature calculated for the image area IRB that has been determined to have a specific relationship with the image area IRA and the self feature of the other image area IRB.

[0045] In addition, when the input region is not used later, such as when a recursive search is not performed, the calculation unit 103 may be configured not to calculate the self feature of the input region or the relation feature of the candidate region.

[0046] 4 and 5 are diagrams showing an example of a determination made by the determining unit 104. Fig. 4 shows an example of a determination made when the image area 211 is input as an input area (image area IRA).

[0047] In this case, the determination unit 104 first determines that the image region IRB having an auto-feature whose similarity to the relation feature of the image region 211 is equal to or greater than a threshold is the image region 212. When performing a recursive search, the determination unit 104 determines that the image region IRB having an auto-feature whose similarity to the relation feature of the image region 212 is equal to or greater than a threshold is the image region 214.

[0048] As a result, the determination unit 104 can determine an image area 212 as an area in a higher hierarchy than the image area 211, and can further determine an image area 214 as an area in a higher hierarchy than the image area 212. The determination unit 104 may output information indicating such a hierarchical structure.

[0049] 5 shows an example of a determination when the image area 213 is input as an input area (image area IRA). In this case, the determination unit 104 determines that the image area IRB having an auto-feature whose similarity to the relation feature of the image area 213 is equal to or greater than a threshold is the image area 214. Although it is possible to perform a recursive search, in the example of FIG. 5, since there is no image area IRB having an auto-feature whose similarity to the relation feature of the image area 214 is equal to or greater than a threshold, an image area corresponding to a higher layer cannot be found.

[0050] 4 and 5 are examples of searching for image areas in a higher hierarchy starting from the input area. The hierarchical relationship may be determined by searching for image areas in a lower hierarchy starting from the input area. For example, when an area corresponding to the title (form name) of a form is specified as an input area, image areas corresponding to one or more writing areas included in the form having the title corresponding to the input area may be searched for as areas corresponding to a lower hierarchy of the input area. The hierarchical relationship may also be searched in both upper and lower directions.

[0051] 1, the output control unit 111 controls the output of various information used by the information processing device 100. For example, the output control unit 111 outputs the determination result (structure information) by the determination unit 104.

[0052] Each of the above units (reception unit 101, extraction unit 102, calculation unit 103, judgment unit 104, and output control unit 111) is realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) execute a program, that is, by software. Each of the above units may be realized by a processor such as a dedicated IC (Integrated Circuit), that is, by hardware. Each of the above units may be realized by using a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.

[0053] The storage unit 121 stores various types of information used in the information processing device. For example, the storage unit 121 stores information accepted by the accepting unit 101 (image data IDA, area information, etc.).

[0054] The storage unit 121 can be configured from any commonly used storage medium, such as a flash memory, a memory card, a RAM (Random Access Memory), a HDD (Hard Disk Drive), or an optical disk.

[0055] The information processing device 100 may be physically configured by one device, or may be physically configured by multiple devices. For example, the information processing device 100 may be constructed on a cloud environment. Furthermore, each unit in the information processing device 100 may be provided in a distributed manner in multiple devices.

[0056] Next, a description will be given of a determination process performed by the information processing apparatus 100 according to the embodiment. Fig. 6 is a flowchart showing an example of the determination process according to the embodiment.

[0057] The receiving unit 101 receives input of image data IDA and area information indicating an input area (image area IRA) in the image data IDA (step S101). The extracting unit 102 extracts one or more candidate areas from the image data IDA (step S102). The calculating unit 103 calculates the self feature and the related feature for each of the input area and the candidate area (step S103).

[0058] The determination unit 104 sets the input region as a region of interest (step S104). The region of interest refers to an image region that is a target for determining other image regions that have a specific relationship with the input region.

[0059] The determination unit 104 calculates the similarity between the relation feature of the region of interest and the self feature of the candidate region (step S105). The determination unit 104 determines whether there is a candidate region whose similarity is equal to or greater than a threshold (step S106).

[0060] If there is a candidate region whose similarity is equal to or greater than the threshold (step S106: Yes), the determination unit 104 stores the candidate region whose similarity is equal to or greater than the threshold in, for example, the storage unit 121 as a region having a specific relationship with the region of interest, and sets it as a new region of interest (step S107). The determination unit 104 also returns to step S105 and repeats the process for the new region of interest. This enables recursive search.

[0061] If there is no candidate region with a similarity equal to or greater than the threshold (step S106: No), the output control unit 111 outputs structural information indicating the specific relationship obtained up to that point by the judgment unit 104 (step S108), and then ends the judgment process.

[0062] The structural information can be used, for example, in the task of setting a heading. For example, the information processing device 100 (or another device having a function of setting a heading) determines an area determined to be in a higher hierarchy than the input area as a heading area, and identifies the heading by performing character recognition on the heading area.

[0063] The function using the structural information is not limited to setting headings, and may be any other function. For example, it may be a function that displays information indicating the relationship between multiple areas based on structural information indicating the relationship between multiple areas included in the document to be edited. Such a function can, for example, make the task of editing a document within an area more efficient.

[0064] Next, a model that can be used for feature extraction processing will be described in detail below.

[0065] A model 710 inputs image data IDA and region information indicating an image region within the image data IDA, and outputs an auto-feature 702a and a relational feature 702b. In Fig. 7, image data 701 corresponds to image data IDA. The image region specified by the region information may be an input region or a candidate region.

[0066] The model 710 mainly includes the following three functions, and outputs a self feature 702a and a relation feature 702b as final outputs. (1) Abstraction (2) Cutting out (3) Consolidation

[0067] As described below, each of the three functions can be realized by a number of different methods. Depending on the combination of the methods, the model 710 can have a number of different configurations. Note that some methods cannot be combined.

[0068] In the abstraction (1), a feature amount 711 is output that is an abstraction of the input image data 701. The feature amount 711 is expressed in a data format such as a third-order tensor having a dimension in the depth direction in addition to width and height. The abstraction is realized, for example, by the following method. (1-1) No abstraction is performed (image data 701 is output as feature quantity 711). (1-2) A feature extractor that has been pre-trained using a database (such as ImageNet) prepared in advance is used. The feature extractor is, for example, a deep neural network (DNN). (1-3) It is realized as an abstraction function among the functions of the model 710 trained as a single model. For example, the model 710 is a DNN, and the abstraction function corresponds to the function of a part of the layer of this DNN.

[0069] In addition, (1-2) takes longer to infer than (1-1), but allows for more accurate processing. Also, (1-3) takes longer to learn than (1-2), but allows for more accurate processing.

[0070] In the extraction of (2), a part of feature quantity 712 extracted from feature quantity 711 is output. The extraction of (2) is realized, for example, by the following method. (2-1) From the feature amount 711, a feature amount 712 corresponding to a specified image region (input region, candidate region) is extracted. (2-2) A feature amount 712 corresponding to a peripheral region of the specified image region is extracted from the feature amount 711. The peripheral region is, for example, a region within a certain distance from the specified image region.

[0071] By extracting the feature amount 712 corresponding to the surrounding area as in (2-2), more accurate processing may be possible.

[0072] In the aggregation of (3), the self feature 702a and the relation feature 702b are output by aggregating the feature 712. The aggregation of (3) is realized, for example, by the following method. (3-1) A representative vector is calculated by pooling the feature 712, and the representative vector is output as the self feature 702a and the relation feature 702b. The pooling is, for example, average pooling and maximum value pooling. Note that if abstraction is not performed (above (1-1)), features to be pooled cannot be obtained, and therefore (3-1) cannot be combined. (3-2) The feature 712 is converted into the self feature 702a and the relation feature 702b by the DNN and output. This method takes time for inference and DNN learning, but enables more accurate processing.

[0073] Note that, as an input to the model, information indicating the position of each element (pixel) included in the image data 701 may be added. Figure 8 is a diagram showing an example of the structure of such a model 710b.

[0074] 8, model 710b receives positional information 801a and 801b in addition to image data IDA and region information. Information 801a is, for example, information including elements corresponding to each pixel of image data 701, with element values ​​varying from a maximum value (e.g., 1) to a minimum value (e.g., 0) from the leftmost element to the rightmost element. Information 801b is, for example, information including elements corresponding to each pixel of image data 701, with element values ​​varying from a maximum value (e.g., 1) to a minimum value (e.g., 0) from the topmost element to the bottommost element.

[0075] The information 801a and 801b may be generated by positional encoding. Also, only one of the information 801a and 801b may be used.

[0076] The feature amount 711b is an abstraction of the input image data 701 and information 801a and 801b.

[0077] The information 801a and 801b may be added in the abstraction function. For example, the information 801a and 801b may be added in any of the intermediate layers of the DNN that realizes the abstraction.

[0078] A specific example of the model will be further described. Fig. 9 is a diagram showing an example of the model (model 710c). Fig. 9 shows an example of the model 710c in which the techniques of the three functions are combined as follows. · Abstraction: (1-1) No abstraction is performed. Extraction: (2-2) Extract the features corresponding to the surrounding area. · Aggregation: (3-2) Transform using DNN.

[0079] 9, region information indicating a peripheral region 902 of an image region 901 is input to the model 710c. Note that the model 710c may be configured to receive region information indicating the image region 901 and to calculate region information of the peripheral region 902 of the input image region 901 inside the model 710c.

[0080] The model 710c is trained to input image data and region information and output the self feature 702a and the relation feature 702b for the region indicated by the region information.

[0081] Fig. 10 is a diagram showing another example of the model (model 710d) in which the three functional methods are combined as follows. Abstraction: (1-3) Functions of abstraction of the single model 710d Extraction: (2-1) Extract the features corresponding to the specified region. Aggregation: (3-1) Calculate representative vectors by pooling

[0082] In the example of FIG. 10, image data 701 and area information of an image area 1001 are input to a model 710d. The model 710d includes a DNN 1011 that corresponds to an abstraction function. The DNN 1011 is trained to output, for example, an autofeature 1012a of the entire image data 701 and a relation feature 1012b of the entire image data 701 from the image data 701. The autofeature 1012a and the relation feature 1012b of the entire image data 701 are, for example, third-order tensors having dimensions of width, height, and depth that correspond to the image data 701 in the vertical and horizontal directions.

[0083] Furthermore, the model 710d cuts out a third-order tensor surrounded by a position 1021 corresponding to the region information of the image region 1001 from the self feature 1012a, and aggregates the width and height directions by pooling, thereby outputting the self feature 702a. Similarly, the model 710d cuts out a third-order tensor surrounded by a position 1021 from the relation feature 1012b, and aggregates the width and height directions by pooling, thereby outputting the relation feature 702b.

[0084] Next, an example of a model learning method will be described. As described above, the model is learned so that the similarity between the self feature of an image area that has a specific relationship with the image area IRA among the image areas included in the image data and the related feature output from the model is greater than the similarity between the self feature of an image area that does not have a specific relationship with the image area IRA and the related feature output from the model.

[0085] The learning is performed, for example, by minimizing a loss function designed according to the following procedure. For each combination of the region of interest and all other candidate regions, the similarity between the self feature of the candidate region and the relation feature of the region of interest is calculated. Design a loss function that increases the similarity between combinations that have a specific relationship and decreases the similarity between combinations that do not have a specific relationship.

[0086] For example, a loss function L as shown in the following formula (3) can be used. Note that the function Sim in formula (3) is a function for calculating the similarity as in formula (1) or (2).

number

[0087] The definitions of each variable in equation (3) are as follows: N: Number of regions R i : i-th region (1≦i≦N) R j :jth region (1≦j≦N) φ self : A function to calculate autofeature values ​​from an image region φ rel : Function to calculate related features from image regions σ i,j :R j R i If it has a specific relationship with, it is set to 1, otherwise it is set to -1 (correct data).

[0088] In addition, the function for obtaining the self-feature in the model is a function φ self In addition, the function of the model to obtain the related feature is a function φ rel is equivalent to.

[0089] While the above description has been given of an example in which one related feature amount is calculated for one image region, the calculation unit 103 may calculate a plurality of related feature amounts from one image region.

[0090] Fig. 11 is a diagram for explaining an example of calculating a plurality of relation feature amounts. Image data 1100 shown in Fig. 11 is an example of an image including elements represented in a table format, for example.

[0091] For example, areas 1101 and 1102 are areas where characters (headings, etc.) corresponding to columns in a table format are output. Area 1111 is an area where characters (headings, etc.) corresponding to rows in a table format are output. Area 1121 is an entry area for entering values ​​corresponding to areas 1101 and 1111. Area 1122 is an entry area for entering values ​​corresponding to areas 1102 and 1111. Therefore, for example, area 1121 can be interpreted as having a specific relationship with area 1101 as well as a specific relationship with area 1111.

[0092] Taking such a case into consideration, the calculation unit 103 calculates the self feature amount 702a for the region 1111, and calculates two related feature amounts 702b corresponding to two directions. The two directions are, for example, the column direction (vertical direction) and the row direction (horizontal direction). Note that the calculated multiple related feature amounts are not limited to these. The calculation unit 103 may calculate three or more related feature amounts.

[0093] When multiple related features are used, a loss function including the similarity between the multiple related features is used during learning. For example, σ in Eq. (3) i,j A loss function is used that includes multiplication of and similarity (Sim) for each of multiple relation features.

[0094] This makes it possible to determine the relationship between multiple areas even for a wider variety of forms, including forms in which multiple headings are set for one entry area.

[0095] Next, another example of the method for calculating the similarity will be described. The similarity may be calculated using a model that is trained to calculate the similarity (hereinafter, referred to as a similarity calculation model).

[0096] The similarity calculation model is, for example, a DNN. The similarity calculation model is trained to input, for example, the relation feature of the attention region and the self feature of the candidate region, and output the similarity. The input may be the relation feature and the self feature of the attention region, and the self feature of the candidate region.

[0097] The similarity calculation model may be trained together with the model used for calculating the feature amount by the calculation unit 103, or may be trained in advance separately from the model. If such a similarity calculation model is used, the calculation time increases, but more accurate processing is possible.

[0098] The similarity may be calculated using a graph neural network (GNN). Fig. 12 is a diagram for explaining a configuration example of input data to be input to the GNN. In this example, the input data is configured such that the self features 1201 to 1203 of each image region are nodes, and a feature obtained by integrating (for example, connecting) the relational features of two image regions is an edge connecting the two image regions.

[0099] A GNN is trained to receive input data with a graph structure as shown in Fig. 12 and output the feature quantities of the input data. For example, a GNN is trained to output a feature quantity of 1 for an edge corresponding to two image regions that have a specific relationship, and to output a feature quantity of 0 for an edge corresponding to two image regions that do not have a specific relationship. Such a feature quantity can be interpreted as information that represents the similarity between the features of the two image regions.

[0100] The determination unit 104 inputs input data with a graph structure as shown in Fig. 12 to the GNN and obtains the feature amount (similarity) output by the GNN. By calculating the similarity using the GNN, the calculation time increases, but more accurate processing is possible.

[0101] In this manner, according to the embodiment, whether or not a plurality of regions have a specific relationship (such as a hierarchical relationship) is determined using the self feature and the relation feature calculated for each of the plurality of regions in the image data. This makes it possible to determine the presence or absence of a specific relationship with a simpler configuration.

[0102] Furthermore, by using information (structural information) indicating the determination result, for example, a function for setting a heading in a document reading service can be realized. In setting a heading, areas with different granularity may be specified as the area in which to set the heading. For example, as an area in which to write the date of birth, one area may be specified for the entire date of birth, or areas may be specified separately for the year, month, and day. According to the above embodiment, in either of these cases, other areas having a specific relationship can be efficiently searched for from the specified area.

[0103] Next, a hardware configuration of the information processing apparatus according to the embodiment will be described with reference to Fig. 13. Fig. 13 is an explanatory diagram illustrating an example of a hardware configuration of the information processing apparatus according to the embodiment.

[0104] The information processing device of the embodiment includes a control device such as a CPU 51, a storage device such as a ROM (Read Only Memory) 52 and a RAM 53, a communication I / F 54 that connects to a network and performs communication, and a bus 61 that connects each unit.

[0105] The programs executed by the information processing apparatus according to the embodiment are provided in a state that they are pre-installed in the ROM 52 or the like.

[0106] The program executed by the information processing device of the embodiment may be configured to be provided as a computer program product by being recorded in an installable or executable format on a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), or a DVD (Digital Versatile Disk).

[0107] Furthermore, the program executed by the information processing apparatus of the embodiment may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the program executed by the information processing apparatus of the embodiment may be configured to be provided or distributed via a network such as the Internet.

[0108] The programs executed by the information processing device of the embodiment can cause the computer to function as each unit of the information processing device described above. In this computer, the CPU 51 can read the programs from a computer-readable storage medium onto a main storage device and execute the programs.

[0109] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0110] 100 Information processing device 101 Reception 102 Extraction part 103 Calculation section 104 Judgment section 111 Output control section 121 Storage section

Claims

1. a receiving unit that receives input of image data including a plurality of image regions having a specific relationship and region information indicating a first image region within the image data; an extractor for extracting one or more second image regions from the image data; a calculation unit that calculates, using the image data, a relation feature amount that indicates a relation between the first image region indicated by the region information and other image regions in the image data, and calculates, for each of the second image regions, an auto-feature amount that indicates a region feature; a determination unit that determines the second image area that is in the specific relationship with the first image area based on a similarity between the relation feature amount and the self feature amount; An information processing device comprising:

2. the calculation unit calculates the related feature amount using a model that outputs the related feature amount based on the input image data and the region information; the model is trained in advance such that the similarity between the self feature of an image region that has the specific relationship with the first image region among image regions included in the image data and the related feature output from the model is greater than the similarity between the self feature of an image region that does not have the specific relationship with the first image region and the related feature output from the model. The information processing device according to claim 1 .

3. the model determines the related feature amount of the entire image data, and extracts the related feature amount of the first image region from the related feature amount of the entire image data based on the region information; The information processing device according to claim 2 .

4. the model further inputs information indicating the position of each element included in the image data, and outputs the relation feature amount; The information processing device according to claim 2 .

5. the calculation unit calculates the relation feature amount for the second image area determined to be in the specific relationship with the first image area; the determination unit further determines, based on a similarity between the relation feature amount and the self feature amount calculated for the second image area determined to be in the specific relationship with the first image area, the second image area being in the specific relationship with the first image area; The information processing device according to claim 1 .

6. The specific relationship is a hierarchical relationship. The information processing device according to claim 1 .

7. The calculation unit calculates a plurality of the related feature amounts for the first image region indicated by the region information. The information processing device according to claim 1 .

8. the determination unit calculates the similarity using a graph neural network in which the self feature is a node and the relation feature is an edge. The information processing device according to claim 1 .

9. An information processing method executed by an information processing device, a receiving step of receiving input of image data including a plurality of image regions having a specific relationship and region information indicating a first image region within the image data; extracting one or more second image regions from the image data; a calculation step of calculating, using the image data, a relation feature amount indicating a relation between the first image region indicated by the region information and other image regions in the image data, and calculating, for each of the second image regions, an auto-feature amount indicating a region feature; a determination step of determining the second image area having the specific relationship with the first image area based on a similarity between the relation feature amount and the self feature amount; An information processing method comprising:

10. On the computer, a receiving step of receiving input of image data including a plurality of image regions having a specific relationship and region information indicating a first image region within the image data; extracting one or more second image regions from the image data; a calculation step of calculating, using the image data, a relation feature amount indicating a relation between the first image region indicated by the region information and other image regions in the image data, and calculating, for each of the second image regions, an auto-feature amount indicating a region feature; a determination step of determining the second image area having the specific relationship with the first image area based on a similarity between the relation feature amount and the self feature amount; A program for executing.

Citation Information

Patent Citations

  • Structure recognition system, structure recognition device, structure recognition method, and program

    JP2021056796A