Information processing device, information processing method, and program

Abstract

To improve comparison accuracy of a plurality of mediums.SOLUTION: An information processing device 10 includes: an image information acquisition unit 122 which acquires image information PINF indicating an image of a receipt REC; a character recognition unit 123 which generates character information CINF indicating a plurality of characters contained in an image indicated by the image information PINF; a storage control unit 124 which stores the image information PINF and the character information CINF in association with each other in a storage device 140; a character comparison unit 126 which executes character comparison for comparing a plurality of characters indicated by the character information CINF with a plurality of characters indicated by the other character information CINF stored in the storage device 140; and an image comparison unit 127 which executes image comparison, when at least one piece of the other character information CINF is the specific character information CINF satisfying a prescribed condition regarding a result of the character comparison, for comparing an image indicated by the image information PINF with an image indicated by specific image information PINF associated with the specific character information CINF.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Character recognition technology that recognizes characters contained in an image is used in various fields. For example, in a campaign application system that accepts applications for a campaign using receipts issued from a cash register, character recognition technology is used to prevent duplicate applications using the same receipt (see, for example, Patent Document 1). In this type of campaign application system, for example, duplicate applications are checked by comparing receipt display items contained in receipt image data of a receipt photographed for a new application with receipt display items contained in receipt image data of an application that has already been submitted. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-57676 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, it is difficult to completely prevent erroneous recognition in character recognition processing. For this reason, for example, in a conventional campaign application system, when extracting receipt display items from receipt image data, the receipt display items may be erroneously recognized. In this case, the accuracy of comparison between the receipt display items included in the newly applied receipt image data and the receipt display items included in the already applied receipt image data decreases. For this reason, in a system that uses the comparison results of multiple receipts, it is required to improve the accuracy of comparison of multiple receipts. Note that in the above explanation, a campaign application system is exemplified, but the same problem as the above also exists when character recognition processing is used in a system other than a campaign application system (for example, a system that creates a household account book using an image of a photographed receipt). Furthermore, it is not limited to a system that uses receipts, and for example, a system that converts documents into electronic data and registers them also has the same problem as the above because it is necessary to prevent duplicate registrations. In other words, it is required to improve the accuracy of comparison of multiple media on which multiple characters are written. [Means for solving the problem]

[0005] An information processing device according to a preferred embodiment of the present invention includes an acquisition unit that acquires image information indicating an image including a plurality of characters written on a medium, a recognition unit that recognizes a plurality of characters included in the image indicated by the image information and generates character information indicating the recognized plurality of characters, a storage control unit that associates the image information and the character information with each other and stores them in a storage device, a character comparison unit that performs a character comparison to compare a plurality of characters indicated by the character information with a plurality of characters indicated by other character information stored in the storage device, and an image comparison unit that performs an image comparison to compare an image indicated by the image information with an image indicated by specific image information associated with the specific character information, among the other image information stored in the storage device, when at least one of the other character information is specific character information that satisfies a predetermined condition regarding the result of the character comparison.

[0006] Moreover, an information processing method according to a preferred embodiment of the present invention acquires image information showing an image including a plurality of characters written on a medium, recognizes a plurality of characters included in the image shown by the image information, generates character information showing the recognized plurality of characters, associates the image information and the character information with each other and stores them in a storage device, performs a character comparison in which the plurality of characters shown by the character information are compared with a plurality of characters shown by other character information stored in the storage device, and if at least one of the other character information is specific character information that satisfies a predetermined condition regarding the result of the character comparison, performs an image comparison in which the image shown by the image information is compared with an image shown by specific image information associated with the specific character information among the other image information stored in the storage device.

[0007] Furthermore, a program according to a preferred embodiment of the present invention causes a processor to function as an acquisition unit that acquires image information indicating an image including a plurality of characters written on a medium, a recognition unit that recognizes a plurality of characters included in the image indicated by the image information and generates character information indicating the recognized plurality of characters, a storage control unit that associates the image information and the character information with each other and stores them in a storage device, a character comparison unit that performs a character comparison to compare a plurality of characters indicated by the character information with a plurality of characters indicated by other character information stored in the storage device, and an image comparison unit that performs an image comparison to compare an image indicated by the image information with an image indicated by specific image information associated with the specific character information, among the other image information stored in the storage device, when at least one of the other character information is specific character information that satisfies a predetermined condition regarding the result of the character comparison. Effect of the Invention

[0008] According to the present invention, it is possible to improve the accuracy of comparison between a plurality of media having a plurality of characters written thereon. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is an explanatory diagram for explaining an overview of an information processing system according to a first embodiment. [Diagram 2] FIG. 11 is an explanatory diagram illustrating an example of a receipt comparison process according to the first embodiment. [Diagram 3] FIG. 10 is an explanatory diagram for explaining an example of a receipt comparison process according to a comparative example. [Figure 4] 4 is a flowchart showing an example of an operation of the information processing device shown in FIG. [Diagram 5] FIG. 11 is an explanatory diagram for explaining an overview of an information processing system according to a second embodiment. [Figure 6] FIG. 11 is an explanatory diagram for explaining an example of return information. [Figure 7] 6 is a flowchart showing an example of an operation of the information processing device shown in FIG. 5. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. In each drawing, the dimensions and scale of each part are appropriately different from the actual ones. In addition, the embodiments described below are preferred examples of the present invention, and therefore various technically preferable limitations are attached, but the scope of the present invention is not limited to these embodiments unless otherwise specified in the following description to the effect that the present invention is limited.

[0011] [1. First embodiment] First, an example of an overview of an information processing system 1 according to the first embodiment will be described with reference to FIG.

[0012] FIG. 1 is an explanatory diagram for explaining an overview of an information processing system 1 according to the first embodiment.

[0013] In this embodiment, it is assumed that the information processing system 1 is a campaign application system that accepts applications for a campaign using a receipt REC issued from a cash register. The receipt REC is an example of a "medium". The information processing system 1 has, for example, an information processing device 10, a service server 20, and a terminal device 30. In the example shown in FIG. 1, each of the information processing device 10, the service server 20, and the terminal device 30 is connected to a network NW. The network NW may include the Internet and a local area network. For example, the network NW includes one or both of a wired network and a wireless network. In addition, the connection between a plurality of elements including the information processing device 10 and the like and the network NW may be, for example, a connection that enables communication between the plurality of elements, and may be a connection using either a wired or wireless connection, or may be a connection using both a wired and wireless connection.

[0014] For example, the information processing device 10 and the service server 20 are connected to each other so as to be able to communicate with each other via a network NW. Similarly, the service server 20 and the terminal device 30 are connected to each other so as to be able to communicate with each other via the network NW. Note that the information processing device 10 may be connected to the service server 20 so as to be able to communicate with each other via a dedicated line instead of the network NW. In addition, in this embodiment, it is assumed that the terminal device 30 is not permitted to connect to the information processing device 10, but the terminal device 30 may be connected to the information processing device 10 so as to be able to communicate with each other via the network NW.

[0015] In the following, unless otherwise specified, communication (such as sending and receiving information) between multiple elements connected to the network NW means communication between multiple elements via the network NW. For example, unless otherwise specified, the terminal device 30 sending information to the service server 20 means that the terminal device 30 sends information to the service server 20 via the network NW.

[0016] Any information processing device connectable to the network NW can be adopted as the terminal device 30. Specifically, the terminal device 30 may be, for example, a stationary information device such as a personal computer, or a portable information terminal such as a notebook personal computer, a tablet terminal, or a smartphone. In this embodiment, it is assumed that the terminal device 30 is a smartphone.

[0017] The terminal device 30 transmits, for example, image information PINF indicating an image including multiple characters written on a receipt REC required for entering a campaign to the service server 20. The image including multiple characters written on the receipt REC is, for example, an image of the receipt REC obtained by photographing the receipt REC with a camera. Note that the image including multiple characters written on the receipt REC is not limited to an image of the receipt REC photographed by a camera. For example, the image including multiple characters written on the receipt REC may be an image of the receipt REC obtained by reading the receipt REC with an image scanner. Hereinafter, the image including multiple characters written on the receipt REC is also simply referred to as the image of the receipt REC.

[0018] For example, the terminal device 30 transmits image information PINF indicating an image of a receipt REC captured by a camera included in the terminal device 30 to the service server 20. Note that the image information PINF transmitted from the terminal device 30 to the service server 20 may be image information generated by a device external to the terminal device 30 (e.g., an optical device such as a camera or image scanner). In this case, the terminal device 30 acquires image information PINF generated by the external device and transmits the acquired image information PINF to the service server 20. Hereinafter, the receipt REC used to apply for a campaign may be referred to as the receipt REC indicated by the image information PINF.

[0019] Any information processing device connectable to the network NW can be used as the service server 20. The service server 20 accepts applications for a campaign using, for example, a receipt REC indicated by image information PINF received from the terminal device 30. In this embodiment, the service server 20 performs a double application check to prevent double applications using the same receipt REC as an already accepted receipt REC.

[0020] For example, in a double application check, the service server 20 transmits image information PINF received from the terminal device 30 (image information PINF showing an image of the currently accepted receipt REC) to the information processing device 10. Although details will be described later, upon receiving the image information PINF, the information processing device 10 executes a receipt comparison process that compares the currently accepted receipt REC by the service server 20 with other receipts REC that have already been accepted. Then, return information RINF showing the results of the receipt comparison process is transmitted from the information processing device 10 to the service server 20.

[0021] The service server 20 determines, for example, whether the application for the campaign using the currently accepted receipt REC (hereinafter also referred to as the current application) is a duplicate application based on the return information RINF. For example, if the return information RINF indicates that at least one image of the already accepted receipt REC is similar to the image of the currently accepted receipt REC, the service server 20 determines that the current application is a duplicate application. Then, for example, if the service server 20 determines that the current application is a duplicate application, it transmits response information AINF indicating that the current application is a duplicate application to the terminal device 30. Also, for example, if the service server 20 determines that the current application is not a duplicate application, it transmits response information AINF indicating that the application for the campaign has been normally accepted to the terminal device 30.

[0022] Any information processing device connectable to a network NW can be used as the information processing device 10. For example, the information processing device 10 executes character recognition processing to recognize characters included in an image using optical character recognition (OCR) technology. In this embodiment, it is assumed that the image to be subjected to the character recognition processing is an image of a receipt REC. For example, in this embodiment, the information processing device 10 uses optical character recognition technology to convert information written on the receipt REC into electronic data.

[0023] The information processing device 10 includes, for example, a processing device 120 that controls each unit of the information processing device 10, a storage device 140 that stores various information, and a communication device 160.

[0024] The communication device 160 is hardware for communicating with an external device located outside the information processing device 10 via one or both of a wired network and a wireless network.

[0025] The storage device 140 includes, for example, one or both of a volatile memory such as a random access memory (RAM) that functions as a working area for the processing device 120 and a non-volatile memory such as an electrically erasable programmable read-only memory (EEPROM) that stores various information such as the control program PG1. The storage device 140 may be detachable from the information processing device 10. Specifically, the storage device 140 may be a storage medium such as a memory card that is detachable from the information processing device 10. The storage device 140 may also be, for example, a storage device (e.g., online storage) that is communicatively connected to the information processing device 10 via a network NW or the like.

[0026] The processing device 120 is a processor that controls the entire information processing device 10, and is configured to include, for example, one or more CPUs (Central Processing Units). For example, the processing device 120 executes a control program PG1 stored in the storage device 140 to function as an image information acquisition unit 122, a character recognition unit 123, a storage control unit 124, a character comparison unit 126, an image comparison unit 127, and an output control unit 128. Note that the control program PG1 may be transmitted from another device via the network NW.

[0027] Furthermore, for example, when the processing device 120 is configured to include multiple CPUs, some or all of the functions of the processing device 120 may be realized by the multiple CPUs operating in cooperation with each other according to a program such as the control program PG1. Furthermore, the processing device 120 may be configured to include hardware such as a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or an FPGA (Field Programmable Gate Array) in addition to one or more CPUs, or instead of some or all of the one or more CPUs. In this case, some or all of the functions of the processing device 120 may be realized by hardware such as a DSP.

[0028] Thus, in this embodiment, the control program PG1 includes application programs for causing the processing device 120 to function as the image information acquisition unit 122, the character recognition unit 123, the memory control unit 124, the character comparison unit 126, the image comparison unit 127, and the output control unit 128. Note that the control program PG1 may include, for example, an operating system program for the processing device 120 to control each unit of the information processing device 10. The control program PG1 is an example of a "program." Also, the image information acquisition unit 122 is an example of an "acquisition unit," and the character recognition unit 123 is an example of a "recognition unit."

[0029] The image information acquisition unit 122 acquires, for example, image information PINF indicating an image of the receipt REC (an image including a plurality of characters written on the receipt REC) from the service server 20 via the communication device 160. The character recognition unit 123 recognizes a plurality of characters included in the image indicated by the image information PINF by performing a character recognition process on the image indicated by the image information PINF acquired by the image information acquisition unit 122. The character recognition unit 123 then generates character information CINF indicating a plurality of characters recognized by the character recognition process on the image indicated by the image information PINF. For example, the character recognition unit 123 may generate the character information CINF by converting the image indicated by the image information PINF into a character code by the character recognition process. The storage control unit 124 stores the image information PINF acquired by the image information acquisition unit 122 and the character information CINF generated by the character recognition unit 123 in the storage device 140 in association with each other.

[0030] The character comparison unit 126 executes character comparison to compare a plurality of characters indicated by the character information CINF generated by the character recognition unit 123 with a plurality of characters indicated by other character information CINF stored in the storage device 140. The other character information CINF is stored in the storage device 140 by the storage control unit 124 in association with the other image information PINF. That is, the other character information CINF and the other image information PINF are stored in the storage device 140 in association with each other.

[0031] For example, as the character comparison, the character comparison unit 126 executes a process of calculating a character similarity between a plurality of characters indicated by the character information CINF to be confirmed and a plurality of characters indicated by another character information CINF. The other character information CINF to be compared with the character information CINF to be confirmed is, for example, character information CINF stored in the storage device 140 prior to the character information CINF to be confirmed. In this embodiment, it is assumed that the character information CINF to be confirmed is the latest character information CINF indicating a plurality of characters recognized by a character recognition process on an image indicated by the latest image information PINF acquired by the image information acquisition unit 122.

[0032] Hereinafter, other character information CINF to be compared with the character information CINF to be confirmed (latest character information CINF) is also referred to as character information CINF to be compared. Also, below, the character similarity between a plurality of characters indicated by one character information CINF (for example, character information CINF to be confirmed) and a plurality of characters indicated by another character information CINF is also referred to as the character similarity between the one character information CINF and the other character information CINF.

[0033] When at least one of the pieces of character information CINF to be compared is specific character information CINF that satisfies a predetermined condition regarding the result of character comparison, the image comparison unit 127 executes image comparison to compare an image indicated by the latest image information PINF with a specific image. The specific image used in image comparison is an image indicated by specific image information PINF associated with the specific character information CINF, among the pieces of image information PINF associated with the character information CINF to be compared. Note that the image information PINF associated with the character information CINF to be compared is an example of "other image information stored in a storage device." Also, the specific character information CINF is an example of "specific character information," and the specific image information PINF is an example of "specific image information."

[0034] In this embodiment, it is assumed that the predetermined condition is that the character similarity between the multiple characters indicated by the latest character information CINF and the multiple characters indicated by the other character information CINF is equal to or greater than a first similarity. In this case, the specific character information CINF is character information CINF that has a character similarity with the latest character information CINF equal to or greater than the first similarity among the pieces of character information CINF to be compared.

[0035] In the present embodiment, it is assumed that the image comparison unit 127 executes a process of determining whether or not the image similarity between an image indicated by the latest image information PINF and an image indicated by a specific image information PINF is equal to or greater than a second similarity as the image comparison. Hereinafter, the image similarity between an image indicated by one image information PINF (e.g., the latest image information PINF) and an image indicated by another image information PINF (e.g., the specific image information PINF) is also referred to as the image similarity between the one image information PINF and the other image information PINF.

[0036] Here, the receipt REC indicated by the latest image information PINF is the receipt REC currently accepted by the service server 20, and the receipt REC indicated by the other image information PINF stored in the storage device 140 is another receipt REC that has already been accepted. Also, the multiple characters indicated by the latest character information CINF are multiple characters recognized by character recognition processing of the image of the receipt REC currently accepted by the service server 20.

[0037] In this way, the receipt comparison process, which compares the receipt REC currently accepted by the service server 20 with other receipts REC that have already been accepted, includes character comparison and image comparison. However, in this embodiment, if none of the character information CINF stored in the storage device 140 meets the predetermined conditions related to the results of the character comparison, image comparison is not performed.

[0038] Furthermore, in this embodiment, as described above, image comparison is not performed on image information PINF associated with text information CINF that does not satisfy the predetermined conditions related to the results of character comparison. Therefore, in this embodiment, the load on the receipt comparison process can be reduced compared to an aspect in which image comparison is performed on all image information PINF stored in storage device 140.

[0039] The output control unit 128 outputs return information RINF indicating the result of the receipt comparison process to the service server 20 via the communication device 160. The return information RINF is, for example, information indicating whether image information PINF indicating an image having an image similarity to the image indicated by the latest image information PINF that is equal to or greater than a second similarity has been detected. Hereinafter, image information PINF indicating an image having an image similarity to the image indicated by the latest image information PINF that is equal to or greater than a second similarity will also be referred to as overlapping candidate image information PINF.

[0040] For example, when at least one of the specific image information PINF is the image information PINF of the duplicate candidate, the image information PINF of the duplicate candidate is detected by image comparison. Therefore, when at least one of the specific image information PINF is the image information PINF of the duplicate candidate, the output control unit 128 outputs the return information RINF indicating that the image information PINF of the duplicate candidate has been detected to the service server 20 via the communication device 160. Note that the detection of the image information PINF of the duplicate candidate corresponds to the image information PINF of the duplicate candidate being stored in the storage device 140. Therefore, when the return information RINF indicates that the image information PINF of the duplicate candidate has been detected, the return information RINF corresponds to "comparison result information". Moreover, the image information PINF of the duplicate candidate is an example of "similar image information".

[0041] In this way, the return information RINF indicates whether or not at least one image of a receipt REC already accepted by the service server 20 is similar to the image of the receipt REC currently accepted. The return information RINF may also include character information CINF that indicates a plurality of characters recognized by a character recognition process on the image of the receipt REC currently accepted by the service server 20.

[0042] The return information RINF may include one or both of character similarity information and image similarity information. The character similarity information indicates, for example, character similarity between a plurality of characters indicated by the latest character information CINF and a plurality of characters indicated by specific character information CINF associated with image information PINF of the overlapping candidate. The image similarity information indicates, for example, image similarity between an image indicated by the latest image information PINF and an image indicated by image information PINF of the overlapping candidate.

[0043] The configuration of the information processing device 10 is not limited to the example shown in Fig. 1. For example, the information processing device 10 may have an operation device that accepts operations by an administrator of the information processing device 10. The storage control unit 124 may store image identification information for identifying the image information PINF in association with the image information PINF in the storage device 140. In this case, the return information RINF may include image identification information indicating the latest image information PINF and image identification information indicating the image information PINF of the duplicate candidate.

[0044] Next, an overview of the receipt comparison process will be described with reference to FIG.

[0045] Fig. 2 is an explanatory diagram for explaining an example of a receipt comparison process according to the first embodiment. In Fig. 2, it is assumed that "m+1" (m is a natural number equal to or greater than 3) pieces of image information PINF have been acquired by the image information acquisition unit 122 from the start of the campaign to the present time. Note that, although Fig. 2 assumes that m is a natural number equal to or greater than 3, m may be any natural number equal to or greater than 1.

[0046] Also, in Fig. 2, in order to distinguish the "m+1" pieces of image information PINF from one another, an alphanumeric character (any of 0, 1, 2, ..., m) is added to the end of the code of each of the "m+1" pieces of image information PINF. Also, in Fig. 2, in order to distinguish the "m+1" pieces of character information CINF from one another, an alphanumeric character is added to the end of the code of each of the "m+1" pieces of character information CINF as the alphanumeric character added to the end of the code of the corresponding image information PINF among the "m+1" pieces of image information PINF.

[0047] 2, it is assumed that the latest image information PINF of the "m+1" pieces of image information PINF acquired by the image information acquisition unit 122 is image information PINF0. That is, the receipt REC indicated by image information PINF0 is the receipt REC currently accepted by the service server 20. The receipt REC indicated by m pieces of image information PINF other than image information PINF0 (PINF1, PINF2, . . . , PINFm) are other receipts REC that have already been accepted. Therefore, in FIG. 2, the character information CINF0 corresponds to the character information CINF to be confirmed, and the m pieces of character information CINF other than character information CINF0 (CINF1, CINF2, . . . , CINFm) correspond to the character information CINF to be compared. Also, in FIG. 2, it is assumed that each of the first similarity TH1 and the second similarity TH2 is 0.8.

[0048] First, the character comparison unit 126 performs a process of calculating character similarity between a plurality of characters indicated by the character information CINF0 and a plurality of characters indicated by each of the other m pieces of character information CINF (CINF1, CINF2, . . . , CINFm) as a character comparison. The method of calculating the character similarity is not particularly limited. For example, a known similarity calculation method for comparing two sentences may be adopted as the method of calculating the character similarity. Examples of the known similarity calculation method for comparing two sentences include a similarity calculation method using N-gram, a similarity calculation based on the Levenshtein distance, and a similarity calculation method such as a cosine similarity calculation. Note that N-gram is a division method for dividing an arbitrary character string into N consecutive notation units (characters or words, etc.) (N is a natural number equal to or greater than 1). In N-gram, the character string to be divided is divided sequentially by moving one notation unit at a time after the first division.

[0049] 2, it is assumed that the character comparing unit 126 calculates the character similarity between the character information CINF0 and each of the "m-2" pieces of comparison target character information CINF other than the character information CINF1 and CINF2 to be less than the first similarity TH1 (0.8). In the example shown in FIG. 2, the character similarity between the character information CINF0 and the character information CINF1 is calculated to be 0.95, the character similarity between the character information CINF0 and the character information CINF2 is calculated to be 0.8, and the character similarity between the character information CINF0 and the character information CINFm is calculated to be 0.75.

[0050] For example, character comparison unit 126 identifies character information CINF1 and CINF2, which have character similarity with character information CINF0 equal to or greater than first similarity TH1, as specific character information CINF among m pieces of character information CINF to be compared (CINF1, CINF2, ..., CINFm). A white star in the figure indicates specific character information CINF. Note that the identification of specific character information CINF may be performed by image comparison unit 127 based on the character similarity calculated by character comparison unit 126.

[0051] Since at least one of the m pieces of character information CINF to be compared is specific character information CINF (character information CINF1 and CINF2), the image comparison unit 127 executes image comparison to compare the image indicated by the latest image information PINF0 with the specific image. The specific image is an image indicated by each of the specific image information PINF1 and PINF2 associated with the specific character information CINF1 and CINF2, respectively.

[0052] For example, the image comparison unit 127 executes a process of determining whether the image similarity between each of the two images represented by specific image information PINF1 and PINF2 and the image represented by the latest image information PINF0 is equal to or greater than the second similarity TH2 (0.8). In Fig. 2, it is assumed that the image similarity between the image information PINF0 and the image information PINF1 is calculated as 0.98, and the image similarity between the image information PINF0 and the image information PINF2 is calculated as 0.7, by the image comparison.

[0053] 2, the image comparison unit 127 determines that the image similarity between the image information PINF0 and the image information PINF1 is equal to or greater than the second similarity TH2, and determines that the image similarity between the image information PINF0 and the image information PINF2 is not equal to or greater than the second similarity TH2. Then, the image comparison unit 127 specifies the image information PINF1, of the specific image information PINF1 and PINF2, whose image similarity with the image information PINF0 is equal to or greater than the second similarity TH2, as the image information PINF of the overlapping candidate. The black star in the figure indicates the image information PINF of the overlapping candidate. The specification of the image information PINF of the overlapping candidate may be performed by the output control unit 128 based on the image similarity calculated by the image comparison unit 127.

[0054] In this manner, in this embodiment, image comparison is performed after character comparison, and thus overlapping candidate image information PINF having image similarity with the latest image information PINF0 equal to or greater than the second similarity TH2 is detected. Note that in this embodiment, image comparison is performed only for specific image information PINF (PINF1 and PINF2) among m pieces of image information PINF (PINF1, PINF2, . . . , PINFm) other than the latest image information PINF0. For this reason, in this embodiment, the load imposed by image comparison can be reduced compared to an aspect in which image comparison is performed for m pieces of image information PINF other than the latest image information PINF0.

[0055] Next, an overview of a receipt comparison process according to a comparative example that is contrasted with this embodiment will be described with reference to FIG.

[0056] 3 is an explanatory diagram for explaining an example of a receipt comparison process according to comparative examples. The first comparative example is a receipt comparison process in which only the character comparison is performed out of the character comparison and image comparison. The second comparative example is a receipt comparison process in which only the image comparison is performed out of the character comparison and image comparison.

[0057] In the first comparison example, as described above, image comparison is not performed. Therefore, in the first comparison example, in order to prevent erroneous detection of the character information CINF of a receipt REC similar to the currently accepted receipt REC, it is possible to set the first similarity THz1 to a value higher than the first similarity TH1 shown in Figure 2. In the first comparison example shown in Figure 2, the first similarity THz1 is 0.9. Hereinafter, the character information CINF of a receipt REC similar to the currently accepted receipt REC is also referred to as character information CINF of a duplicate candidate.

[0058] For example, in the first comparative example, character information CINF1, whose character similarity (0.95) with the latest character information CINF0 is equal to or greater than the first similarity THz1 (0.9), is detected as character information CINF of a duplicate candidate. Note that character information CINF2, whose character similarity (0.8) with the latest character information CINF0 is less than the first similarity THz1, is not detected as character information CINF of a duplicate candidate. In this way, in the first comparative example, the first similarity THz1 is set to a value higher than the first similarity TH1, thereby preventing character information CINF2 from being erroneously detected as character information CINF of a duplicate candidate.

[0059] However, in the first comparative example, if a misrecognized character is included among the multiple characters recognized by the character recognition process on the image of the receipt REC, the comparison accuracy of the receipt REC decreases. For example, if a misrecognized character is included among the multiple characters indicated by the character information CINF0, a character similarity less than the first similarity THz1 may be calculated as the character similarity between the character information CINF0 and the character information CINF1. In this case, the character information CINF1 that should have been detected as the character information CINF of the duplicate candidate is not detected as the character information CINF of the duplicate candidate. Alternatively, if a misrecognized character is included among the multiple characters indicated by the character information CINF0, a character similarity equal to or greater than the first similarity THz1 may be calculated as the character similarity between the character information CINF0 and the character information CINF2. In this case, the character information CINF2 is erroneously detected as the character information CINF of the duplicate candidate.

[0060] In contrast, in this embodiment, after character comparison, image comparison is performed on specific image information PINF associated with specific character information CINF identified by character comparison, as described with reference to Fig. 2 etc. Therefore, in this embodiment, even if the first similarity TH1 is set to a value lower than the first similarity THz1, it is possible to prevent erroneous detection of a receipt REC similar to the currently received receipt REC.

[0061] For example, in this embodiment, even if a misrecognized character is included among the multiple characters indicated by the character information CINF, it is possible to prevent the character information CINF that should be identified as specific character information CINF from not being identified. As an example, assume that a character similarity between the character information CINF0 and the character information CINF1 is calculated to be 0.8 because a misrecognized character is included among the multiple characters indicated by the character information CINF0. Even in this case, in this embodiment, the character similarity between the character information CINF0 and the character information CINF1 is equal to or greater than the first similarity TH1 (0.8), so the character information CINF1 is identified as specific character information CINF. Note that in the first comparative example, in the case of the above example, the character similarity between the character information CINF0 and the character information CINF1 is less than the first similarity THz1 (0.9), so the character information CINF1 is not detected as the character information CINF of the overlap candidate.

[0062] As another example, assume that the character similarity between the character information CINF0 and the character information CINF2 is calculated to be 0.96 because the multiple characters indicated by the character information CINF0 include an erroneously recognized character. In this case as well, in this embodiment, image comparison is performed for the specific image information PINF1 and PINF2 associated with the specific character information CINF1 and CINF2, respectively. Therefore, in this embodiment, as described in FIG. 2, the image information PINF1 is detected as the image information PINF of the overlapping candidate. Note that in the first comparative example, in the above example, the character similarity between the character information CINF0 and the character information CINF2 is equal to or greater than the first similarity THz1 (0.9), so the character information CINF2 is erroneously detected as the character information CINF of the overlapping candidate.

[0063] Next, the second comparative example will be described.

[0064] In the second comparative example shown in FIG. 2, the second similarity THz2 is 0.8, which is the same value as the second similarity TH2 shown in FIG. 2. For example, in the second comparative example, the image information PINF1, whose image similarity (0.98) with the latest image information PINF0 is equal to or greater than the second similarity THz2 (0.8), is detected as the image information PINF of the overlapping candidate. In this way, in the second comparative example, the image information PINF1 is detected as the image information PINF of the overlapping candidate, similarly to the example shown in FIG. 2. However, in the second comparative example, since no character comparison is performed, all image information PINF other than the latest image information PINF0 among the multiple image information PINF is compared with the latest image information PINF0. Therefore, in the second comparative example, image processing for image comparison is performed on all of the multiple images respectively indicated by the multiple image information PINF. Therefore, in the second comparative example, the load and time required for image comparison are increased compared to the present embodiment. In general, image comparison has a larger processing load and the like than character comparison. For this reason, the load and time required for the receipt comparison process increases in the second comparative example compared to this embodiment.

[0065] Furthermore, in image comparison, even when two images obtained by photographing the same receipt REC are compared, the image similarity of the two images may be calculated to be less than the second similarity THz2 due to differences in photographing conditions, such as photographing outdoors and photographing indoors. That is, in image comparison, two images of the same receipt REC may be erroneously determined to be images of different receipts REC due to differences in photographing conditions. For this reason, in order to prevent erroneous determination due to differences in photographing conditions, it is possible to perform image correction processing, such as a process to correct the exposure and a process to remove the background from the image, on the image indicated by the image information PINF. When image correction processing is performed in image comparison, the load and time required for image comparison increases compared to when image correction processing is not performed.

[0066] In this embodiment, only specific image information PINF0 among the multiple pieces of image information PINF is compared with the latest image information PINF0 as described in Fig. 2. Therefore, in this embodiment, even when an image correction process is executed in the image comparison, an increase in the load and time required for the image comparison can be suppressed compared to the second comparison example.

[0067] In this way, in this embodiment, it is possible to prevent a decrease in the comparison accuracy of the receipt REC while also preventing an increase in the load and time required for the receipt comparison process.

[0068] Next, an overview of the operation of the information processing device 10 will be described with reference to FIG.

[0069] Fig. 4 is a flowchart showing an example of the operation of the information processing device 10 shown in Fig. 1. The operation shown in Fig. 4 corresponds to a receipt comparison process. For example, the receipt comparison process shown in Fig. 4 is executed every time the service server 20 transmits image information PINF to the information processing device 10.

[0070] First, in step S100, the processing device 120 functions as the image information acquisition unit 122 and acquires image information PINF indicating an image of the receipt REC from the service server 20. Hereinafter, the image information PINF acquired in step S100 is also referred to as the latest image information PINF.

[0071] Next, in step S200, the processing device 120 functions as the character recognition unit 123 and recognizes a plurality of characters included in the image indicated by the latest image information PINF. For example, the character recognition unit 123 recognizes a plurality of characters included in the image indicated by the latest image information PINF by performing a character recognition process on the image indicated by the latest image information PINF, and generates character information CINF indicating the recognized plurality of characters. Hereinafter, the character information CINF indicating the plurality of characters recognized in step S200 is also referred to as the latest character information CINF.

[0072] Next, in step S220, the processing device 120 functions as the storage control unit 124 and stores in the storage device 140 the character information CINF indicating the multiple characters recognized in step S200 and the image information PINF acquired in step S100 in association with each other.

[0073] Then, in step S300, the processing device 120 functions as the character comparison unit 126 and determines whether or not the character information CINF to be compared is stored in the storage device 140. The character information CINF to be compared is, for example, other character information CINF to be compared with the latest character information CINF.

[0074] If the result of the determination in step S300 is positive, the processing device 120 proceeds to step S400. On the other hand, if the result of the determination in step S300 is negative, the processing device 120 in step S620 outputs return information RINF indicating that no duplicate candidate image information PINF was detected to the service server 20, and ends the receipt comparison process. Note that in step S620, the processing device 120 functions as the output control unit 128.

[0075] In step S400, the processing device 120 functions as the character comparison unit 126 and compares the latest character information CINF with the comparison target character information CINF. For example, the character comparison unit 126 calculates the character similarity between the multiple characters indicated by the latest character information CINF and the multiple characters indicated by the comparison target character information CINF.

[0076] Next, in step S420, the processing device 120 functions as the character comparison unit 126 and determines whether the character similarity between the latest character information CINF and the comparison target character information CINF is equal to or greater than the first similarity TH1. Note that, when there are multiple pieces of comparison target character information CINF, the determination in step S420 is performed for each of the multiple pieces of comparison target character information CINF.

[0077] If the determination result in step S420 is positive, the processing device 120 proceeds to step S500. On the other hand, if the determination result in step S420 is negative, the processing device 120 executes the process of step S620 and ends the receipt comparison process. If there are multiple pieces of character information CINF to be compared, the determination result in step S420 will be as follows. For example, if at least one of the multiple pieces of character information CINF to be compared has a character similarity with the latest character information CINF that is equal to or greater than the first similarity TH1, the determination result in step S420 will be positive. In other words, if the character similarity with the latest character information CINF of all of the multiple pieces of character information CINF to be compared is less than the first similarity TH1, the determination result in step S420 will be negative.

[0078] In step S500, the processing device 120 functions as the image comparison unit 127 and compares the latest image information PINF with the specific image information PINF. Note that the specific image information PINF is image information PINF associated with the specific character information CINF whose character similarity with the latest character information CINF is equal to or greater than the first similarity TH1, as described with reference to Fig. 2 etc. For example, the image comparison unit 127 calculates the image similarity between the image indicated by the latest image information PINF and the image indicated by the specific image information PINF.

[0079] Then, in step S520, the processing device 120 functions as the image comparison unit 127 and determines whether the image similarity between the latest image information PINF and the specific image information PINF is equal to or greater than the second similarity TH2. Note that, when there are multiple pieces of specific image information PINF, the determination in step S520 is performed for each of the multiple pieces of specific image information PINF.

[0080] If the result of the determination in step S520 is positive, the processing device 120 proceeds to step S600. On the other hand, if the result of the determination in step S520 is negative, the processing device 120 executes the process of step S620 and ends the receipt comparison process. If there are multiple pieces of specific image information PINF, the result of the determination in step S520 will be as follows. For example, if at least one of the multiple pieces of specific image information PINF has an image similarity with the latest image information PINF that is equal to or greater than the second similarity TH2, the result of the determination in step S520 will be positive. In other words, if the image similarity with the latest image information PINF of all of the multiple pieces of specific image information PINF is less than the second similarity TH2, the result of the determination in step S520 will be negative.

[0081] In step S600, the processing device 120 functions as the output control unit 128 and outputs return information RINF indicating that duplicate candidate image information PINF was detected to the service server 20. The processing device 120 then ends the receipt comparison process.

[0082] The operation of the information processing device 10 is not limited to the example shown in Fig. 4. For example, the process of step S300 may be executed by a functional block (e.g., the memory control unit 124, etc.) other than the character comparison unit 126. Also, for example, the process of step S420 may be executed by a functional block (e.g., the image comparison unit 127, etc.) other than the character comparison unit 126. Also, for example, the process of step S520 may be executed by a functional block (e.g., the output control unit 128, etc.) other than the image comparison unit 127.

[0083] As described above, in this embodiment, the information processing device 10 has an image information acquisition unit 122, a character recognition unit 123, a storage control unit 124, a character comparison unit 126, and an image comparison unit 127. The image information acquisition unit 122 acquires image information PINF indicating an image including a plurality of characters written on a receipt REC. The character recognition unit 123 recognizes a plurality of characters included in the image indicated by the image information PINF, and generates character information CINF indicating the recognized plurality of characters. The storage control unit 124 stores the image information PINF and the character information CINF in the storage device 140 in association with each other. The character comparison unit 126 performs a character comparison to compare a plurality of characters indicated by the character information CINF with a plurality of characters indicated by other character information CINF stored in the storage device 140. When at least one of the above-mentioned other character information CINF is specific character information CINF that satisfies a predetermined condition regarding the result of the character comparison, the image comparison unit 127 performs an image comparison to compare the image indicated by the image information PINF with an image indicated by specific image information PINF that is associated with the specific character information CINF among the other image information PINF stored in the storage device 140.

[0084] In this manner, in this embodiment, image comparison is performed in addition to character comparison as a receipt comparison process for comparing multiple receipts REC, so that the accuracy of comparison of multiple receipts can be improved. That is, in this embodiment, the accuracy of comparison of multiple media on which multiple characters are written can be improved. Note that in this embodiment, when specific character information CINF is detected by character comparison, image comparison is performed on specific image information PINF associated with the specific character information CINF. The specific character information CINF is, for example, character information CINF indicating multiple characters that are highly similar to multiple characters included in the image indicated by the image information PINF acquired by the image information acquisition unit 122. In this case, in this embodiment, the object to be compared with the image information PINF acquired by the image information acquisition unit 122 is limited to the specific image information PINF that is highly similar to the image information PINF. As a result, in this embodiment, a comparison between the image of the receipt REC indicated by the image information PINF acquired by the image information acquisition unit 122 and the image of the receipt REC indicated by the image information PINF already stored in the storage device 140 can be performed quickly.

[0085] In the present embodiment, the predetermined condition is that the character similarity between the multiple characters indicated by the character information CINF and the multiple characters indicated by the other character information CINF is equal to or greater than the first similarity TH1. Then, the image comparison unit 127 executes a process of determining whether the image similarity between the image indicated by the image information PINF and the image indicated by the specific image information PINF is equal to or greater than the second similarity TH2 as the image comparison.

[0086] In this manner, in this embodiment, the target to be compared with the image information PINF acquired by the image information acquisition unit 122 is limited to specific image information PINF associated with specific character information CINF that has a high degree of character similarity with the character information CINF associated with the image information PINF. This makes it possible in this embodiment to quickly determine whether or not duplicate candidate image information PINF indicating an image of a receipt REC similar to the image of the receipt REC indicated by the image information PINF acquired by the image information acquisition unit 122 is stored in the storage device 140.

[0087] In the present embodiment, the information processing device 10 further includes an output control unit 128 that, when at least one of the specific image information PINF is image information PINF of a duplicate candidate, outputs return information RINF indicating that the image information PINF of the duplicate candidate is stored in the storage device 140. The image information PINF of the duplicate candidate is image information PINF indicating an image whose image similarity with the image indicated by the image information PINF is equal to or higher than the second similarity TH2.

[0088] Thus, in this embodiment, when the image information PINF of the duplicate candidate is stored in the storage device 140, the output control unit 128 outputs return information RINF indicating that the image information PINF of the duplicate candidate is stored in the storage device 140. For example, the output control unit 128 outputs return information RINF indicating that the image information PINF of the duplicate candidate is stored in the storage device 140 to a device (e.g., the service server 20) that transmitted the image information PINF to the information processing device 10. This makes it possible, in this embodiment, to cause the device that transmitted the image information PINF to recognize that the image information PINF of the duplicate candidate indicating an image of a receipt REC similar to the image of the receipt REC indicated by the image information PINF is stored in the storage device 140.

[0089] Furthermore, in this embodiment, the return information RINF may include character similarity information and image similarity information. The character similarity information indicates the character similarity between multiple characters indicated by the character information CINF and multiple characters indicated by specific character information CINF associated with the image information PINF of the duplicate candidate. The image similarity information indicates the image similarity between the image indicated by the image information PINF and the image indicated by the image information PINF of the duplicate candidate. When the return information RINF includes character similarity information and image similarity information, a device that receives the return information RINF can easily recognize the degree of similarity between the receipt REC indicated by the image information PINF and the receipt REC indicated by the image information PINF of the duplicate candidate.

[0090] In addition, in this embodiment, the storage control unit 124 may store image identification information for identifying the image information PINF in association with the image information PINF in the storage device 140. In this case, the return information RINF may include image identification information indicating the image information PINF and image identification information indicating the image information PINF of the duplicate candidate. When the return information RINF includes image identification information, a device that receives the return information RINF can easily identify the image information PINF of the duplicate candidate.

[0091] [2. Second embodiment] Next, an example of an overview of an information processing system 1 according to the second embodiment will be described with reference to FIG.

[0092] Fig. 5 is an explanatory diagram for explaining an overview of an information processing system 1 according to a second embodiment. Elements similar to those explained in Fig. 1 to Fig. 4 are given the same reference numerals and detailed explanations are omitted.

[0093] The information processing system 1 shown in FIG. 5 has the same configuration as the information processing system 1 shown in FIG. 1. For example, the information processing system 1 has an information processing device 10 including a processing device 120, a storage device 140, and a communication device 160, a service server 20, and a terminal device 30. However, the control program PG2 stored in the storage device 140 of the information processing device 10 is different from the control program PG1 shown in FIG. 1. The control program PG2 is an example of a "program". In this embodiment, the control program PG2 includes application programs for causing the processing device 120 to function as the image information acquisition unit 122, the character recognition unit 123, the storage control unit 124, the structuring unit 125, the character comparison unit 126, the image comparison unit 127, and the output control unit 128. The control program PG2 may include, for example, an operating system program for the processing device 120 to control each unit of the information processing device 10. The control program PG2 may be transmitted from another device via the network NW.

[0094] The processing device 120 shown in Fig. 5 is similar to the processing device 120 shown in Fig. 1, except that the processing device 120 executes a control program PG2 instead of the control program PG1 shown in Fig. 1. For example, the processing device 120 executes the control program PG2 stored in the storage device 140 to function as an image information acquisition unit 122, a character recognition unit 123, a storage control unit 124, a structuring unit 125, a character comparison unit 126, an image comparison unit 127, and an output control unit 128. That is, this embodiment differs from the first embodiment described above in that the processing device 120 functions as the structuring unit 125.

[0095] Image information acquisition unit 122, character recognition unit 123, memory control unit 124, character comparison unit 126, image comparison unit 127, and output control unit 128 are the same as those in the above-mentioned first embodiment. The following description will focus on structuring unit 125, which is different from the above-mentioned first embodiment.

[0096] The structuring unit 125 structures the characters represented by the character information CINF, for example, by dividing the characters represented by the character information CINF into words and associating each divided word with one of a number of items. The number of items is, for example, the store name and other items written on the receipt REC.

[0097] For example, the structuring unit 125 divides the characters indicated by the character information CINF into multiple character strings based on the positions of the characters on the receipt REC, thereby dividing the multiple characters into words. The structuring unit 125 then associates the divided words with one of the multiple items based on the correspondence between the positions of the characters and the items. The correspondence between the positions of the characters and the items has been learned, for example, by machine learning.

[0098] The position of a character written on the receipt REC may be, for example, a line number indicating which line the character belongs to on the receipt REC. Furthermore, the position of a character written on the receipt REC may be the line number of the line to which the character belongs and the position of the character on the line to which the character belongs. The position of a character written on the receipt REC is not limited to the above example. For example, the position of a character written on the receipt REC may be a position relative to a predetermined position on the receipt REC.

[0099] In this manner, in this embodiment, the multiple characters indicated by the character information CINF are structured by the structuring unit 125. That is, the multiple characters indicated by each of the latest character information CINF and the comparison target character information CINF compared by the character comparing unit 126 are structured by the structuring unit 125. For example, the character comparing unit 126 executes a comparison between the multiple characters indicated by the latest character information CINF and the multiple characters indicated by the comparison target character information CINF for each structured item. As a result, in this embodiment, for example, it is possible to accurately and quickly calculate the character similarity between the latest character information CINF and the comparison target character information CINF.

[0100] Next, an example of the return information RINF will be described with reference to FIG.

[0101] Fig. 6 is an explanatory diagram for explaining an example of the return information RINF. Fig. 6 shows an example of the return information RINF when a duplicate candidate image information PINF showing an image similar to the image shown by the latest image information PINF is detected.

[0102] The return information RINF includes character information CINF indicating a plurality of characters recognized by a character recognition process for the image indicated by the latest image information PINF, and similarity information SINF regarding the image information PINF of the overlapping candidate. Note that, in this embodiment, the plurality of characters indicated by the character information CINF are structured as described in FIG. 5.

[0103] The text information CINF includes information indicating a plurality of items and information indicating character strings corresponding to the plurality of items. In the "Result" of Fig. 6, the portion corresponding to the text information CINF shows examples of character strings associated with each item by structuring. In Fig. 6, examples of the plurality of items include the store name, purchase date and time, phone number, product, unit price, number of products, total product price, JAN code, total price, and payment method, but the plurality of items are not limited to the examples shown in Fig. 6. The JAN code is a product identification code for identifying which business entity owns what product.

[0104] The similarity information SINF also includes identification information IINF0, duplicate candidate identification information IINF1, character similarity information CSINF, and image similarity information PSINF. The portion of the "Results" in Figure 6 that corresponds to the similarity information SINF shows examples of targets and results of the receipt comparison process. Note that the similarity information SINF is an example of "comparison result information."

[0105] Identification information IINF0 is image identification information that identifies image information PINF (latest image information PINF) associated with text information CINF included in return information RINF. Furthermore, duplicate candidate identification information IINF1 is image identification information that identifies duplicate candidate image information PINF detected by the receipt comparison process, and is an example of "other image identification information." Below, identification information IINF0 and IINF1 may be collectively referred to as identification information IINF.

[0106] The character similarity information CSINF indicates the character similarity between a plurality of characters indicated by character information CINF included in the return information RINF and a plurality of characters indicated by specific character information CINF associated with image information PINF of the overlapping candidate. The image similarity information PSINF indicates the image similarity between an image indicated by image information PINF associated with character information CINF included in the return information RINF and an image indicated by image information PINF of the overlapping candidate.

[0107] Here, when the image information PINF of the overlapping candidate is not detected, the similarity information SINF does not include the identification information IINF1 of the overlapping candidate, the character similarity information CSINF, and the image similarity information PSINF. That is, the return information RINF that does not include the identification information IINF1 of the overlapping candidate, the character similarity information CSINF, and the image similarity information PSINF is the return information RINF indicating that the image information PINF of the overlapping candidate was not detected. On the other hand, the return information RINF that includes at least one of the identification information IINF1 of the overlapping candidate, the character similarity information CSINF, and the image similarity information PSINF is the return information RINF indicating that the image information PINF of the overlapping candidate was detected.

[0108] The configuration of the return information RINF is not limited to the example shown in FIG. 6. For example, when a plurality of overlapping candidate image information PINFs are detected, the return information RINF may include overlapping candidate identification information IINF1, character similarity information CSINF, and image similarity information PSINF corresponding to each of the plurality of overlapping candidate image information PINFs. Alternatively, the output control unit 128 may select a predetermined number of image information PINFs from among the plurality of overlapping candidate image information PINFs in descending order of image similarity or character similarity as image information PINFs to be reflected in the return information RINF. In this case, the return information RINF includes overlapping candidate identification information IINF1, character similarity information CSINF, and image similarity information PSINF corresponding to each of the predetermined number of image information PINFs selected by the output control unit 128 from among the plurality of overlapping candidate image information PINFs.

[0109] Also, for example, one of the character similarity information CSINF and the image similarity information PSINF may be omitted from the similarity information SINF. Also, the identification information IINF0 and IINF1 may be omitted from the similarity information SINF. In this case, the identification information IINF does not need to be stored in the storage device 140.

[0110] Next, an overview of the operation of the information processing device 10 according to the present embodiment will be described with reference to FIG.

[0111] Fig. 7 is a flow chart showing an example of the operation of the information processing device 10 shown in Fig. 5. The operation shown in Fig. 7 corresponds to a receipt comparison process. For example, the receipt comparison process shown in Fig. 7 is executed each time the service server 20 transmits image information PINF to the information processing device 10, similar to the receipt comparison process shown in Fig. 4. The operation shown in Fig. 7 is similar to the operation shown in Fig. 4, except that the process of step S210 is executed after the process of step S200 is executed. A detailed description of the same processes as those described in Fig. 4 will be omitted.

[0112] The series of processes in steps S100 and S200 is similar to the series of processes in steps S100 and S200 shown in Fig. 4. After executing the process in step S200, the processing device 120 advances the process to step S210.

[0113] In step S210, the processing device 120 functions as the structuring unit 125 and structures the multiple characters recognized in step S200. As a result, character information CINF indicating the multiple structured characters is generated. After executing the process of step S210, the processing device 120 advances the process to step S220.

[0114] In step S220, the processing device 120 functions as the storage control unit 124, and stores the character information CINF indicating the multiple characters structured in step S210 and the image information PINF acquired in step S100 in association with each other in the storage device 140. Also, although not shown in Fig. 7, in this embodiment, in step S220, the storage control unit 124 stores identification information IINF for identifying the image information PINF in association with the image information PINF in the storage device 140.

[0115] Then, after executing the process of step S220, the processing device 120 advances the process to step S300. A series of processes after step S300 is similar to the series of processes after step S300 shown in Fig. 4. Note that the operation of the information processing device 10 is not limited to the example shown in Fig. 7.

[0116] As described above, the present embodiment can also provide the same effects as the first embodiment. In the present embodiment, the information processing device 10 further includes a structuring unit 125 that divides the characters indicated by the character information CINF into words and associates the divided words with any of a plurality of items, thereby structuring the characters indicated by the character information CINF. For example, the structuring unit 125 divides the characters indicated by the character information CINF into a plurality of character strings based on the position of the characters on the receipt REC, thereby dividing the characters into words. Therefore, in the present embodiment, the characters indicated by the character information CINF and the other character information CINF compared by the character comparison unit 126 are structured by the structuring unit.

[0117] For example, in this embodiment, the character comparison unit 126 performs a comparison between a plurality of characters indicated by the latest character information CINF and a plurality of characters indicated by the comparison target character information CINF for each structured item. As a result, in this embodiment, it is possible to accurately and quickly calculate the character similarity between the latest character information CINF and the comparison target character information CINF.

[0118] [3. Modifications] The present invention is not limited to the above-described embodiments. Specific modified embodiments are exemplified below. Two or more embodiments selected from the following examples may be combined.

[0119] [First Modification] In the second embodiment described above, character strings to be compared by character comparison may be limited to character strings associated with a predetermined item among the multiple items. The predetermined item is, for example, an item required to determine whether multiple receipts REC are the same receipt REC. In this case, since the number of items to be compared is limited, the comparison between the latest character information CINF and the comparison target character information CINF can be performed quickly.

[0120] Alternatively, in the second embodiment described above, a weight value may be associated with each of the multiple items. For example, a larger weight value may be associated with an item that has a greater influence on determining whether multiple receipts REC are the same receipt REC than an item that has a smaller influence. The character comparison unit 126 may then calculate the character similarity using the weight value associated with each item. In this case, it is possible to accurately detect specific character information CINF associated with specific image information PINF for which image comparison is to be performed.

[0121] As described above, in this modified example as well, the same effects as those of the second embodiment described above can be obtained.

[0122] [Second modified example] In the above-mentioned embodiment and modified example, the information processing system 1 is exemplified as a campaign application system, but the present invention is not limited to such an embodiment. For example, the information processing system 1 may be a system that creates a household account book using an image of a receipt REC. Furthermore, the information processing system 1 is not limited to a system that uses a receipt REC, and may be, for example, a system that converts a document into electronic data and registers it. That is, the medium on which a plurality of characters are written is not limited to a receipt REC. For example, the medium on which a plurality of characters are written includes accounting documents such as invoices, financial statements, transfer forms, and estimates, recording sheets that record measured values ​​in experiments and observations, business cards, and documents in a wide range of fields such as application stickers. As described above, the modified example can also obtain the same effects as the above-mentioned embodiment and modified example. [Explanation of symbols]

[0123] 1...information processing system, 10...information processing device, 20...service server, 30...terminal device, 120...processing device, 122...image information acquisition unit, 123...character recognition unit, 124...memory control unit, 125...structuring unit, 126...character comparison unit, 127...image comparison unit, 128...output control unit, 140...memory device, 160...communication device, AINF...response information, CINF...character information, PINF...image information, RINF...return information.

Claims

1. an acquisition unit that acquires image information indicating an image including a plurality of characters written on a medium; a recognition unit that recognizes a plurality of characters included in the image represented by the image information and generates character information representing the recognized plurality of characters; a storage control unit that stores the image information and the character information in a storage device in association with each other; a character comparison unit that performs character comparison to compare a plurality of characters indicated by the character information with a plurality of characters indicated by other character information stored in the storage device; an image comparison unit that performs image comparison to compare an image represented by the image information with an image represented by specific image information associated with the specific character information among other image information stored in the storage device when at least one of the other character information is specific character information that satisfies a predetermined condition related to the result of the character comparison; An information processing device comprising:

2. The predetermined condition is: a character similarity between the plurality of characters indicated by the character information and the plurality of characters indicated by the other character information is equal to or greater than a first similarity; The image comparison unit As the image comparison, a process of determining whether or not an image similarity between the image indicated by the image information and the image indicated by the specific image information is equal to or greater than a second similarity is executed. The information processing device according to claim 1 .

3. an output control unit that outputs comparison result information indicating that the similar image information is stored in the storage device when at least one of the specific image information is similar image information indicating an image whose image similarity with the image indicated by the image information is equal to or greater than the second similarity; The information processing device according to claim 2 .

4. The comparison result information is character similarity information indicating the character similarity between a plurality of characters indicated by the character information and a plurality of characters indicated by specific character information associated with the similar image information, and image similarity information indicating the image similarity between an image indicated by the image information and an image indicated by the similar image information, The information processing device according to claim 3 .

5. The storage control unit image identification information for identifying the image information is stored in the storage device in association with the image information; The comparison result information is The image identification information indicating the image information and other image identification information indicating the similar image information are further included. The information processing device according to claim 4 .

6. acquiring image information indicating an image including a plurality of characters written on a medium; Recognizing a plurality of characters included in the image represented by the image information, and generating character information representing the recognized plurality of characters; storing the image information and the character information in a storage device in association with each other; performing a character comparison to compare a plurality of characters indicated by the character information with a plurality of characters indicated by other character information stored in the storage device; when at least one of the other character information is specific character information that satisfies a predetermined condition regarding the result of the character comparison, performing image comparison to compare an image represented by the image information with an image represented by specific image information associated with the specific character information, among other image information stored in the storage device; Information processing methods.

7. The processor, an acquisition unit that acquires image information indicating an image including a plurality of characters written on a medium; a recognition unit that recognizes a plurality of characters included in the image represented by the image information and generates character information representing the recognized plurality of characters; a storage control unit that stores the image information and the character information in a storage device in association with each other; a character comparison unit that performs character comparison to compare a plurality of characters indicated by the character information with a plurality of characters indicated by other character information stored in the storage device; an image comparison unit that performs image comparison to compare an image represented by the image information with an image represented by specific image information associated with the specific character information among other image information stored in the storage device when at least one of the other character information is specific character information that satisfies a predetermined condition related to the result of the character comparison; To function as, program.

8. A program that, when executed by a processor, acquires image information showing an image including a plurality of characters written on a medium, converts the acquired image information into electronic data, and registers the electronic data, Recognizing a plurality of characters included in the image represented by the image information, and generating character information representing the recognized plurality of characters; storing the image information and the character information in a storage device in association with each other; performing a character comparison to compare a plurality of characters indicated by the character information with a plurality of characters indicated by other character information stored in the storage device; when at least one of the other character information is specific character information that satisfies a predetermined condition related to the result of the character comparison, performing an image comparison in which an image represented by the image information is compared with an image represented by specific image information associated with the specific character information, among other image information stored in the storage device; If a predetermined condition regarding the result of the image comparison is satisfied, the image information is not converted into electronic data and registered. A program characterized by:

Images