Information processing system, server device, information processing method, and information processing program

Abstract

This invention provides an information processing system, user terminal, server device, method, and program for aggregating transaction information without relying on API integration. [Solution] In the information processing system, the user terminal includes an image acquisition unit 2401 that acquires image data of at least one target image on which multiple transaction information of the user is displayed, an OCR processing unit 2404 that performs OCR processing on the image data to convert the image data into text data, and an output control unit that controls the output of multiple structured transaction information to the user. The server device includes a prompt generation unit 3202 that generates a prompt including text data and a command statement that instructs the extraction of multiple transaction information from the text data in a predetermined format, and a transaction information acquisition unit 3204 that inputs the prompt to a large-scale language model to acquire multiple structured transaction information in which multiple transaction information has been structured.

Description

Technical Field

[0001] This disclosure relates to information processing technology.

Background Art

[0002] Patent Document 1 discloses a technique in which an information processing apparatus acquires account data or history data from a server with API linkage.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When managing transaction items such as expenses using an application such as a household ledger, it is conceivable to aggregate expense information from the servers of multiple service providers by utilizing the information acquisition technology based on API linkage in Patent Document 1. However, when relying on API linkage, there is a problem that information cannot be obtained from service providers that do not support API linkage.

[0005] This disclosure has been made to solve such problems, and an object thereof is to provide an information processing technology capable of aggregating transaction information without relying on API linkage.

Means for Solving the Problems

[0006] The information processing system according to the embodiment of the present disclosure includes: an image acquisition unit that acquires image data of at least one target image on which multiple transaction information of a user is displayed; an OCR processing unit that performs OCR processing on the image data to convert the image data into text data; a prompt generation unit that generates a prompt including the text data and a command statement that instructs the extraction of the multiple transaction information from the text data in a predetermined format; a transaction information acquisition unit that inputs the prompt to a large-scale language model to acquire a plurality of structured transaction information in which the multiple transaction information has been structured; and an output control unit that controls the output of the plurality of structured transaction information to the user. [Effects of the Invention]

[0007] According to the information processing technology of the embodiment of this disclosure, transaction information can be aggregated without relying on API integration. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram showing an example of the configuration of an information processing system. [Figure 2] This figure shows an example of a user terminal configuration. [Figure 3] This figure shows an example of a server device configuration. [Figure 4A] This figure shows an example of the hardware configuration of a control device. [Figure 4B] This figure shows an example of the hardware configuration of a control device. [Figure 5] This is a flowchart showing the operation of an information processing system. [Figure 6] This is an example of a screenshot image of the transaction history screen in a payment app. [Figure 7] This is an example of a screenshot image of the transaction history screen in a payment app. [Modes for carrying out the invention]

[0009] Various embodiments of this disclosure will be described in detail below with reference to the drawings. In the drawings, identical or similar parts are denoted by the same or similar reference numerals, and redundant descriptions of such parts will be omitted. In this disclosure, the term "or" is used in the sense of an inclusive OR unless otherwise stated.

[0010] "Aggregating transaction information without relying on API integration" means that the process of acquiring transaction information without using API integration is included in the process of gathering the transaction information into a single device, but API integration may be used in the processing after the transaction information has been gathered into a single device. Therefore, even if some transaction information is acquired via API integration in the process of gathering the transaction information into a single device, if there is transaction information acquired without relying on API integration, it will still be considered "aggregating transaction information without relying on API integration."

[0011] The technology disclosed herein processes images containing information on multiple transactions. Here, in this disclosure, unity of transactions means unity of settlement. Unity of transactions is determined by whether at least one of the following is the same: the time of settlement, the counterparty, the amount, or the content. To illustrate with an example of determining unity of transactions by considering the time of settlement or the counterparty, the transaction may be determined to be a single transaction if at least one of the time or the counterparty is the same, or if both the time and the counterparty are the same. Therefore, for example, when determining the unity of transactions in an example where multiple items were purchased and settled at a convenience store called Eight Eleven at 1:11 p.m. on July 11, 2025, the same time may be adopted as the criterion for determining unity of transactions, and the transaction may be determined to be a single transaction at 1:11 p.m. on July 11, 2025. As another example, when determining the unity of a transaction in which one or more items are purchased and paid for at a convenience store named Eight Eleven, and one or more items are purchased and paid for at a convenience store named Hyson, the identity of the counterparty may be adopted as the criterion for determining the unity of the transaction, and the transaction may be determined to be two transactions, including a single transaction at Eight Eleven and a single transaction at Hyson. As yet another example, consider a case where, at 1:11 p.m. on July 11, 2025, a customer purchases and pays for multiple items at a convenience store called Eight Eleven, and at 1:11 p.m. on July 11, 2025, a customer purchases and pays for multiple items at a convenience store called Hyson. By adopting the criterion of both the time and the counterparty being the same for determining the unity of transactions, the transaction may be determined to consist of two transactions: the transaction at Eight Eleven at 1:11 p.m. on July 11, 2025, and the transaction at Hyson at 1:11 p.m. on July 11, 2025.

[0012] <System Configuration> The following describes an example of the configuration of the information processing system 1 according to an embodiment of the present disclosure, with reference to Figure 1. As shown in Figure 1, the information processing system 1 is, in one aspect, a system in which a user terminal 2, a server device 3, a store terminal 4, and an LLM server device 5 are connected via a network NW. The network NW is an existing telecommunications line such as the Internet.

[0013] For example, user terminal 2 has one or more payment applications (hereinafter, applications are simply referred to as "apps") pre-installed, and user U of user terminal 2 conducts transactions using the payment app in a store equipped with a store terminal 4 that supports payments via payment apps, either using a customer display method or a store display method.

[0014] In the customer display method, when user U pays for goods at a store equipped with a store terminal 4, user U operates their own app to display a 2D barcode, such as a QR code (registered trademark), representing user U's ID number on the display device of user terminal 2 and presents it to the store. The store's POS reads the presented 2D barcode, and store terminal 4 transmits the read ID number, as well as transaction information such as the store name, payment date and time, and payment amount, to a payment service provider's server device (not shown) to complete the payment. The payment service provider's server device transmits the payment result to store terminal 4 and user terminal 2.

[0015] In the store display method, user U reads a two-dimensional barcode, such as a QR code (registered trademark) representing the store's ID number, displayed on the store terminal 4 or in the store, using the camera on user terminal 2. User U enters the payment amount into the payment app, and user terminal 2 transmits transaction information such as user ID, store ID, payment date and time, and payment amount to a payment service provider's server device (not shown) to complete the payment. The payment service provider's server device (not shown) transmits the payment result to store terminal 4 and user terminal 2.

[0016] In this disclosure, the term "product" includes services.

[0017] In addition, the user U may conduct a transaction using the payment card UC at the store where the store terminal 4 is installed. Examples of payment cards include any card that can be used for payment, such as a prepaid card, a cash card, a debit card, or a credit card. When the user U purchases goods at a store equipped with the store terminal 4 compatible with the payment card UC and pays for the goods using the payment card UC, the store terminal 4 transmits various transaction information related to the commercial transaction, such as the store name, product name, service name, payment date and time, and identification information of the payment card UC, to a server device (not shown) used for providing services using the payment card UC. A confirmation app for the usage details of the payment card UC, such as a credit card, may be installed on the user terminal 2.

[0018] In this way, the user U conducts transactions using the user terminal 2 or the payment card UC. The user U can check the transaction history on the user terminal 2 using a payment app or a confirmation app for the usage details of the payment card UC, such as a credit card.

[0019] Note that the service system may be constructed so that the user U can check the transaction history based on a paper statement. In the present disclosure, a "statement" means a record of multiple transactions.

[0020] The technology of the present disclosure aims to support the user U in reflecting their own transaction history in a management app such as a household ledger. To achieve such an objective, in one aspect, the information processing system 1 provides a function of performing OCR (optical character recognition) processing on the image data of a screen shot image of an app such as a payment app captured by the user terminal 2 to convert the image data into text data and reading multiple transaction information from the text data. Instead of the screen shot image, similar processing may be performed on an image captured by a camera. In the present disclosure, the screen shot image and the display image are used as interchangeable terms.

[0021] To realize such functions, the information processing system 1 may be configured, for example, to provide an OCR processing function at the user terminal 2, transmit the text data read by the OCR processing from the user terminal 2 to the server device 3, and have the server device 3 read the transaction information based on the text data. Alternatively, the user terminal 2 may be configured to acquire and transmit image data, and the server device 3 may be configured to implement the OCR processing function and the function of reading transaction information based on the text data. The configuration of each device or terminal will be described below.

[0022] <User terminal> As shown in Figure 2, the user terminal 2 includes an input device 22, a display device 23, a control device 24, and a communication device 25 on one side. The user terminal 2 may further include a camera 21. The camera 21 can capture details containing multiple transaction information to generate a camera image, and the camera image may be processed in the same way as a screenshot image.

[0023] (camera) Camera 21 is a device that captures details and generates images based on user input via input device 22, in accordance with the control of control device 24. Camera 21 transmits the generated images to control device 24. More specifically, camera 21 captures details of multiple transactions issued after user U makes a payment with payment card UC based on user input, generates camera images, and transmits the generated camera images to control device 24.

[0024] (Input device) The input device 22 is implemented by an input device such as a touch panel or mechanical keys, and provides an interface for user U to operate user terminal 2. The input device 22 receives instructions from user U and transmits the received instructions to control device 24.

[0025] For example, it accepts various inputs related to acquiring, selecting, processing, or transmitting display images, such as instructions to take a screenshot to capture the screen displayed on the display device 23, or instructions to select the target from among multiple captured display images from which transaction information should be read.

[0026] Furthermore, the input device 22 may accept various inputs related to the acquisition, selection, processing, or transmission of camera images, such as instructions to activate the camera 21, instructions for shutter timing, and instructions to select the target from among multiple acquired camera images from which transaction information should be read.

[0027] The input device 22 receives a read request to read transaction information from a displayed image or camera image based on user operation by user U. The input device 22 supplies the received read request to the control device 24.

[0028] (display device) The display device 23 is a display device such as a liquid crystal panel that provides information to the user U visually. The display device 23 may be implemented as a touch panel together with the input device 22, or it may be implemented as a separate display device from the input device 22. The display device 23 may display camera images captured by the camera 21 and data processed by the control device 24, and may also visually feed back the results of operations from the input device 22.

[0029] (Communication equipment) The communication device 25 is a device that communicates with other devices connected to the network NW, such as the server device 3. On one side, the communication device 25 transmits text data output from the control device 24 to the server device 3.

[0030] On the other hand, the communication device 25 receives data transmitted from the server device 3 to the user terminal 2 and supplies the received data to the control device 24. For example, the communication device 25 receives structured transaction information, which will be described later, and supplies it to the control device 24.

[0031] (Control device) As an example, the control device 24 implements a function to perform OCR processing on the display image data of a screenshot. Alternatively, the control device 24 may implement a function to perform OCR processing on the camera image data of a camera image. To implement such functions, the control device 24 includes a control unit 240, a display image acquisition unit 2401, a camera image acquisition unit 2402, an image processing control unit 2403, an OCR processing unit 2404, an output control unit 2405, a storage unit 2406, and a user input acquisition unit 2407. The functional units of the control device 24 will be described below.

[0032] (Control Unit) The control unit 240 controls the overall operation of each functional unit of the control device 24. The control unit 240 may also perform various functions. For example, the control unit 240 may be responsible for processing payments in a payment application.

[0033] (Image acquisition unit) The display image acquisition unit 2401 is an example of an image acquisition unit in this disclosure. The image acquisition unit is a functional unit that acquires image data of at least one target image on which multiple transaction information of the user is displayed. The display image acquisition unit 2401 acquires the display image displayed on the display device 23 according to instructions from the input device 22 by user operation. User U operates the input device 22 to give an instruction to take a screenshot of an image to be captured, such as the transaction history of a payment application or the transaction history of a credit card, and the user input acquisition unit 2407 acquires the screenshot instruction from the input device 22. The display image acquisition unit 2401 acquires the image displayed on the display device 23 at the time the screenshot instruction acquired by the user input acquisition unit 2407 was given from the storage unit 2406. The display image acquisition unit 2401 outputs the display image data of the acquired display image to the OCR processing unit 2404.

[0034] (Camera image acquisition unit) The camera image acquisition unit 2402 is an example of an image acquisition unit in this disclosure. The camera image acquisition unit 2402 acquires images captured by the camera 21 from the camera 21. User U operates the input device 22 to capture details on paper using the camera 21, and the camera 21 supplies the camera image data of the generated camera image to the camera image acquisition unit 2402. The camera image acquisition unit 2402 outputs the acquired camera image data to the OCR processing unit 2404.

[0035] (Image Processing Control Unit) In this disclosure, the image processing control unit 2403 is an example of a processing control unit. The processing control unit is a functional unit that determines, at the time of acquiring image data of at least one target image, whether the number of times the OCR processing unit 2404 has performed OCR processing on image data or the acquisition of multiple structured transaction information by the transaction information acquisition unit 3204 during a predetermined period (for example, one month) has reached the upper limit of the number of processing operations set for user U for each predetermined period, and if it is determined that the count has not reached the upper limit, it determines that the OCR processing unit 2404 will perform OCR processing on the image data and the transaction information acquisition unit 3204 will perform the acquisition of multiple structured transaction information. The processing control unit may be provided as a single functional unit, or it may be provided as two functional units depending on whether it is OCR processing or the acquisition of structured transaction information. If the processing control unit is provided as two functional units, one may be provided as the image processing control unit 2403 and the other as the control unit 320. Furthermore, the counting of the number of attempts may be performed by the processing control unit, or by a separate functional unit (for example, the control unit 240). The processing control unit outputs control information indicating the determination result to the OCR processing unit 2404 or the transaction information acquisition unit 3204. Note that if the OCR function is implemented in the server device 3, the image processing control unit 2403 may be provided in the server device 3.

[0036] The image processing control unit 2403, as an example of a processing control unit, is a functional unit that determines whether or not to perform OCR image processing on the image data of a display image captured by the display image acquisition unit 2401 or a camera image acquired by the camera image acquisition unit 2402. The criteria for this determination may be defined according to the user U's service plan subscription status, as described later.

[0037] When the image processing control unit 2403 receives a read request from the input device 22, it accesses the storage unit 2406 to obtain information indicating the number of times user U can use the transaction information reading service (i.e., the number of times it can process). This number of available uses (number of times it can process) is set for predetermined periods based on user U's attributes. Here, examples of user attributes include the service plan the user subscribes to, the household budgeting service the user subscribes to, and, if the information processing system 1 constitutes a payment system, various items or categories such as the number of payments the user has made, the payment amount, and the amount deposited (in a prepaid card or account) during the same or different period as the predetermined period. The user's attributes are determined considering these items or categories, and the number of available uses (number of times it can process) is set for predetermined periods according to the determined user attributes. The number of available uses (number of times it can process) may be determined based on the number of images, the number of lines contained in the images, or the number of transactions.

[0038] For example, if the number of available uses is 1 or more, the image processing control unit 2403 determines that it will perform image processing using OCR and supplies the determination result to the OCR processing unit 2404. It also decrements the number of available uses by 1 and stores information indicating the number of available uses after the decrement in the storage unit 2406.

[0039] If the number of available uses is 0, the image processing control unit 2403 determines that it will not perform image processing by OCR and supplies the determination result to the OCR processing unit 2404. It also supplies the determination result to the control unit 240, which then performs display control via the display device 23 to notify the user that the transaction information reading service will be available once the service is upgraded.

[0040] Thus, the control of the number of processing uses may be performed by setting the maximum number of processing uses as an initial value and decrementing it each time processing is performed, or by setting the initial number of processing uses to 0 and determining whether the maximum value has been reached by incrementing it each time processing is performed.

[0041] (OCR processing load) Hereinafter, screenshot images (display images) or camera images containing multiple transaction information will be collectively referred to as target images. The OCR processing unit 2404 is a functional unit that acquires image data of the target images from the display image acquisition unit 2401 or the camera image acquisition unit 2402, performs OCR processing on the image data of the target images, and converts the information such as characters, numbers, and symbols contained in the target images into line-by-line text data. If there are multiple target images, OCR processing may be performed on only some of the multiple target images based on user input. In order to convert into line-by-line text data, the OCR processing unit 2404 may detect a rectangular area containing characters corresponding to one line, perform OCR processing on each of the multiple rectangular areas contained in the target image, and extract text data from the 1st line to the Nth line (where N is a positive integer). Alternatively, the OCR processing unit 2404 may use the positional information of each character within the acquired image, which can be obtained by OCR processing, to convert the information in the acquired image into line-by-line text data. More specifically, characters that are substantially the same in height within the image may be grouped as characters in the same line and converted into line-by-line text data. The OCR processing unit 2404 stores the extracted text data in the storage unit 2406 along with the position information of line n (n=1 to N). The position information of line n may be position information indicating the position and size of a rectangular area. In cases where a single (logo-like) character spanning multiple lines (such as the "8" in R21) and strings in multiple lines within the display area R2 are mixed, as in the character in display area R2 of the screenshot image SS1 described later, the range in which the size or height of the characters is recognized as the same may be different based on the results of the pattern matching described later. When the OCR processing is complete, the OCR processing unit 2404 outputs information indicating completion to the output control unit 2405.

[0042] Furthermore, the OCR processing unit 2404 may identify type information indicating the type of details in the target image, for example by pattern matching, and output the result of the identification. Here, an example of the details in the target image will be explained in more detail with reference to Figures 6 and 7. Figures 6 and 7 are examples of screenshots of the transaction history screen of a fictitious identical payment application. Figures 6 and 7 show the transaction history of that fictitious payment application for July 2025.

[0043] The screenshot image SS1 in Figure 6 broadly includes six display areas R1 to R6. The screenshot image SS2 in Figure 7 is a screenshot image taken by scrolling down the screen in which the screenshot image SS1 in Figure 6 was taken. The screenshot image SS2 in Figure 7 broadly includes six display areas R1, R5 to R9.

[0044] Display area R1 is a fixed display area whose content does not change even when scrolling down. Display areas R2 to R9 are variable display areas whose content changes when scrolling down.

[0045] In display area R1, display area R11 displays the logo of the payment app BayBay, labeled "Bay2," while the display areas other than R11 display the numbers and text "July 2025 Transaction History." The term "text" is used as a general term for numbers, letters, or symbols.

[0046] In display area R2, the logo "8" of Eight Eleven is displayed in display area R21, and in display area R12, the transaction information shows text indicating that a payment of 1,200 yen was completed to Eight Eleven on July 1, 2025 at 13:22.

[0047] In display area R3, the Hyson logo "HS" is shown in display area R31, and in display area R12, transaction information is shown indicating that a payment of 980 yen was completed to Hyson on July 3, 2025 at 9:17.

[0048] In display area R4, the Eight Eleven logo "8" is displayed in display area R41, and in display area R42, the transaction information shows text indicating that a payment of 2,400 yen was completed to Eight Eleven at 18:45 on July 7, 2025.

[0049] In display area R5, the Hyson logo "HS" is shown in display area R51, and in display area R52, transaction information is shown indicating that a payment of 1050 yen was completed to Hyson at 11:05 on July 10, 2025.

[0050] In display area R6, the Eight Eleven logo "8" is displayed in display area R61, and in display area R62, the transaction information shows text indicating that a payment of 1800 yen was completed to Eight Eleven at 20:33 on July 13, 2025.

[0051] In display area R7, the Hyson logo "HS" is shown in display area R71, and in display area R72, transaction information is shown indicating that a payment of 1400 yen was completed to Hyson at 18:55 on July 17, 2025.

[0052] In display area R8, the logo "8" of Eight Eleven is displayed in display area R81, and in display area R82, the transaction information shows text indicating that a payment of 1150 yen was completed to Eight Eleven at 11:11 on July 19, 2025.

[0053] In display area R9, the Hyson logo "HS" is shown in display area R91, and in display area R92, transaction information is shown indicating that a payment of 1,500 yen was completed to Hyson at 19:33 on July 22, 2025.

[0054] In the case of screenshot images SS1 and SS2 as described above, the BayBay logo "Bay2" displayed in the display area R11 allows identification that it is the BayBay payment app. Once the type of transaction can be identified in this way, subsequent processing can be performed with greater accuracy using the identified information. Therefore, the OCR processing unit 2404 may identify the type of transaction by performing pattern matching between a predetermined pattern image and the target image such as the screenshot image. The template image such as "Bay2" used in pattern matching is stored in the storage unit 2406, and the OCR processing unit 2404 accesses the storage unit 2406 to obtain the template image. The OCR processing unit 2404 may output type information indicating the identified type of transaction.

[0055] (Output control unit) The output control unit 2405 is a functional unit that controls the output of data or signals to the camera 21, the display device 23, or the communication device 25. To the camera 21, it outputs instructions such as shutter timing. To the display device 23, it performs display control to display image data held in the storage unit 2406.

[0056] Furthermore, the output control unit 2405 outputs the data processed by the control device 24 to the communication device 25. More specifically, when it receives information from the OCR processing unit 2404 that the OCR processing is complete, it accesses the storage unit 2406 and instructs it to transmit the text data obtained by the OCR processing.

[0057] Furthermore, the control unit 240 or an acquisition unit (not shown) of the control device 24 receives data indicating structured transaction information supplied from the communication device 25, and the output control unit 2405 performs display control to display various information such as structured transaction information indicated by the received data on the display device 23. When the output control unit 2405 performs the process of acquiring structured transaction information from two or more target images, it performs display control of the display device 23 to display multiple transaction information acquired from each of the two or more target images together to the user U.

[0058] (Storage part) The storage unit 2406 is a functional unit that stores information used for processing by the user terminal 2, information generated by the user terminal 2, and so on. As an example, the storage unit 2406 stores user U's attribute information as information used for processing by the user terminal 2. User U's attribute information includes various information such as an ID that uniquely identifies user U, user U's subscription plan information, and information on the number of times the reading service can be used. The upper limit on the number of times it can be used may be determined according to the user's subscription status to a service plan related to a service provider, such as a household budget app, provided by a service provider using the server device 3. For example, if there are multiple service plans, such as 10 times / month for service plan A and 20 times / month for service plan B, user U can use the reading service 10 times / month or 20 times / month depending on their subscription status. The storage unit 2406 stores this specific information related to user U's attribute information, linking them together, for example, in a table format.

[0059] Furthermore, the memory unit 2406 stores image data of payment completion, transaction history, etc., which are generated as interfaces for payment apps, transaction history confirmation apps, etc.

[0060] Furthermore, the memory unit 2406 stores the data of the camera image captured by the camera 21.

[0061] (User input acquisition unit) The user input acquisition unit 2407 is a functional unit that receives user input entered by the input device 22. The user input acquisition unit 2407 acquires user input via the input device 22 that specifies the target image to be processed for reading transaction information from among a plurality of target images acquired by the display image acquisition unit 2401 or the camera image acquisition unit 2402.

[0062] <Server equipment> Next, the configuration of the server device 3 will be described with reference to Figure 3. In one aspect, the server device 3 provides a function to acquire multiple desired transaction information based on text data relating to target images such as display images transmitted from the user terminal 2. To provide such a function, as an example, the server device 3 includes a communication device 31, a control device 32, and a storage device 33.

[0063] (Communication equipment) The communication device 31 is a device that communicates with other devices connected to the network NW, such as the user terminal 2, the store terminal 4, and the LLM server device 5. For example, the communication device 31 receives text data transmitted from the user terminal 2 and transmits information obtained from the text data to the user terminal 2. The communication device 31 also sends prompts to the LLM server device 5 and receives responses to the prompts from the LLM server device 5.

[0064] (Control device) The control device 32 generates a prompt to be input to the LLM of the LLM server device 5 based on the text data transmitted from the user terminal 2, obtains a response to the prompt from the LLM, and sends the obtained response to the user terminal 2.

[0065] (storage device) The storage device 33 consists of a hard disk or an SSD (Solid State Drive) and is a device that stores various information necessary to provide services to user U.

[0066] In one aspect, the server device 3 provides a function to acquire multiple desired transaction information based on text data transmitted from the user terminal 2. To provide this function, the control device 32 of the server device 3 includes, as an example, a control unit 320, a data acquisition unit 3201, a prompt generation unit 3202, a prompt input unit 3203, a transaction information acquisition unit 3204, a duplicate determination processing unit 3205, and an output control unit 3206. These functional units will be described below.

[0067] (Control Unit) The control unit 320 controls the overall operation of each functional unit provided by the control device 32.

[0068] (Data acquisition unit) The data acquisition unit 3201 acquires text data of the target image obtained by OCR processing from the communication device 31 and outputs the acquired text data to the prompt generation unit 3202.

[0069] (Prompt generation unit) The prompt generation unit 3202 generates a prompt that includes line-by-line text data for each image obtained through OCR processing via the data acquisition unit 3201, and a command statement that instructs the extraction of transaction information from the line-by-line text data according to a predetermined format. This allows the large-scale language model to structure more accurately based on the line-by-line data contained in the text data. The command statement may be stored in the storage device 33 as a pre-prepared standard phrase, and the prompt generation unit 3202 may access the storage device 33 to obtain the command statement and generate a prompt.

[0070] There may be multiple types of formats as predetermined formats. Here, in this disclosure, "format" means a format that specifies the information that the LLM should output in order for the LLM to extract specific structured information from the information contained in the text data. The format may be for describing structured data such as JSON. The format specifies the items to be imported into the household budget app, and includes items such as payee, settlement amount, and date.

[0071] Such formats can come in multiple variations. For example, different formats may be available depending on the payment app or the credit card statement viewing app. By using different formats depending on the type of statement, it becomes possible to extract transaction information with greater accuracy.

[0072] The type of item may be identified by the OCR processing unit 2404, by the prompt generation unit 3202, or by user U.

[0073] When identification is performed by the OCR processing unit 2404, the target image is identified using pattern matching or the like, as described above.

[0074] When the prompt generation unit 3202 identifies the target, it does so by determining whether the text data obtained by the OCR process contains a predetermined string (for example, the string "Bay2").

[0075] When the format is specified by user input, the control device 24 may perform display control to display a screen on the user terminal 2's display device 23 that allows the user to select a format when receiving a request to read the text of the target image. For example, it may be configured to display multiple types of application names along with a question such as "Which application is the target of the text reading?" and accept the selection.

[0076] The prompt generation unit 3202 may include information in the prompt indicating the structure in which the information is displayed, depending on the type of item identified. Once the type of item is identified, the specifications related to the display format of the transaction history corresponding to that type of item can be identified as publicly known. In this way, even if the display format of the item differs depending on the application, transaction information can be extracted with high accuracy.

[0077] (Prompt input section) The prompt input unit 3203 is a functional unit that inputs the prompt generated by the prompt generation unit 3202 to the LLM. In the examples shown in Figures 1 to 3, the prompt input unit 3203 performs transmission control to send the prompt to the LLM server device 5. In order to send the prompt to the LLM server device 5 and obtain a response, the prompt input unit 3203 adds an API key for using the LLM server device 5 to the prompt generated by the prompt generation unit 3202 and then sends it.

[0078] (Transaction Information Acquisition Department) The transaction information acquisition unit 3204 is a functional unit that acquires multiple structured transaction information obtained by inputting the generated prompt into the LLM. In the examples in Figures 1 to 3, the response returned by the LLM server device 5 is acquired as structured transaction information. When reading transaction information from multiple target images, the transaction information acquisition unit 3204 acquires multiple structured transaction information related to one target image and multiple structured transaction information related to other target images. For example, When reading transaction information from multiple target images, including a first target image containing multiple transaction information, a second target image containing multiple transaction information, and a third target image containing multiple transaction information, the transaction information acquisition unit 3204 acquires a first set of multiple structured transaction information related to the first target image, a second set of multiple structured transaction information related to the second target image, and a third set of multiple structured transaction information related to the third target image. The transaction information acquisition unit 3204 stores the multiple structured transaction information output from the LLM server device 5 in the storage device 33, linked to the user ID of user U, in a state where individual structured transaction information can be identified. In other words, each set of structured transaction information is distinguished from the others, and multiple set of structured transaction information is stored in the storage device 33, linked to the user ID.

[0079] (Duplicate detection processing unit) The duplicate detection processing unit 3205 is a functional unit that operates when reading transaction information from multiple target images. When reading from multiple target images, there may be cases where the information to be read is duplicated between one target image and another. The duplicate detection processing unit 3205 determines whether there is any such duplicate transaction information, and if there is duplicate transaction information, it may exclude the duplicate transaction information. The duplicate detection processing unit 3205 may exclude duplicate transaction information based on user input.

[0080] The duplicate detection processing unit 3205 compares, for example, the structured transaction information for each display area (R2~R6) obtained from screenshot image SS1 with the structured transaction information for each display area (R5~R9) obtained from screenshot image SS2, and determines the degree of similarity between the two using a rule-based approach or an LLM such as Claude. Specifically, it compares the structured transaction information for display area R2 obtained from screenshot image SS1 with the structured transaction information for each display area R5~R9 obtained from screenshot image SS2. Similarly, it compares the structured transaction information for each display area R3~R6 obtained from screenshot image SS1 with the structured transaction information for each display area R5~R9 obtained from screenshot image SS2. The degree of similarity may be determined for each structured transaction information or for each data item included in each structured transaction information. If the degree of similarity is above a predetermined threshold, it may be determined that the structured transaction information being compared is duplicated. The LLM used for duplicate detection may be a dedicated duplicate detection LLM held by the LLM server device 5, different from the LLM used to acquire structured transaction information, or it may be the same LLM held by the LLM server device 5. When using LLM for duplicate detection, the prompt should include a command statement instructing the LLM to show duplicate transaction information if duplicate transaction information exists, or a command statement instructing it to exclude duplicate transaction information.

[0081] If there is duplication of transaction information, it is expected that the duplicate transaction information will be included consecutively, so the duplicate detection process may be performed using the structured order of each transaction information. Referring to Figures 6 and 7, if there is duplicate transaction information between the transaction information of one screenshot image (for example, screenshot image SS1; the first target image) (for example, the transaction information in display area R5 of screenshot image SS1) and the transaction information of another screenshot image (for example, screenshot image SS2; the second target image) (for example, the transaction information in display area R5 of screenshot image SS2), it is expected that the duplicate transaction information will be included between the transaction information following the transaction information of the first screenshot image (screenshot image SS1) (the transaction information in display area R5 of screenshot image SS1) (the transaction information in display area R6 of screenshot image SS1) and the transaction information following the transaction information of the other screenshot image (screenshot image SS2) (the transaction information in display area R5 of screenshot image SS2) (the transaction information in display area R6 of screenshot image SS2). Generally, when there is a first set of multiple structured transaction information containing P (P is an integer greater than or equal to 2) ordered structured transaction information related to a first target image, and a second set of multiple structured transaction information containing Q (Q is an integer greater than or equal to 2) ordered structured transaction information related to a second target image, if it is determined that the p-th structured transaction information contained in the first set of multiple structured transaction information and the first structured transaction information contained in the second set of multiple structured transaction information are duplicates, then it can be estimated that the p+1 to p-th structured transaction information contained in the first set of multiple structured transaction information and the second to p-p+1-th structured transaction information contained in the second set of multiple structured transaction information are also duplicates. Note that when p=P-1, the p+1-th and p-th are the same. Therefore, if it is determined that there is a duplication of transaction information between the two images, the duplication determination for the transaction information that is estimated to be duplicated may be skipped using the order information of the transaction information that has been determined to be duplicated.To elaborate on the specific example described above, the determination of overlap between the p+1th to the Pth structured transaction information contained in the first set of structured transaction information and the 2nd to the P-p+1th structured transaction information contained in the second set of structured transaction information can be skipped. The order of the transaction information can be determined using the positional information of the text data read by OCR processing. By doing so, the determination of overlapping transaction information can be limited to a portion of the transaction information contained in an image, rather than all of it, thereby reducing the amount of computation and increasing processing speed.

[0082] (Output control unit) The output control unit 3206 outputs structured transaction information acquired by the transaction information acquisition unit 3204 (when there is one target image), structured transaction information determined to be duplicated by the duplicate determination processing unit 3205 (when there are two or more target images), or structured transaction information after duplicate structured transaction information has been removed by the duplicate determination processing unit 3205 (when there are two or more target images) to the user terminal 2. If there are two or more target images, the output control unit 3206 outputs multiple transaction information acquired from each of the two or more target images together to the user terminal 2. The output control unit 2405 of the user terminal 2 displays the received structured transaction information on the display device 23 and requests confirmation of whether the transaction information reading result is correct. The user terminal 2 accepts input on whether the read transaction information is correct or not. If it accepts input that it is correct, it performs processing to reflect the structured transaction information in a household budget application or the like. Specifically, if the read transaction information is correct, the newly read transaction information is combined with previously read past transaction information to calculate how much was spent within a specific period, and how much user U's balance and assets were at what point in time. This information is then output and displayed on user terminal 2 as expenditure information, balance information, and asset information. The calculation of expenditure information, balance information, and asset information may be performed by the control unit 24 of user terminal 2, or by the control unit 320 of server device 3. If the calculation of expenditure information, balance information, and asset information is performed on user terminal 2, past transaction information may be stored in the storage unit 2406 of user terminal 2. If the calculation of expenditure information, balance information, and asset information is performed on server device 3, past transaction information may be stored in the storage device 33 of server device 3. If user U is required to confirm the result of the duplicate detection, user terminal 2 displays the result of the duplicate detection, accepts instructions from user U to exclude duplicate transaction information according to their judgment, sends the exclusion instructions to server device 3, and the duplicate detection processing unit 3205 excludes the duplicate transaction information. If the read transaction information is incorrect, the system may be configured to accept corrections by user U through manual input.

[0083] <Control device hardware configuration> Next, with reference to Figures 4A and 4B, examples of the hardware configuration of the control devices 24 and 32 will be described. The control devices 24 and 32 are implemented by processing circuitry. The processing circuitry may be a dedicated processing circuit 100a as shown in Figure 4A, or a processor 100b that executes a program stored in memory 100c as shown in Figure 4B.

[0084] If the processing circuitry is a dedicated processing circuit 100a, then the dedicated processing circuit 100a may include, for example, an ASIC (application-specific integrated circuit) or an FPGA (field-programmable gate array).

[0085] When the processing circuitry is a processor 100b as a computer, the functions realized by the control devices 24 and 32 are described as programs, and these programs are stored in memory 100c. The processor 100b realizes the functions of the control devices 24 and 32 by reading and executing the programs stored in memory 100c. Examples of memory 100c include non-volatile or volatile semiconductor memories such as RAM (random access memory), DRAM (dynamic random access memory), ROM (read-only memory), flash memory, EPROM (erasable programmable read-only memory), and EEPROM (electrically erasable programmable read-only memory).

[0086] <Operation> Next, the operation of the information processing system 1 will be explained with reference to Figure 5.

[0087] (Step ST1) In step ST1, based on user operation, the display image acquisition unit 2401 acquires a display image, or the camera image acquisition unit 2402 acquires a camera image, as the target image.

[0088] (Step ST2) In step ST2, when the image processing control unit 2403 receives a request to read transaction information from the input device 22, it accesses the storage unit 2406 to determine whether user U can use the reading service. If it is not available, the process proceeds to step ST3; if it is available, the process proceeds to step ST5.

[0089] (Step ST3) In step ST3, the control unit 240 asks user U whether or not to perform a service upgrade and accepts the user U's response. If the user does not wish to upgrade, the flow shown in Figure 5 ends. If the user wishes to upgrade, the process proceeds to step ST4.

[0090] (Step ST4) In step ST4, the control unit 240 performs the necessary processing, such as billing, required to upgrade the service that user U is subscribed to.

[0091] (Step ST5) In step ST5, the OCR processing unit 2404 performs OCR processing on the acquired image, converting the information such as characters, numbers, and symbols contained in the acquired image into line-by-line text data. The OCR processing unit 2404 performs OCR processing on each acquired image.

[0092] (Step ST6) In step ST6, the prompt generation unit 3202 generates a prompt that includes text data for each image obtained by OCR processing acquired via the data acquisition unit 3201, and a command statement that instructs the extraction of transaction information from the text data in a predetermined format.

[0093] (Step ST7) In step ST7, the prompt input unit 3203 inputs the prompt generated by the prompt generation unit 3202 to the LLM.

[0094] (Step ST8) In step ST8, the transaction information acquisition unit 3204 obtains a response to the prompt from the LLM. The prompt contains a command to extract transaction information from text data containing transaction information in a predetermined format, so structured transaction information is obtained as the response from the LLM.

[0095] (Step ST9) Step ST9 is performed when there are multiple images to be processed; it is skipped if there is only one image to be processed.

[0096] In step ST9, as an example, the duplicate determination processing unit 3205 determines whether there is any overlapping structured transaction information between a first set of multiple structured transaction information obtained from LLM for one image and a second set of multiple structured transaction information obtained from LLM for another image. If there is overlapping structured transaction information, the duplicate structured transaction information is removed from either the first set of multiple structured transaction information or the second set of multiple structured transaction information. Rather than immediately removing the overlapping structured transaction information after determining that it exists, the duplicate determination processing unit 3205 may output the determination result and remove the information after confirmation of the determination result by the user U.

[0097] In step ST9, as another example, the duplicate determination processing unit 3205 may determine whether there is any overlapping structured transaction information between a first set of structured transaction information obtained from LLM for one image and a second set of structured transaction information obtained from LLM for another image. If there is overlapping structured transaction information, the unit may use the order information of the structured transaction information determined to be overlapping to determine whether there is any overlapping structured transaction information between the first set of structured transaction information or the second set of structured transaction information and the third set of structured transaction information. Referring to the examples in Figures 6 and 7, if it is determined that the display area R6 overlaps between screenshot image SS1 and screenshot image SS2, the unit may use the position information of the display area R6 in screenshot image SS1 to determine whether there is overlapping transaction information between screenshot image SS2 and an unshown screenshot image. The duplicate determination processing unit 3205 may exclude the overlapping structured transaction information based on the determination result that it is overlapping, or it may exclude it based on user judgment.

[0098] (Step ST10) In step ST10, the output control unit 2405 performs the process of displaying the read structured transaction information on the display device 23.

[0099] The above description describes a configuration in which user terminal 2 performs OCR processing on the image data of the target image to obtain text data, user terminal 2 transmits the text data to server device 3, and server device 3 uses LLM to obtain structured transaction information from the text data. The information processing technology disclosed in this disclosure is not limited to this configuration. For example, the OCR processing may be performed on server device 3. In this case, user terminal 2 transmits the image data of the target image to server device 3.

[0100] As another example, the text data structuring process using LLM may be performed on user terminal 2. Because LLM has a very large number of parameters, it has conventionally been difficult to store LLM in the storage device of a small terminal device such as a smartphone. However, the technology proposed in Non-Patent Document 1 below makes it possible to store LLM in the storage device of a small terminal device and perform inference. [Non-patent document 1] Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S. Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. 2024. LLM in a flash: Efficient Large Language Model Inference with Limited Memory. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12562-12584, Bangkok, Thailand. Association for Computational Linguistics.

[0101] Non-Patent Document 1, as its title suggests, is a document that teaches a technique to enable efficient inference using LLM even in environments with limited memory capacity (DRAM memory capacity) by storing LLM in flash memory. More specifically, it teaches a method of storing LLM model parameters in flash memory and moving the model parameters to DRAM as needed to perform inference. Following the instructions in this document, LLM may be installed on user terminal 2, text data obtained by OCR processing may be input to LLM for structuring processing, and the information after structuring processing may be acquired locally by user terminal 2.

[0102] Furthermore, it is possible to combine embodiments, or to modify or omit each embodiment as appropriate. [Industrial applicability]

[0103] The information processing technology disclosed herein can be applied to and used in a device or system that extracts multiple transaction information from an image, such as a screenshot image, that contains multiple transaction information. [Explanation of Symbols]

[0104] 1 Information processing system, 2 User terminal, 3 Server device, 4 Store terminal, 5 LLM server device, 21 Camera, 22 Input device, 23 Display device, 24 Control device, 25 Communication device, 31 Communication device, 32 Control device, 33 Storage device, 100a Processing circuit, 100b Processor, 100c Memory, 240 Control unit, 320 Control unit (processing control unit), 2401 Display image acquisition unit (image acquisition unit), 2402 Camera image acquisition unit (image acquisition unit), 2403 Image processing control unit (processing control unit), 2404 OCR processing unit, 2405 Output control unit, 2406 Storage unit, 2407 User input acquisition unit, 3201 Data acquisition unit, 3202 Prompt generation unit, 3203 Prompt input unit, 3204 Transaction information acquisition unit, 3205 Duplicate determination processing unit, 3206 Output control unit.

Claims

1. An image acquisition unit that acquires image data of at least one target image on which multiple transaction information of a user is displayed, An OCR processing unit performs OCR processing on the image data to convert the image data into text data, A prompt generation unit that generates a prompt including the text data and a command statement that instructs the extraction of the multiple transaction information from the text data in accordance with a predetermined format, A transaction information acquisition unit inputs the aforementioned prompt into a large-scale language model to acquire multiple structured transaction information in which the multiple transaction information is structured, An output control unit that controls the output of the plurality of structured transaction information to the user, Equipped with, The aforementioned at least one target image is a plurality of target images including a first target image and a second target image, The transaction information acquisition unit acquires a first plurality of structured transaction information relating to the first target image and a second plurality of structured transaction information relating to the second target image. A duplicate determination processing unit determines whether there is any overlapping structured transaction information between the first set of multiple structured transaction information and the second set of multiple structured transaction information, and if it determines that there is overlapping structured transaction information, it excludes the structured transaction information that was determined to be overlapping from either the first set of multiple structured transaction information or the second set of multiple structured transaction information, or outputs the determination result and excludes it based on user input. Furthermore, The output control unit controls the output of the transaction information to the user after the duplicate transaction information has been removed by the duplicate determination processing unit. Information processing system.

2. An image acquisition unit that acquires image data of at least one target image on which multiple transaction information of the user is displayed, An OCR processing unit performs OCR processing on the image data to convert the image data into text data, A prompt generation unit that generates a prompt including the text data and a command statement that instructs the extraction of the multiple transaction information from the text data in accordance with a predetermined format, A transaction information acquisition unit inputs the aforementioned prompt into a large-scale language model to acquire multiple structured transaction information in which the multiple transaction information is structured, An output control unit that controls the output of the plurality of structured transaction information to the user, Equipped with, The aforementioned at least one target image is a plurality of target images including a first target image and a second target image, The transaction information acquisition unit acquires a first plurality of structured transaction information, which includes P (where P is an integer of 2 or more) ordered structured transaction information pertaining to the first target image, and a second plurality of structured transaction information, which includes Q (where Q is an integer of 2 or more) ordered structured transaction information pertaining to the second target image. A duplicate determination processing unit determines whether there is any overlapping structured transaction information between the first set of multiple structured transaction information and the second set of multiple structured transaction information, and if it determines that the p-th structured transaction information (where p is an integer greater than or equal to 2 and less than P) included in the first set of multiple structured transaction information is an overlap with the first structured transaction information included in the second set of multiple structured transaction information, it estimates that the p+1th to Pth structured transaction information included in the first set of multiple structured transaction information and the second to P-p+1th structured transaction information included in the second set of multiple structured transaction information, and excludes the structured transaction information that has been determined to be an overlap or estimated to be an overlap from either the first set of multiple structured transaction information or the second set of multiple structured transaction information, or outputs the determination result or estimation result and excludes it based on user input. Furthermore, The output control unit controls the output of transaction information to the user after the transaction information that has been duplicated or estimated to be duplicated by the duplicate determination processing unit has been excluded. Information processing system.

3. An image acquisition unit that acquires image data of at least one target image on which multiple transaction information of the user is displayed, An OCR processing unit performs OCR processing on the image data to convert the image data into text data, A prompt generation unit that generates a prompt including the text data and a command statement that instructs the extraction of the multiple transaction information from the text data in accordance with a predetermined format, A transaction information acquisition unit inputs the aforementioned prompt into a large-scale language model to acquire multiple structured transaction information in which the multiple transaction information is structured, An output control unit that controls the output of the plurality of structured transaction information to the user, Equipped with, A processing control unit determines, at the time of acquiring image data of at least one target image, whether the number of times the OCR processing on the image data by the OCR processing unit or the acquisition of the multiple structured transaction information by the transaction information acquisition unit has been performed for a predetermined period has reached the upper limit of the number of processing times set for the user for each predetermined period, and if it is determined that the counted number has not reached the upper limit, it determines to perform at least one of the OCR processing on the image data by the OCR processing unit or the acquisition of the multiple structured transaction information by the transaction information acquisition unit. Furthermore, The OCR processing unit performs OCR processing on the image data when the processing control unit determines that OCR processing should be performed on the image data. The transaction information acquisition unit performs the acquisition process of the multiple structured transaction information when the processing control unit determines that it will perform the acquisition process of the multiple structured transaction information. Information processing system.

4. The prompt generation unit acquires type information indicating the type of detail in the image data, and generates a prompt using a format corresponding to the type information. An information processing system according to any one of claims 1 to 3.

5. The aforementioned at least one target image is a plurality of target images, A user input acquisition unit that acquires user input specifying two or more target images from the aforementioned multiple target images for processing the reading of transaction information. Furthermore, The OCR processing unit performs OCR processing on the image data of the two or more target images specified by the user input. The output control unit performs control to output a collection of transaction information obtained from each of the two or more target images to the user. The information processing system described in claim 4.

6. The OCR processing unit converts the image data into line-by-line text data based on the positional information of the text contained in the image data. The prompt generation unit generates a prompt that includes the line-by-line text data and a command statement that instructs the unit to structure and extract the multiple transaction information from the line-by-line text data according to a predetermined format. The information processing system described in claim 4.

7. An image acquisition unit that acquires image data of at least one target image on which multiple transaction information of a user is displayed, An OCR processing unit performs OCR processing on the image data to convert the image data into text data, A prompt generation unit that generates a prompt including the text data and a command statement that instructs the extraction of the multiple transaction information from the text data in accordance with a predetermined format, A transaction information acquisition unit inputs the aforementioned prompt into a large-scale language model to acquire multiple structured transaction information in which the multiple transaction information is structured, An output control unit that controls the output of the plurality of structured transaction information to the user, Equipped with, The prompt generation unit acquires type information indicating the type of detail in the image data, and generates a prompt using a format corresponding to the type information. Information processing system.

8. A prompt generation unit generates a prompt that includes text data obtained by performing OCR processing on image data of at least one target image displaying multiple transaction information of a user, and a command statement that instructs the unit to structure and extract the multiple transaction information from the text data according to a predetermined format. A transaction information acquisition unit inputs the aforementioned prompt into a large-scale language model to acquire multiple structured transaction information in which the multiple transaction information is structured, An output control unit that controls the output of the plurality of structured transaction information to the user, Equipped with, The aforementioned at least one target image is a plurality of target images including a first target image and a second target image, The transaction information acquisition unit acquires a first plurality of structured transaction information relating to the first target image and a second plurality of structured transaction information relating to the second target image. A duplicate determination processing unit determines whether there is any overlapping structured transaction information between the first set of multiple structured transaction information and the second set of multiple structured transaction information, and if it determines that there is overlapping structured transaction information, it excludes the structured transaction information that was determined to be overlapping from either the first set of multiple structured transaction information or the second set of multiple structured transaction information, or outputs the determination result and excludes it based on user input. Furthermore, The output control unit controls the output of the transaction information to the user after the duplicate transaction information has been removed by the duplicate determination processing unit. Server device.

9. A prompt generation unit generates a prompt that includes text data obtained by performing OCR processing on image data of at least one target image displaying multiple transaction information of a user, and a command statement that instructs the unit to structure and extract the multiple transaction information from the text data according to a predetermined format. A transaction information acquisition unit inputs the aforementioned prompt into a large-scale language model to acquire multiple structured transaction information in which the multiple transaction information is structured, An output control unit that controls the output of the plurality of structured transaction information to the user, Equipped with, The prompt generation unit acquires type information indicating the type of detail in the image data, and generates a prompt using a format corresponding to the type information. Server device.

10. An information processing method performed by a computer comprising an image acquisition unit, an OCR processing unit, a prompt generation unit, a transaction information acquisition unit, a duplicate determination processing unit, and an output control unit, The aforementioned information processing method is: A step of obtaining image data of at least one target image on which multiple transaction information of a user is displayed, wherein the at least one target image is a plurality of target images including a first target image and a second target image, The steps include: performing OCR processing on the image data to convert the image data into text data; A step of generating a prompt that includes the text data and a command statement that instructs the extraction of the multiple transaction information from the text data in accordance with a predetermined format, A step of inputting the prompt into a large-scale language model to obtain a plurality of structured transaction information in which the plurality of transaction information is structured, comprising the step of obtaining a first plurality of structured transaction information relating to the first target image and a second plurality of structured transaction information relating to the second target image, The process involves determining whether there is any overlapping structured transaction information between the first set of structured transaction information and the second set of structured transaction information, and if it is determined that there is overlapping structured transaction information, excluding the overlapping structured transaction information from either the first set of structured transaction information or the second set of structured transaction information, or outputting the determination result and excluding it based on user input. A step of controlling the output of the plurality of structured transaction information to the user, comprising a step of controlling the output of the transaction information after duplicate transaction information has been removed to the user in a consolidated manner, It is equipped with.

11. A function to acquire image data of at least one target image on which multiple transaction information of a user is displayed, wherein the at least one target image is a plurality of target images including a first target image and a second target image, A function to perform OCR processing on the aforementioned image data and convert the image data into text data, A function that generates a prompt including the text data and a command statement that instructs the extraction of the multiple transaction information from the text data in accordance with a predetermined format, A function that inputs the aforementioned prompt into a large-scale language model to obtain a plurality of structured transaction information in which the plurality of transaction information is structured, the function of obtaining a first plurality of structured transaction information relating to the first target image and a second plurality of structured transaction information relating to the second target image, The system includes a function to determine whether there is any overlapping structured transaction information between the first set of multiple structured transaction information and the second set of multiple structured transaction information, and if it is determined that there is overlapping structured transaction information, to exclude the overlapping structured transaction information from either the first set of multiple structured transaction information or the second set of multiple structured transaction information, or to output the determination result and exclude it based on user input, A function for controlling the output of the aforementioned multiple structured transaction information to the user, comprising a function for controlling the output of the transaction information after duplicate transaction information has been removed, to the user in a consolidated manner, An information processing program that causes a computer to execute something.

Images