System

Abstract

A system is provided.SOLUTION: A system comprising: a user device for inputting contract information; a server for converting the input contract information into text data using a character recognition technique or a speech recognition technique; a database for storing the contract information; and a generated AI model for providing the bereaved family with the contract information and a cancellation procedure method.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] As Japan's super-aging society advances, the increase in the number of elderly people living alone is becoming a problem. Under these circumstances, there are many cases where the contract information and social relationships held by the individual are not properly passed on to the surviving family members. This creates a problem in that it takes a great deal of time and effort for the surviving family members to understand the contract information and carry out the cancellation procedures. The present invention aims to solve these problems by providing a system that enables the surviving family members to smoothly obtain the contract information and smoothly proceed with the cancellation procedures. [Means for solving the problem]

[0005] The present invention is a system including the following means: user terminal means for inputting contract information, server means for converting the input contract information into text data using character recognition or voice recognition technology, means for analyzing the text data to extract the contract information and saving it in a database, means for verifying the identity of surviving family members registered from the user terminal means when they access the system, and generation AI model means for providing contract information and cancellation procedure methods to surviving family members. This system enables centralized management of contract information, allowing surviving family members to smoothly obtain the necessary information and carry out cancellation procedures.

[0006] "Contract information" refers to information about various services and products that an individual has contracted for, and includes detailed information such as the name of the contractor, contract details, contract number, and contract period.

[0007] A "user terminal" is a device for entering and accessing contract information, and refers to electronic devices such as smartphones, tablets, and personal computers.

[0008] "Character recognition technology" is a technology that extracts character information from image data and converts it into text data, and includes optical character recognition (OCR) technology.

[0009] "Speech recognition technology" is a technology that analyzes the content of speech from voice data and converts it into text data, and extracts linguistic information from voice.

[0010] A "server" is a central computer system that receives, processes, stores, and provides data, and communicates with user terminals via a network.

[0011] A "database" is an information repository that systematically stores contract information and other related data and allows it to be searched and retrieved as needed.

[0012] "Identity verification" is a procedure to confirm that a user has legitimate access rights, and may involve questions, a QR code (registered trademark), a password, or the like.

[0013] A "generative AI model" is an artificial intelligence model built to analyze contract information and provide optimal cancellation procedures, utilizing machine learning and natural language processing technologies.

[0014] "Cancellation Procedures" refers to information that indicates the specific steps and procedures required to terminate a contract, including necessary documents and contact information. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] As an embodiment of the invention, the system that centrally manages contract information and smoothly processes cancellations is shown below. This system consists of a user terminal, a server, and a generative AI model.

[0037] 1. Enter contract information

[0038] First, the user (elderly person) enters the contract information. To do this, the user uses a device such as a smartphone or PC. They either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input.

[0039] 2. Data processing by the server

[0040] The device then sends the captured image or audio data to the server, which processes the received data. The image data is converted into text using optical character recognition (OCR) technology, and the audio data is similarly converted into text using automatic speech recognition (ASR) technology.

[0041] 3. Analysis and storage of contract information

[0042] The server analyzes the converted text data and extracts the necessary contract information (contractor name, contract details, contract number, etc.). The extracted contract information is stored in a database. The server also provides this contract information to a generative AI model for future cancellation procedures.

[0043] 4. User Access and Identity Verification

[0044] When a user dies, the surviving family members access the system. At this stage, the surviving family members (users) log in using a user terminal and begin the process of verifying the contract information of the deceased. The server performs identity verification to verify the identity of the surviving family members, and provides information only if this is successful.

[0045] 5. Provision of cancellation procedure information

[0046] The server uses the contract information stored in the generative AI model to provide details of the necessary cancellation procedures, including the necessary documents, contact information, and specific cancellation steps. Survivors can use this information to smoothly complete the cancellation procedures for various contracts.

[0047] Specific examples

[0048] Example 1: Entering a contract

[0049] User: An elderly person launches the smartphone app and takes a photo of their electricity contract and uploads it.

[0050] Terminal: Sends the contract image to the server.

[0051] Server: The server receives the image and converts it into text data using OCR technology. It then analyzes the contract information and stores it in a database.

[0052] Example 2: Obtaining contract information by a surviving family member

[0053] User (surviving family member): The deceased person's surviving family member logs into the system and checks the cancellation information for the electricity contract.

[0054] Server: Performs identity verification, and if successful, extracts contract information and queries the generation AI model.

[0055] Generative AI model: Generates optimal cancellation details and provides them to the bereaved family, who then follow the cancellation process.

[0056] In this way, the system efficiently manages the contract information of elderly people and helps surviving family members to complete the procedures quickly and smoothly.

[0057] The processing flow will be explained below.

[0058] Step 1:

[0059] The user enters the contract information. Specifically, the user takes a picture of the contract using a device such as a smartphone or PC, or explains the contract details using voice input. The user's device then uploads this input data to the application.

[0060] Step 2:

[0061] The device sends the input data to the server. Specifically, image data or audio data of the photographed contract is sent to the server via the Internet. The data may be encrypted during transmission.

[0062] Step 3:

[0063] The server processes the received data. Specifically, image data is converted to text using OCR technology, and voice data is converted to text using automatic speech recognition (ASR). Through this process, the contents of the contract and the contract contents explained in the voice are extracted as text data.

[0064] Step 4:

[0065] The server analyzes the text data. Specifically, it extracts the necessary contract information (e.g., contract holder name, contract details, contract number, etc.) from the text data and organizes it as contract information. The analysis results are saved in a database.

[0066] Step 5:

[0067] The server provides the contract information to the generative AI model. Specifically, the extracted and organized contract information is passed to the generative AI model, and the model is updated based on that information to generate recommended cancellation procedures.

[0068] Step 6:

[0069] The user (surviving family member) accesses the system. Specifically, the surviving family member logs into the system using a user terminal and inputs that they would like to check the contract information of the deceased person.

[0070] Step 7:

[0071] The server performs identity verification. Specifically, it uses security questions, QR codes, passwords, etc. to verify the identity of the family member. Only if identity verification is successful is access to the contract information granted.

[0072] Step 8:

[0073] The server extracts contract information. Specifically, it searches the database for contract information of the deceased person and extracts the relevant contract information. It then queries the generative AI model to generate cancellation procedure information.

[0074] Step 9:

[0075] The server provides cancellation procedure information, specifically detailed cancellation procedures provided by the generative AI model, as well as necessary documents and contact information, to the bereaved family, helping them to complete the cancellation process smoothly.

[0076] This flow provides a system that enables centralized management of contract information and streamlines cancellation procedures.

[0077] Example 1

[0078] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0079] It is extremely complicated for elderly people to manage a large number of contract information, and the cumbersome procedures become a problem, especially when contract cancellation procedures are required. In addition, in order for surviving family members to smoothly handle the contract information of the deceased, it is necessary to collect appropriate contract information and quickly cancel the contracts, but these procedures are not simple.

[0080] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0081] In this invention, the server includes means for converting input contract information into text data using optical character recognition technology or voice recognition technology, means for analyzing the text data using natural language processing technology to extract contract information and store it in a database, means for verifying the identity of surviving family members who log in from user terminal means using multi-factor authentication, and generation AI model means for providing contract information and details of cancellation procedures to surviving family members. This makes it possible to centrally manage contract information and provide quick and smooth cancellation procedures.

[0082] "Contract information" refers to data related to the contracts and contract contents owned by the user, including the name of the contract holder, contract contents, contract number, etc.

[0083] "User terminal" refers to a device used by a user to enter contract information. This includes smartphones, personal computers, etc.

[0084] "Server" means a computer system that receives, processes, and manages data sent from user terminals.

[0085] Optical character recognition (OCR) is a technology that extracts character information from image data and converts it into text data.

[0086] "ASR (audio recognition)" is a technology that analyzes voice data and converts it into text data.

[0087] "Natural language processing technology (NLP)" is a technology for analyzing text data and extracting specific information.

[0088] "Database" refers to a system for storing and managing extracted contract information.

[0089] "Multi-factor authentication (MFA)" is an authentication method that uses multiple means of verification to authenticate a user.

[0090] A "generative AI model" is a model that uses artificial intelligence to automatically perform specific tasks. In this case, its role is to generate cancellation procedure details based on contract information.

[0091] This invention is a system for centrally managing contract information and smoothly carrying out cancellation procedures. This system consists of a user terminal, a server, and a generative AI model, and is implemented using the following hardware and software.

[0092] Use of user terminals

[0093] First, the user enters contract information using a device such as a smartphone or PC. The user can either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input. This allows the user to easily provide the contract information to the system.

[0094] Server Processing

[0095] Next, the image data or audio data captured by the device is sent to a server. The server processes the received data. The image data is converted into text data using optical character recognition (OCR) technology, and the audio data is similarly converted into text using automatic speech recognition (ASR) technology. To achieve this, services such as Google® Cloud Vision API and IBM Watson® Speech to Text are used. The converted text data is then analyzed using natural language processing (NLP) technology (e.g., spaCy or Natural Language Toolkit (NLTK)) to extract the necessary contract information.

[0096] Information storage and analysis

[0097] The server stores the extracted contract information in a database, typically an SQL database (e.g., MySQL (registered trademark) or PostgreSQL), and provides the extracted contract information to a generative AI model for future cancellation procedures.

[0098] Access by family members

[0099] When a user dies, the surviving family members access the system. At this stage, the surviving family members log in to the system using the user's device and begin the process of verifying the deceased's contract information. The server then uses a multi-factor authentication (MFA) system (e.g., SMS authentication or email authentication) to verify the identity of the surviving family members. Only if the identity verification is successful will the deceased's contract information be provided.

[0100] Support for cancellation procedures

[0101] The server uses a generative AI model to generate cancellation procedure details based on the contract information. For example, by using OpenAI's (registered trademark) GPT series, the server provides optimal cancellation procedure details to the bereaved family.

[0102] Specific examples

[0103] Entering the contract

[0104] User: An elderly person launches the smartphone app, takes a photo of their electricity contract, and uploads it.

[0105] Terminal: Sends the contract image to the server.

[0106] Server: The server receives the image and converts it into text data using OCR technology (Google Cloud Vision API), then analyzes the contract information using natural language processing (NLP) technology and stores it in an SQL database.

[0107] Acquisition of contract information by surviving family members

[0108] User (surviving family member): The deceased person's surviving family member logs into the system and checks the cancellation information for the electricity contract.

[0109] Server: Performs identity verification, and if successful, extracts contract information and queries the generative AI model (OpenAI GPT).

[0110] Generative AI model: Generates optimal cancellation details and provides them to the bereaved family, who then follow the cancellation process.

[0111] Prompt Sentence Examples

[0112] Please tell me the procedure for canceling my electricity contract.

[0113] How do I cancel a deceased person's mobile phone contract?

[0114] "Please tell me the documents required to cancel the water contract."

[0115] In this way, the purpose of this system is to efficiently manage the contract information of elderly people and to support the surviving family members in completing the procedures quickly and smoothly.

[0116] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0117] Step 1:

[0118] Enter contract information

[0119] Specific operation: The user enters contract information from a smartphone or PC. The user uses a dedicated application to take a photo of a document such as a contract and upload it to the application. Alternatively, the user can dictate the contract details using the voice input function.

[0120] Input: Image data or audio recording of a contract uploaded by the user.

[0121] Output: Image or audio data sent by the device to the server.

[0122] Step 2:

[0123] Data reception and initial processing by the server

[0124] Specific operation: The server receives image data or audio data sent from the user terminal. The received data is temporarily stored in storage.

[0125] Input: Image or audio data sent from the device.

[0126] Output: Data saved in temporary storage.

[0127] Step 3:

[0128] OCR and ASR text conversion

[0129] Specific operation: The server applies optical character recognition (OCR) technology to the received image data and converts it into text data. It also applies automatic speech recognition (ASR) technology to the audio data and converts it into text in the same way. It uses Google Cloud Vision API, IBM Watson Speech to Text, etc.

[0130] Input: Image or audio data stored in temporary storage.

[0131] Output: Contract information converted to text data.

[0132] Step 4:

[0133] Analysis of text data and extraction of contract information

[0134] Specific operation: The server analyzes the text data using natural language processing (NLP) technology to extract necessary contract information such as the contract holder's name, contract details, contract number, etc. This analysis is performed using tools such as spaCy or the Natural Language Toolkit (NLTK).

[0135] Input: Text data.

[0136] Output: The extracted contract information.

[0137] Step 5:

[0138] Saving to a database

[0139] Specific operation: The server stores the extracted contract information in an SQL database (e.g., MySQL or PostgreSQL).

[0140] Input: The extracted contract information.

[0141] Output: Contract information stored in a database.

[0142] Step 6:

[0143] System access and identity verification by family members

[0144] Specific operation: The user (survivor) accesses the system and logs in using the user's device. The server performs multi-factor authentication (MFA) to verify the identity of the surviving family member, and performs SMS authentication and email authentication.

[0145] Input: Family member's credentials.

[0146] Output: Access granted if identity verification is successful.

[0147] Step 7:

[0148] Generative AI model generates cancellation procedures

[0149] How it works: The server sends the contract information to a generative AI model (e.g., OpenAI's GPT series) and inputs a prompt to generate the appropriate cancellation procedure details. The generative AI model then generates the cancellation procedure accordingly.

[0150] Input: Contract information and prompt text stored in the database.

[0151] Output: Detailed steps of the cancellation procedure generated.

[0152] Step 8:

[0153] Provision of cancellation procedure information

[0154] Specific operation: The server analyzes the cancellation procedure information received from the generated AI model and displays the details on the user's device.

[0155] Input: Cancellation process information from the generative AI model.

[0156] Output: Detailed cancellation information displayed on the user's device.

[0157] (Application example 1)

[0158] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0159] There is a lack of means for surviving family members to quickly and accurately grasp the contract information of the deceased and smoothly complete the cancellation procedures. As a result, the procedures take time and often cause stress for surviving family members. In addition, there are limited means for elderly people to properly manage contract information, which runs the risk of the information being lost. It is important to resolve these issues.

[0160] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0161] In this invention, the server includes a user terminal means for inputting contract information, a server means for converting the input contract information into text data using character recognition or voice recognition technology, a means for analyzing the text data to extract the contract information and saving it in a database, a means for verifying the identity of surviving family members registered from the user terminal means when they access the server, a generation AI model means for providing the surviving family members with contract information and a cancellation procedure method, and a means for generating optimal cancellation procedure details based on the contract information saved in the server means and providing them with the necessary documents, contact information, and cancellation procedure. This enables surviving family members to quickly and accurately check the contract information of the deceased person and easily complete the cancellation procedure.

[0162] "Contract information" is information related to the contractor, including the contractor's name, contract details, contract number, and the like.

[0163] The "user terminal means" is a device for a user to input contract information, and refers to an electronic terminal such as a smartphone or a personal computer.

[0164] "Character recognition technology" refers to technology that extracts text data from captured image data, and includes optical character recognition (OCR) technology.

[0165] "Speech recognition technology" refers to technology that extracts text data from voice data, and includes automatic speech recognition (ASR) technology.

[0166] "Server means" refers to the equipment and software used to process, analyze and store input data.

[0167] "Text data" refers to data obtained by converting image data and audio data into character information.

[0168] "Database" refers to the system and storage for systematically storing and managing analyzed contract information.

[0169] "Personal identification means" refers to means for verifying the identity of the user and their family members, and includes authentication information and biometric authentication technology.

[0170] "Generative AI model means" refers to an artificial intelligence model and software that executes it to generate optimal cancellation procedure details based on stored contract information.

[0171] "Contact Information" refers to the contact information required to complete the cancellation procedure, including, for example, contact information for the company's customer support or related departments.

[0172] "Cancellation Procedure" refers to the steps and procedures required to cancel a contract.

[0173] As an embodiment of this invention, we propose a system that centrally manages contract information and smoothly processes cancellation procedures. This system consists of a user terminal, a server, and a generative AI model.

[0174] 1. Enter contract information

[0175] First, the user enters the contract information. Using a device such as a smartphone or PC, the user can either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input. This process uses optical character recognition (OCR) and automatic speech recognition (ASR).

[0176] 2. Data processing by the server

[0177] Next, the image data or audio data captured by the user device is sent to the server. The server processes the received data, converting the image data into text data using optical character recognition (OCR) technology and the audio data into text using automatic speech recognition (ASR) technology. The server then analyzes the text data, extracts necessary information such as the contract holder's name, contract details, and contract number, and stores it in a database.

[0178] 3. Analysis and storage of contract information

[0179] The server analyzes the converted text data and extracts the necessary contract information, which is then stored in a database and provided to the generative AI model for future cancellation procedures.

[0180] 4. User Access and Identity Verification

[0181] When a user dies, the surviving family members access the system. At this stage, the surviving family members log in using the user terminal and begin the process of verifying the contract information of the deceased. The server performs identity verification to verify the identity of the surviving family members, and provides the information only if this is successful.

[0182] 5. Provision of cancellation procedure information

[0183] The server generates optimal cancellation procedure details based on the contract information stored in the generative AI model, and provides the necessary documents, contact information, and specific cancellation procedures to the bereaved family members, allowing them to smoothly complete the cancellation procedures for various contracts according to the information provided.

[0184] Specific examples

[0185] Example 1: Entering a contract

[0186] User: An elderly person launches the smartphone app and takes a photo of their electricity contract and uploads it.

[0187] Terminal: Sends the contract image to the server.

[0188] Server: The server receives the image and converts it into text data using OCR technology. It then analyzes the contract information and stores it in a database.

[0189] Example 2: Obtaining contract information by a surviving family member

[0190] User (surviving family member): The deceased person's surviving family member logs into the system and checks the cancellation information for the electricity contract.

[0191] Server: Performs identity verification, and if successful, extracts contract information and queries the generation AI model.

[0192] Generative AI model: Generates optimal cancellation details and provides them to the bereaved family, who then follow the cancellation process.

[0193] Prompt Sentence Examples

[0194] Example prompt 1: "Contract holder: Taro Yamada, Contract details: Mobile phone service contract. Please tell me the cancellation procedure."

[0195] Example prompt 2: "Contract holder: Ichiro Tanaka, Contract details: Electricity contract. Please tell me the cancellation procedure."

[0196] In this way, the system efficiently manages the contract information of elderly people and helps surviving family members to complete the procedures quickly and smoothly.

[0197] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0198] Step 1:

[0199] Users enter contract information using devices such as smartphones or PCs. Specifically, they either take a photo of a document such as a contract and upload it, or explain the contract details using voice input. The input data in this case is image data or voice data.

[0200] Step 2:

[0201] The device sends the captured image data or audio data to the server, which processes the data.

[0202] Step 3:

[0203] The server converts the received image data into text data using optical character recognition (OCR) technology. For audio data, it converts it into text using automatic speech recognition (ASR). This is the process of converting input data (image data or audio data) into text data.

[0204] Step 4:

[0205] The server analyzes the converted text data and extracts necessary information such as the contract holder's name, contract details, contract number, etc. The contract information obtained through this analysis process is stored in a database. This output data is the contract information text data.

[0206] Step 5:

[0207] When a user dies, the surviving family members access the system. They log in using the user terminal, and the server verifies the identity of the surviving family members. Information is provided only if the conditions for successful identity verification are met. The input data is the login credentials, and the output data is access authentication.

[0208] Step 6:

[0209] The server queries the generative AI model to generate cancellation procedure details based on the contract information stored in the database. The generative AI model generates optimal cancellation procedure details, required documents, contact information, and cancellation procedures based on the provided contract information. The output data obtained from this process is the cancellation procedure details.

[0210] Step 7:

[0211] The server provides the bereaved family with details of the cancellation procedures obtained from the generative AI model. Based on the information provided, the bereaved family can smoothly proceed with the cancellation procedures for various contracts. The output data is the specific cancellation procedures and related information.

[0212] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0213] As an embodiment of the invention, the system below recognizes the user's emotions during the process from entering contract information to the cancellation procedure and adjusts the process accordingly. This system is composed of a user terminal, a server, a generative AI model, and an emotion engine.

[0214] 1. Enter contract information

[0215] First, the user (elderly person) uses a device such as a smartphone or PC to enter contract information. They either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input. The user device then sends the input data to an emotion engine, which analyzes the user's emotional state.

[0216] 2. Emotion analysis using an emotion engine

[0217] The emotion engine analyzes the user's emotional state based on the voice and image data sent by the device. For example, it analyzes the speaking style and tone of voice from the voice data to recognize feelings of relief or anxiety. It also analyzes facial expressions from image data to recognize feelings of happiness or sadness. This emotional information is sent to the server.

[0218] 3. Data processing by the server

[0219] The server processes the received data. Specifically, it converts image data into text using OCR technology and voice data into text using ASR technology. It also takes into account the emotional information obtained from the emotion engine and provides appropriate feedback to the user.

[0220] 4. Analysis and storage of contract information

[0221] The server analyzes the converted text data and extracts the necessary contract information (e.g., contract holder name, contract details, contract number, etc.). The extracted contract information is stored in a database. It also provides an appropriate interface and explanation based on the user's emotional state based on the results of the emotion analysis.

[0222] 5. The server provides the contract information to the generated AI model

[0223] The server provides contract information to the generative AI model and updates the model to generate recommended procedures for contract cancellation. Feedback to users based on emotional information is also optimized.

[0224] 6. Access and identity verification by family members

[0225] When a user passes away, the surviving family members access the system. At this stage, they log in to the system using a user terminal and begin the process of checking the contract information of the deceased. At the same time, an emotion engine runs to analyze the emotional state of the surviving family members and provide appropriate feedback.

[0226] 7. Providing cancellation procedure information to surviving family members

[0227] The server retrieves details of the cancellation procedure from the generative AI model and provides them to the bereaved family after taking into account emotional information, allowing the bereaved family to complete the cancellation procedure smoothly and without stress.

[0228] Specific examples

[0229] Example 1: Entering a contract

[0230] User: An elderly person launches the smartphone app and takes a photo of their electricity contract and uploads it. The emotion engine analyzes the user's facial expression and determines that they are relaxed.

[0231] Terminal: Sends the contract image and emotion information to the server.

[0232] Server: Converts the image into text data using OCR technology, analyzes and saves the contract information, and provides feedback in the usual way since the subject is in a relaxed state.

[0233] Example 2: Obtaining contract information by a surviving family member

[0234] User (surviving family member): The family member of the deceased logs into the system and checks the cancellation information for the electricity contract. The emotion engine analyzes the family member's emotional state and determines that they are experiencing high levels of stress.

[0235] Server: After verifying the identity of the bereaved, the server extracts the contract information and queries the generative AI model. The generative AI model provides details of the cancellation procedure in an easy-to-understand manner, taking into consideration the emotional state of the bereaved.

[0236] In this way, the system combines an emotion engine to optimize the process from entering contract information to the cancellation procedure in a way that responds to the emotional state of the user and their family members, ensuring smooth operation.

[0237] The processing flow will be explained below.

[0238] Step 1:

[0239] The user enters the contract information. Specifically, the user takes a picture of the contract using a device such as a smartphone or PC, or explains the contract details using voice input. The user's device then uploads this input data to the application.

[0240] Step 2:

[0241] The device sends input data to the emotion engine, which receives image and audio data and analyzes the user's emotional state. For example, it recognizes feelings of relief, anxiety, happiness, sadness, etc. from the user's facial expressions and tone of voice.

[0242] Step 3:

[0243] The emotion engine sends the analysis results to the server. Specifically, if the user is relaxed while inputting, emotion data indicating that state is sent to the server.

[0244] Step 4:

[0245] The device sends the input data to the server. Specifically, image data or audio data of the photographed contract is sent to the server via the Internet. The data may be encrypted during transmission.

[0246] Step 5:

[0247] The server processes the received data. Specifically, image data is converted to text using OCR technology, and voice data is converted to text using ASR (audio-speech recognition) technology. Through this process, the contents of the contract and the contract details explained in the voice are extracted as text data.

[0248] Step 6:

[0249] The server analyzes the text data. Specifically, it extracts the necessary contract information (e.g., contract holder name, contract details, contract number, etc.) from the text data and organizes it as contract information. The analysis results are saved in a database.

[0250] Step 7:

[0251] The server provides feedback based on the emotional information: if the user is relaxed, it provides normal feedback, and if the user is anxious, it provides a reassuring message or additional support information.

[0252] Step 8:

[0253] The server provides the contract information to the generative AI model. Specifically, the extracted and organized contract information is passed to the generative AI model, and the model is updated based on that information to generate recommended cancellation procedures.

[0254] Step 9:

[0255] The user (surviving family member) accesses the system. Specifically, the family member logs into the system using a user terminal and requests to check the contract information of the deceased. The emotion engine analyzes the emotional state of the family member and recognizes high levels of stress and sadness.

[0256] Step 10:

[0257] The server performs identity verification. Specifically, it uses security questions, QR codes, passwords, etc. to verify the identity of the family member. Only if identity verification is successful is access to the contract information granted.

[0258] Step 11:

[0259] The server extracts contract information. Specifically, it searches the database for contract information of the deceased person and extracts the relevant contract information. It then queries the generative AI model to generate cancellation procedure information.

[0260] Step 12:

[0261] The server provides cancellation procedure information to the bereaved family. Specifically, it provides detailed cancellation procedure instructions, necessary documents, and contact information provided by the generative AI model. The server takes into consideration the emotional state of the bereaved family and provides the information in an easy-to-understand and stress-reducing format.

[0262] This flow enables centralized management of contract information, streamlines cancellation procedures, and provides support tailored to the emotional state of the user and their family members.

[0263] Example 2

[0264] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0265] Many users, including the elderly, find the process from entering contract information to the cancellation procedure complicated and stressful. The process of entering contract details and then completing the cancellation procedure can be particularly stressful for elderly users. Furthermore, when a user passes away, there are few ways for their surviving family members to smoothly cancel the contract. To address these issues, a system is needed that recognizes the user's emotional state and optimizes the process based on that information.

[0266] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0267] In this invention, the server includes a user terminal means for inputting contract information, an emotion engine means for analyzing emotion data and recognizing the user's emotional state, a server means for converting the input contract information into text data using character recognition or voice recognition technology, and a means for providing appropriate feedback to the user based on the emotion analysis results. This provides an interface and explanations that correspond to the user's emotional state, reducing stress when entering contract information and completing cancellation procedures and enabling smoother procedures. Furthermore, by utilizing emotion analysis, appropriate support can be provided to surviving family members when they check the deceased's contract information and complete cancellation procedures.

[0268] "Contract information" refers to information in which a user provides details regarding the use of a service or product, including the name of the contractor, contract details, contract number, etc.

[0269] "User terminal" refers to an electronic device used by a user to input contract information, and includes smartphones, personal computers, tablets, etc.

[0270] An "emotion engine" refers to software or a system that analyzes audio and image data to recognize a user's emotional state.

[0271] "Character recognition technology" refers to the technology of identifying characters in an image and converting them into text data, and OCR (Optical Character Recognition) is one such technology.

[0272] "Speech recognition technology" refers to the technology that analyzes voice data and converts it into text data, and ASR (Automatic Speech Recognition) is one such technology.

[0273] "Server" refers to a computer system that processes, stores, and operates generative AI models.

[0274] A "generative AI model" refers to an artificial intelligence model that generates recommended procedures or solutions based on given data.

[0275] "Feedback" refers to information and guidance provided to users based on the results of emotion analysis.

[0276] "Identity verification" refers to the process of verifying that a system user is the person registered, and is carried out to protect access rights.

[0277] A "database" refers to a collection of data that organizes and stores contract information and user information so that it can be efficiently accessed as needed.

[0278] This invention provides a system that recognizes the user's emotions throughout the process, from entering contract information to completing cancellation procedures, and adjusts the process accordingly. This system consists of a user terminal, a server, a generative AI model, and an emotion engine. The details and operation of each component are explained below.

[0279] Enter contract information

[0280] Users enter contract information using devices such as smartphones or PCs. They can either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input. The user device sends the input data (image data or voice data) to the emotion engine.

[0281] Emotion analysis using an emotion engine

[0282] The device receives voice and image data sent by the user, and the emotion engine analyzes them. In the case of voice data, the emotion engine recognizes feelings of relief or anxiety from the speaking style and tone of voice. In the case of image data, the emotion engine recognizes feelings of happiness or sadness from facial expressions. This emotion information is sent to the server.

[0283] Data processing by the server

[0284] The server processes the data (voice data, image data, and emotional information) received from the emotion engine. Specifically, it converts the voice data into text data using automatic speech recognition (ASR) technology, and converts the image data into text data using optical character recognition (OCR) technology. It also provides appropriate feedback to the user based on the emotional information.

[0285] Analysis and storage of contract information

[0286] The server analyzes the converted text data and extracts necessary contract information such as the contract holder's name, contract details, and contract number. The extracted contract information is stored in a database. Based on the results of the sentiment analysis, the server also provides the user with an appropriate interface and explanation.

[0287] The server provides contract information to the generated AI model

[0288] The server provides the contract information to the generative AI model. The generative AI model updates the model to generate recommended procedures for contract cancellation, and also optimizes feedback to the user based on the generated procedures. For example, it may provide the user with specific procedures such as "Please follow the steps below to cancel the contract."

[0289] Family access and identity verification

[0290] When the deceased passes away, the user (survivor) accesses the system and logs in using a user terminal. At this stage, the system analyzes the emotional state of the bereaved and provides appropriate feedback as needed.

[0291] Providing cancellation procedure information to surviving family members

[0292] The server retrieves details of the cancellation procedure from the generative AI model and provides information while taking into consideration the emotional state of the bereaved, allowing the cancellation procedure to proceed smoothly and without stress for the bereaved.

[0293] Specific examples

[0294] Contract entry: The user (elderly person) launches the smartphone app and takes a photo of their electricity contract and uploads it. The emotion engine analyzes the user's facial expression and determines that they are in a relaxed state. The device then sends the contract image and emotion information to the server, which then converts the image into text data using OCR technology and analyzes and saves the contract information. Because the user is in a relaxed state, feedback is provided using the normal procedure.

[0295] Obtaining contract information by surviving family members: The user (surviving family member) logs in to the system and checks the cancellation information for the deceased person's electricity contract. The emotion engine analyzes the emotional state of the surviving family member and determines that they are experiencing high levels of stress. The server performs identity verification, and if successful, extracts the contract information and queries the generation AI model, which then provides details of the cancellation procedure in an easy-to-understand manner, taking into consideration the emotional state of the surviving family member.

[0296] Example prompts for generative AI models

[0297] "What is the best procedure for a user to cancel their electricity contract?"

[0298] "If the emotion analysis results indicate high stress, please suggest ways to provide appropriate feedback to the bereaved."

[0299] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0300] Step 1:

[0301] The user enters contract information using a smartphone or PC. Specifically, they take a photo of the contract and upload it, or explain the contract details using voice input. The input is image data or voice data of the contract, and the output is the initial stage in which this data is processed on the device.

[0302] Step 2:

[0303] The terminal sends image data or voice data of the contract information to the emotion engine. In the case of voice data, the emotion engine analyzes the speaking style and tone of voice, and in the case of image data, it analyzes facial expressions. The input is the image data or voice data of the contract information, and the output is emotional state information (e.g., relaxed, anxious, happy, sad) as the analysis result.

[0304] Step 3:

[0305] The emotion engine analyzes the user's emotional state and sends the results to the server. For example, it analyzes the tone of voice from audio data to recognize feelings of relief or anxiety, or recognizes facial expressions from image data to identify feelings of happiness or sadness. The input is audio or image data, and the output is the analysis result of the emotional state.

[0306] Step 4:

[0307] The server processes contract information based on the received emotional information. Specifically, it converts voice data into text using automatic speech recognition (ASR) technology, and converts image data into text using optical character recognition (OCR) technology. The input is voice data and image data, and the output is text data.

[0308] Step 5:

[0309] The server analyzes the converted text data and extracts necessary contract information such as the contract holder's name, contract details, and contract number. The extracted contract information is stored in a database. For example, OCR technology can be used to extract the contract holder's name and contract details from image data of a contract. The input is text data, and the output is the extracted contract information.

[0310] Step 6:

[0311] The server provides appropriate feedback to the user based on the emotion analysis results. For example, if the user feels anxious, it provides a detailed and friendly explanation, and if the user feels relaxed, it provides standard procedures. The input is emotion information and contract information, and the output is the feedback content.

[0312] Step 7:

[0313] The server provides the contract information to the generative AI model, which then updates the model to generate recommended procedures for contract cancellation. The input is the contract information, and the output is the generated recommended procedures for cancellation.

[0314] Step 8:

[0315] When the server starts the process for the user (survivor) to confirm the contract information of the deceased, it analyzes the emotional state of the survivor and provides appropriate feedback. At this stage, the server receives input information from the survivor and the emotion engine provides the analysis results. The input is the survivor's login information and emotional information, and the output is appropriate feedback.

[0316] Step 9:

[0317] The server obtains details of the cancellation procedure from the generative AI model and provides information while taking into consideration the emotional state of the bereaved. Specifically, this includes gentle language and detailed explanations based on emotion analysis. The input is the details of the cancellation procedure, which are the output of the generative AI model, and the emotional information of the bereaved, and the output is optimized feedback.

[0318] (Application example 2)

[0319] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0320] Conventional systems for entering contract information and for canceling contracts are unable to provide optimal feedback based on the user's emotional state, making it difficult to alleviate the psychological burden, especially for elderly people and bereaved families. Electronic payment services also face the issue of not responding appropriately to users' emotions, resulting in a decline in the quality of the user experience.

[0321] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0322] In this invention, the server includes emotion engine means for analyzing the user's emotions, user terminal means for inputting contract information, and server means for converting the input contract information into text data using character recognition or voice recognition technology. This makes it possible to provide optimal feedback based on the user's emotions and reduce the psychological burden, particularly on elderly people and bereaved families. Furthermore, this invention can improve the quality of the user experience by providing optimal support based on emotion analysis in electronic payment services.

[0323] The "emotion engine means" is a system for analyzing the user's voice data and image data and recognizing the user's emotional state.

[0324] "User terminal means" refers to a device used by a user to input contract information, and includes a smartphone, a personal computer, etc.

[0325] "Character recognition technology" is a technology that extracts character information from image data and converts it into text data, and OCR (optical character recognition) is a representative example.

[0326] "Speech recognition technology" is a technology that analyzes voice data and converts it into text data, and a representative technology is ASR (automatic speech recognition).

[0327] "Server means" means a computer system that processes data sent from a user terminal and stores and provides necessary information.

[0328] The "generative AI model means" is an artificial intelligence model for generating optimal cancellation procedure steps and feedback based on analyzed contract information.

[0329] The "database" is a system for storing processed contract information and emotion analysis results, and providing information as needed.

[0330] "Electronic payment service" means a service for sending and receiving funds over the Internet.

[0331] "Feedback" refers to providing real-time responses and support information based on the user's emotional state and behavior.

[0332] "Transaction information" refers to specific data related to a transaction, including information such as the amount, currency, and transaction details.

[0333] The system for realizing this invention comprises an emotion engine means, a user terminal means, character recognition technology, voice recognition technology, a server means, a generation AI model means, and a processing means for electronic payment services.

[0334] The user terminal means is used to input contract information, voice data, and image data. For example, a user can use a smartphone or PC to take a picture of the contract and explain the contract contents by voice. This data is transmitted to the server means in real time.

[0335] The server means converts the received image data into text data using OCR (Optical Character Recognition) technology, and converts the voice data into text using ASR (Automatic Speech Recognition) technology. The server also uses an emotion engine means to analyze emotions from the user's voice and facial expressions, and recognizes the user's emotional state.

[0336] The emotion engine means is a system for analyzing the user's emotions based on voice data and image data, and grasps the user's emotional state by analyzing the tone of voice, facial expressions, etc. For example, it can recognize emotions such as joy, sadness, and anxiety.

[0337] In particular, in the case of electronic payment services, appropriate feedback and support can be provided based on the user's emotional state based on the results of emotion analysis. For example, if a user feels anxious during the payment process, additional confirmation messages or support information can be displayed.

[0338] The generative AI model generates optimal steps for the cancellation and settlement procedures based on the analyzed contract information and emotional information, allowing the procedures to proceed smoothly and without stress for the user and their family members.

[0339] As a specific example, consider a case where a user takes a photo of an electricity contract through a smartphone app and uploads it to the app. At this time, the emotion engine means analyzes the user's facial expression and, if it determines that the user is relaxed, provides normal feedback. On the other hand, if it determines that the surviving family members are feeling high stress when checking the contract information of the deceased person, it takes the emotional information into consideration and provides the surviving family members with easy-to-understand information on cancellation procedures.

[0340] The following sentences are used as example prompts:

[0341] "Analyze the emotions users have when they hit the checkout button and provide additional feedback if they are feeling anxious or uncertain."

[0342] This allows for appropriate responses depending on the user's emotional state, improving the overall user experience.

[0343] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0344] Step 1:

[0345] The user terminal means inputs contract information. The user then takes a photo of the contract using a smartphone or PC and explains the contract contents in voice. This data is sent to the server in real time. The inputs are image data and voice data, and the server receives these data as output. Specific operations use the smartphone's camera function and voice recording function.

[0346] Step 2:

[0347] The server converts the received image data into text data using OCR technology, and simultaneously converts the voice data into text using ASR technology. The inputs are image data and voice data, and the output is text data. This data conversion uses OpenCV and Google's voice recognition API.

[0348] Step 3:

[0349] The server analyzes the converted text data, extracts contract information (e.g., contract holder name, contract details, contract number, etc.), and stores it in a database. The input is text data, and the output is contract information stored in the database. Specifically, the text data is analyzed using natural language processing technology.

[0350] Step 4:

[0351] The server uses an emotion engine to analyze the voice and image data sent by the user and recognize the user's emotional state. The input is voice and image data, and the output is emotional information. Specific operations include DeepFace and emotion recognition APIs.

[0352] Step 5:

[0353] The server provides feedback according to the user's emotional state based on the emotional information acquired from the emotion engine means. The emotional information is input, and a feedback message is sent to the user terminal as output. A specific operation is to display a customized message.

[0354] Step 6:

[0355] When a registered family member accesses the system from a user terminal, the server verifies the identity of the person. The input is authentication information, and the output is the authentication result. Specific security methods such as two-factor authentication are used.

[0356] Step 7:

[0357] The generative AI model generates detailed steps for the cancellation procedure based on emotional information and contract information, and provides them to the bereaved family. The inputs are emotional information and contract information, and the cancellation procedure is generated as the output. The specific operation uses natural language processing technology and machine learning models.

[0358] Step 8:

[0359] The server processes electronic payment services, confirms payment based on the results of emotion analysis, and displays additional support messages according to the user's emotional state. The inputs are emotion information and payment information, and the output is the payment result and feedback message. Specifically, communication with the payment gateway is performed using an HTTP request.

[0360] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0361] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0362] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0363] [Second embodiment]

[0364] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0365] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0366] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0367] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0368] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0369] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0370] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0371] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0372] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0373] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0374] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0375] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0376] As an embodiment of the invention, the system that centrally manages contract information and smoothly processes cancellations is shown below. This system consists of a user terminal, a server, and a generative AI model.

[0377] 1. Enter contract information

[0378] First, the user (elderly person) enters the contract information. To do this, the user uses a device such as a smartphone or PC. They either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input.

[0379] 2. Data processing by the server

[0380] The device then sends the captured image or audio data to the server, which processes the received data. The image data is converted into text using optical character recognition (OCR) technology, and the audio data is similarly converted into text using automatic speech recognition (ASR) technology.

[0381] 3. Analysis and storage of contract information

[0382] The server analyzes the converted text data and extracts the necessary contract information (contractor name, contract details, contract number, etc.). The extracted contract information is stored in a database. The server also provides this contract information to a generative AI model for future cancellation procedures.

[0383] 4. User Access and Identity Verification

[0384] When a user dies, the surviving family members access the system. At this stage, the surviving family members (users) log in using a user terminal and begin the process of verifying the contract information of the deceased. The server performs identity verification to verify the identity of the surviving family members, and provides information only if this is successful.

[0385] 5. Provision of cancellation procedure information

[0386] The server uses the contract information stored in the generative AI model to provide details of the necessary cancellation procedures, including the necessary documents, contact information, and specific cancellation steps. Survivors can use this information to smoothly complete the cancellation procedures for various contracts.

[0387] Specific examples

[0388] Example 1: Entering a contract

[0389] User: An elderly person launches the smartphone app and takes a photo of their electricity contract and uploads it.

[0390] Terminal: Sends the contract image to the server.

[0391] Server: The server receives the image and converts it into text data using OCR technology. It then analyzes the contract information and stores it in a database.

[0392] Example 2: Obtaining contract information by a surviving family member

[0393] User (surviving family member): The deceased person's surviving family member logs into the system and checks the cancellation information for the electricity contract.

[0394] Server: Performs identity verification, and if successful, extracts contract information and queries the generation AI model.

[0395] Generative AI model: Generates optimal cancellation details and provides them to the bereaved family, who then follow the cancellation process.

[0396] In this way, the system efficiently manages the contract information of elderly people and helps surviving family members to complete the procedures quickly and smoothly.

[0397] The processing flow will be explained below.

[0398] Step 1:

[0399] The user enters the contract information. Specifically, the user takes a picture of the contract using a device such as a smartphone or PC, or explains the contract details using voice input. The user's device then uploads this input data to the application.

[0400] Step 2:

[0401] The device sends the input data to the server. Specifically, image data or audio data of the photographed contract is sent to the server via the Internet. The data may be encrypted during transmission.

[0402] Step 3:

[0403] The server processes the received data. Specifically, image data is converted to text using OCR technology, and voice data is converted to text using automatic speech recognition (ASR). Through this process, the contents of the contract and the contract contents explained in the voice are extracted as text data.

[0404] Step 4:

[0405] The server analyzes the text data. Specifically, it extracts the necessary contract information (e.g., contract holder name, contract details, contract number, etc.) from the text data and organizes it as contract information. The analysis results are saved in a database.

[0406] Step 5:

[0407] The server provides the contract information to the generative AI model. Specifically, the extracted and organized contract information is passed to the generative AI model, and the model is updated based on that information to generate recommended cancellation procedures.

[0408] Step 6:

[0409] The user (surviving family member) accesses the system. Specifically, the surviving family member logs into the system using a user terminal and inputs that they would like to check the contract information of the deceased person.

[0410] Step 7:

[0411] The server performs identity verification. Specifically, it uses security questions, QR codes, passwords, etc. to verify the identity of the family member. Only if identity verification is successful is access to the contract information granted.

[0412] Step 8:

[0413] The server extracts contract information. Specifically, it searches the database for contract information of the deceased person and extracts the relevant contract information. It then queries the generative AI model to generate cancellation procedure information.

[0414] Step 9:

[0415] The server provides cancellation procedure information, specifically detailed cancellation procedures provided by the generative AI model, as well as necessary documents and contact information, to the bereaved family, helping them to complete the cancellation process smoothly.

[0416] This flow provides a system that enables centralized management of contract information and streamlines cancellation procedures.

[0417] Example 1

[0418] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0419] It is extremely complicated for elderly people to manage a large number of contract information, and the cumbersome procedures become a problem, especially when contract cancellation procedures are required. In addition, in order for surviving family members to smoothly handle the contract information of the deceased, it is necessary to collect appropriate contract information and quickly cancel the contracts, but these procedures are not simple.

[0420] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0421] In this invention, the server includes means for converting input contract information into text data using optical character recognition technology or voice recognition technology, means for analyzing the text data using natural language processing technology to extract contract information and store it in a database, means for verifying the identity of surviving family members who log in from user terminal means using multi-factor authentication, and generation AI model means for providing contract information and details of cancellation procedures to surviving family members. This makes it possible to centrally manage contract information and provide quick and smooth cancellation procedures.

[0422] "Contract information" refers to data related to the contracts and contract contents owned by the user, including the name of the contract holder, contract contents, contract number, etc.

[0423] "User terminal" refers to a device used by a user to enter contract information. This includes smartphones, personal computers, etc.

[0424] "Server" means a computer system that receives, processes, and manages data sent from user terminals.

[0425] Optical character recognition (OCR) is a technology that extracts character information from image data and converts it into text data.

[0426] "ASR (audio recognition)" is a technology that analyzes voice data and converts it into text data.

[0427] "Natural language processing technology (NLP)" is a technology for analyzing text data and extracting specific information.

[0428] "Database" refers to a system for storing and managing extracted contract information.

[0429] "Multi-factor authentication (MFA)" is an authentication method that uses multiple means of verification to authenticate a user.

[0430] A "generative AI model" is a model that uses artificial intelligence to automatically perform specific tasks. In this case, its role is to generate cancellation procedure details based on contract information.

[0431] This invention is a system for centrally managing contract information and smoothly carrying out cancellation procedures. This system consists of a user terminal, a server, and a generative AI model, and is implemented using the following hardware and software.

[0432] Use of user terminals

[0433] First, the user enters contract information using a device such as a smartphone or PC. The user can either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input. This allows the user to easily provide the contract information to the system.

[0434] Server Processing

[0435] The device then sends the captured image or audio data to a server. The server processes the received data. The image data is converted into text using optical character recognition (OCR) technology, and the audio data is similarly converted into text using automatic speech recognition (ASR) technology. To achieve this, services such as Google Cloud Vision API and IBM Watson Speech to Text are used. The converted text data is then analyzed using natural language processing (NLP) technology (such as spaCy or Natural Language Toolkit (NLTK)) to extract the necessary contract information.

[0436] Information storage and analysis

[0437] The server stores the extracted contract information in a database, typically an SQL database (e.g., MySQL or PostgreSQL), and provides the extracted contract information to a generative AI model for future cancellation procedures.

[0438] Access by family members

[0439] When a user dies, the surviving family members access the system. At this stage, the surviving family members log in to the system using the user's device and begin the process of verifying the deceased's contract information. The server then uses a multi-factor authentication (MFA) system (e.g., SMS authentication or email authentication) to verify the identity of the surviving family members. Only if the identity verification is successful will the deceased's contract information be provided.

[0440] Support for cancellation procedures

[0441] The server uses a generative AI model to generate cancellation procedure details based on the contract information. For example, by using OpenAI's GPT series, the server can provide optimal cancellation procedure details to the bereaved family.

[0442] Specific examples

[0443] Entering the contract

[0444] User: An elderly person launches the smartphone app, takes a photo of their electricity contract, and uploads it.

[0445] Terminal: Sends the contract image to the server.

[0446] Server: The server receives the image and converts it into text data using OCR technology (Google Cloud Vision API), then analyzes the contract information using natural language processing (NLP) technology and stores it in an SQL database.

[0447] Acquisition of contract information by surviving family members

[0448] User (surviving family member): The deceased person's surviving family member logs into the system and checks the cancellation information for the electricity contract.

[0449] Server: Performs identity verification, and if successful, extracts contract information and queries the generative AI model (OpenAI GPT).

[0450] Generative AI model: Generates optimal cancellation details and provides them to the bereaved family, who then follow the cancellation process.

[0451] Prompt Sentence Examples

[0452] Please tell me the procedure for canceling my electricity contract.

[0453] How do I cancel a deceased person's mobile phone contract?

[0454] "Please tell me the documents required to cancel the water contract."

[0455] In this way, the purpose of this system is to efficiently manage the contract information of elderly people and to support the surviving family members in completing the procedures quickly and smoothly.

[0456] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0457] Step 1:

[0458] Enter contract information

[0459] Specific operation: The user enters contract information from a smartphone or PC. The user uses a dedicated application to take a photo of a document such as a contract and upload it to the application. Alternatively, the user can dictate the contract details using the voice input function.

[0460] Input: Image data or audio recording of a contract uploaded by the user.

[0461] Output: Image or audio data sent by the device to the server.

[0462] Step 2:

[0463] Data reception and initial processing by the server

[0464] Specific operation: The server receives image data or audio data sent from the user terminal. The received data is temporarily stored in storage.

[0465] Input: Image or audio data sent from the device.

[0466] Output: Data saved in temporary storage.

[0467] Step 3:

[0468] OCR and ASR text conversion

[0469] Specific operation: The server applies optical character recognition (OCR) technology to the received image data and converts it into text data. It also applies automatic speech recognition (ASR) technology to the audio data and converts it into text in the same way. It uses Google Cloud Vision API, IBM Watson Speech to Text, etc.

[0470] Input: Image or audio data stored in temporary storage.

[0471] Output: Contract information converted to text data.

[0472] Step 4:

[0473] Analysis of text data and extraction of contract information

[0474] Specific operation: The server analyzes the text data using natural language processing (NLP) technology to extract necessary contract information such as the contract holder's name, contract details, contract number, etc. This analysis is performed using tools such as spaCy or the Natural Language Toolkit (NLTK).

[0475] Input: Text data.

[0476] Output: The extracted contract information.

[0477] Step 5:

[0478] Saving to a database

[0479] Specific operation: The server stores the extracted contract information in an SQL database (e.g., MySQL or PostgreSQL).

[0480] Input: The extracted contract information.

[0481] Output: Contract information stored in a database.

[0482] Step 6:

[0483] System access and identity verification by family members

[0484] Specific operation: The user (survivor) accesses the system and logs in using the user's device. The server performs multi-factor authentication (MFA) to verify the identity of the surviving family member, and performs SMS authentication and email authentication.

[0485] Input: Family member's credentials.

[0486] Output: Access granted if identity verification is successful.

[0487] Step 7:

[0488] Generative AI model generates cancellation procedures

[0489] How it works: The server sends the contract information to a generative AI model (e.g., OpenAI's GPT series) and inputs a prompt to generate the appropriate cancellation procedure details. The generative AI model then generates the cancellation procedure accordingly.

[0490] Input: Contract information and prompt text stored in the database.

[0491] Output: Detailed steps of the cancellation procedure generated.

[0492] Step 8:

[0493] Provision of cancellation procedure information

[0494] Specific operation: The server analyzes the cancellation procedure information received from the generated AI model and displays the details on the user's device.

[0495] Input: Cancellation process information from the generative AI model.

[0496] Output: Detailed cancellation information displayed on the user's device.

[0497] (Application example 1)

[0498] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0499] There is a lack of means for surviving family members to quickly and accurately grasp the contract information of the deceased and smoothly complete the cancellation procedures. As a result, the procedures take time and often cause stress for surviving family members. In addition, there are limited means for elderly people to properly manage contract information, which runs the risk of the information being lost. It is important to resolve these issues.

[0500] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0501] In this invention, the server includes a user terminal means for inputting contract information, a server means for converting the input contract information into text data using character recognition or voice recognition technology, a means for analyzing the text data to extract the contract information and saving it in a database, a means for verifying the identity of surviving family members registered from the user terminal means when they access the server, a generation AI model means for providing the surviving family members with contract information and a cancellation procedure method, and a means for generating optimal cancellation procedure details based on the contract information saved in the server means and providing them with the necessary documents, contact information, and cancellation procedure. This enables surviving family members to quickly and accurately check the contract information of the deceased person and easily complete the cancellation procedure.

[0502] "Contract information" is information related to the contractor, including the contractor's name, contract details, contract number, and the like.

[0503] The "user terminal means" is a device for a user to input contract information, and refers to an electronic terminal such as a smartphone or a personal computer.

[0504] "Character recognition technology" refers to technology that extracts text data from captured image data, and includes optical character recognition (OCR) technology.

[0505] "Speech recognition technology" refers to technology that extracts text data from voice data, and includes automatic speech recognition (ASR) technology.

[0506] "Server means" refers to the equipment and software used to process, analyze and store input data.

[0507] "Text data" refers to data obtained by converting image data and audio data into character information.

[0508] "Database" refers to the system and storage for systematically storing and managing analyzed contract information.

[0509] "Personal identification means" refers to means for verifying the identity of the user and their family members, and includes authentication information and biometric authentication technology.

[0510] "Generative AI model means" refers to an artificial intelligence model and software that executes it to generate optimal cancellation procedure details based on stored contract information.

[0511] "Contact Information" refers to the contact information required to complete the cancellation procedure, including, for example, contact information for the company's customer support or related departments.

[0512] "Cancellation Procedure" refers to the steps and procedures required to cancel a contract.

[0513] As an embodiment of this invention, we propose a system that centrally manages contract information and smoothly processes cancellation procedures. This system consists of a user terminal, a server, and a generative AI model.

[0514] 1. Enter contract information

[0515] First, the user enters the contract information. Using a device such as a smartphone or PC, the user can either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input. This process uses optical character recognition (OCR) and automatic speech recognition (ASR).

[0516] 2. Data processing by the server

[0517] Next, the image data or audio data captured by the user device is sent to the server. The server processes the received data, converting the image data into text data using optical character recognition (OCR) technology and the audio data into text using automatic speech recognition (ASR) technology. The server then analyzes the text data, extracts necessary information such as the contract holder's name, contract details, and contract number, and stores it in a database.

[0518] 3. Analysis and storage of contract information

[0519] The server analyzes the converted text data and extracts the necessary contract information, which is then stored in a database and provided to the generative AI model for future cancellation procedures.

[0520] 4. User Access and Identity Verification

[0521] When a user dies, the surviving family members access the system. At this stage, the surviving family members log in using the user terminal and begin the process of verifying the contract information of the deceased. The server performs identity verification to verify the identity of the surviving family members, and provides the information only if this is successful.

[0522] 5. Provision of cancellation procedure information

[0523] The server generates optimal cancellation procedure details based on the contract information stored in the generative AI model, and provides the necessary documents, contact information, and specific cancellation procedures to the bereaved family members, allowing them to smoothly complete the cancellation procedures for various contracts according to the information provided.

[0524] Specific examples

[0525] Example 1: Entering a contract

[0526] User: An elderly person launches the smartphone app and takes a photo of their electricity contract and uploads it.

[0527] Terminal: Sends the contract image to the server.

[0528] Server: The server receives the image and converts it into text data using OCR technology. It then analyzes the contract information and stores it in a database.

[0529] Example 2: Obtaining contract information by a surviving family member

[0530] User (surviving family member): The deceased person's surviving family member logs into the system and checks the cancellation information for the electricity contract.

[0531] Server: Performs identity verification, and if successful, extracts contract information and queries the generation AI model.

[0532] Generative AI model: Generates optimal cancellation details and provides them to the bereaved family, who then follow the cancellation process.

[0533] Prompt Sentence Examples

[0534] Example prompt 1: "Contract holder: Taro Yamada, Contract details: Mobile phone service contract. Please tell me the cancellation procedure."

[0535] Example prompt 2: "Contract holder: Ichiro Tanaka, Contract details: Electricity contract. Please tell me the cancellation procedure."

[0536] In this way, the system efficiently manages the contract information of elderly people and helps surviving family members to complete the procedures quickly and smoothly.

[0537] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0538] Step 1:

[0539] Users enter contract information using devices such as smartphones or PCs. Specifically, they either take a photo of a document such as a contract and upload it, or explain the contract details using voice input. The input data in this case is image data or voice data.

[0540] Step 2:

[0541] The device sends the captured image data or audio data to the server, which processes the data.

[0542] Step 3:

[0543] The server converts the received image data into text data using optical character recognition (OCR) technology. For audio data, it converts it into text using automatic speech recognition (ASR). This is the process of converting input data (image data or audio data) into text data.

[0544] Step 4:

[0545] The server analyzes the converted text data and extracts necessary information such as the contract holder's name, contract details, contract number, etc. The contract information obtained through this analysis process is stored in a database. This output data is the contract information text data.

[0546] Step 5:

[0547] When a user dies, the surviving family members access the system. They log in using the user terminal, and the server verifies the identity of the surviving family members. Information is provided only if the conditions for successful identity verification are met. The input data is the login credentials, and the output data is access authentication.

[0548] Step 6:

[0549] The server queries the generative AI model to generate cancellation procedure details based on the contract information stored in the database. The generative AI model generates optimal cancellation procedure details, required documents, contact information, and cancellation procedures based on the provided contract information. The output data obtained from this process is the cancellation procedure details.

[0550] Step 7:

[0551] The server provides the bereaved family with details of the cancellation procedures obtained from the generative AI model. Based on the information provided, the bereaved family can smoothly proceed with the cancellation procedures for various contracts. The output data is the specific cancellation procedures and related information.

[0552] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0553] As an embodiment of the invention, the system below recognizes the user's emotions during the process from entering contract information to the cancellation procedure and adjusts the process accordingly. This system is composed of a user terminal, a server, a generative AI model, and an emotion engine.

[0554] 1. Enter contract information

[0555] First, the user (elderly person) uses a device such as a smartphone or PC to enter contract information. They either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input. The user device then sends the input data to an emotion engine, which analyzes the user's emotional state.

[0556] 2. Emotion analysis using an emotion engine

[0557] The emotion engine analyzes the user's emotional state based on the voice and image data sent by the device. For example, it analyzes the speaking style and tone of voice from the voice data to recognize feelings of relief or anxiety. It also analyzes facial expressions from image data to recognize feelings of happiness or sadness. This emotional information is sent to the server.

[0558] 3. Data processing by the server

[0559] The server processes the received data. Specifically, it converts image data into text using OCR technology and voice data into text using ASR technology. It also takes into account the emotional information obtained from the emotion engine and provides appropriate feedback to the user.

[0560] 4. Analysis and storage of contract information

[0561] The server analyzes the converted text data and extracts the necessary contract information (e.g., contract holder name, contract details, contract number, etc.). The extracted contract information is stored in a database. It also provides an appropriate interface and explanation based on the user's emotional state based on the results of the emotion analysis.

[0562] 5. The server provides the contract information to the generated AI model

[0563] The server provides contract information to the generative AI model and updates the model to generate recommended procedures for contract cancellation. Feedback to users based on emotional information is also optimized.

[0564] 6. Access and identity verification by family members

[0565] When a user passes away, the surviving family members access the system. At this stage, they log in to the system using a user terminal and begin the process of checking the contract information of the deceased. At the same time, an emotion engine runs to analyze the emotional state of the surviving family members and provide appropriate feedback.

[0566] 7. Providing cancellation procedure information to surviving family members

[0567] The server retrieves details of the cancellation procedure from the generative AI model and provides them to the bereaved family after taking into account emotional information, allowing the bereaved family to complete the cancellation procedure smoothly and without stress.

[0568] Specific examples

[0569] Example 1: Entering a contract

[0570] User: An elderly person launches the smartphone app and takes a photo of their electricity contract and uploads it. The emotion engine analyzes the user's facial expression and determines that they are relaxed.

[0571] Terminal: Sends the contract image and emotion information to the server.

[0572] Server: Converts the image into text data using OCR technology, analyzes and saves the contract information, and provides feedback in the usual way since the subject is in a relaxed state.

[0573] Example 2: Obtaining contract information by a surviving family member

[0574] User (surviving family member): The family member of the deceased logs into the system and checks the cancellation information for the electricity contract. The emotion engine analyzes the family member's emotional state and determines that they are experiencing high levels of stress.

[0575] Server: After verifying the identity of the bereaved, the server extracts the contract information and queries the generative AI model. The generative AI model provides details of the cancellation procedure in an easy-to-understand manner, taking into consideration the emotional state of the bereaved.

[0576] In this way, the system combines an emotion engine to optimize the process from entering contract information to the cancellation procedure in a way that responds to the emotional state of the user and their family members, ensuring smooth operation.

[0577] The processing flow will be explained below.

[0578] Step 1:

[0579] The user enters the contract information. Specifically, the user takes a picture of the contract using a device such as a smartphone or PC, or explains the contract details using voice input. The user's device then uploads this input data to the application.

[0580] Step 2:

[0581] The device sends input data to the emotion engine, which receives image and audio data and analyzes the user's emotional state. For example, it recognizes feelings of relief, anxiety, happiness, sadness, etc. from the user's facial expressions and tone of voice.

[0582] Step 3:

[0583] The emotion engine sends the analysis results to the server. Specifically, if the user is relaxed while inputting, emotion data indicating that state is sent to the server.

[0584] Step 4:

[0585] The device sends the input data to the server. Specifically, image data or audio data of the photographed contract is sent to the server via the Internet. The data may be encrypted during transmission.

[0586] Step 5:

[0587] The server processes the received data. Specifically, image data is converted to text using OCR technology, and voice data is converted to text using ASR (audio-speech recognition) technology. Through this process, the contents of the contract and the contract details explained in the voice are extracted as text data.

[0588] Step 6:

[0589] The server analyzes the text data. Specifically, it extracts the necessary contract information (e.g., contract holder name, contract details, contract number, etc.) from the text data and organizes it as contract information. The analysis results are saved in a database.

[0590] Step 7:

[0591] The server provides feedback based on the emotional information: if the user is relaxed, it provides normal feedback, and if the user is anxious, it provides a reassuring message or additional support information.

[0592] Step 8:

[0593] The server provides the contract information to the generative AI model. Specifically, the extracted and organized contract information is passed to the generative AI model, and the model is updated based on that information to generate recommended cancellation procedures.

[0594] Step 9:

[0595] The user (surviving family member) accesses the system. Specifically, the family member logs into the system using a user terminal and requests to check the contract information of the deceased. The emotion engine analyzes the emotional state of the family member and recognizes high levels of stress and sadness.

[0596] Step 10:

[0597] The server performs identity verification. Specifically, it uses security questions, QR codes, passwords, etc. to verify the identity of the family member. Only if identity verification is successful is access to the contract information granted.

[0598] Step 11:

[0599] The server extracts contract information. Specifically, it searches the database for contract information of the deceased person and extracts the relevant contract information. It then queries the generative AI model to generate cancellation procedure information.

[0600] Step 12:

[0601] The server provides cancellation procedure information to the bereaved family. Specifically, it provides detailed cancellation procedure instructions, necessary documents, and contact information provided by the generative AI model. The server takes into consideration the emotional state of the bereaved family and provides the information in an easy-to-understand and stress-reducing format.

[0602] This flow enables centralized management of contract information, streamlines cancellation procedures, and provides support tailored to the emotional state of the user and their family members.

[0603] Example 2

[0604] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0605] Many users, including the elderly, find the process from entering contract information to the cancellation procedure complicated and stressful. The process of entering contract details and then completing the cancellation procedure can be particularly stressful for elderly users. Furthermore, when a user passes away, there are few ways for their surviving family members to smoothly cancel the contract. To address these issues, a system is needed that recognizes the user's emotional state and optimizes the process based on that information.

[0606] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0607] In this invention, the server includes a user terminal means for inputting contract information, an emotion engine means for analyzing emotion data and recognizing the user's emotional state, a server means for converting the input contract information into text data using character recognition or voice recognition technology, and a means for providing appropriate feedback to the user based on the emotion analysis results. This provides an interface and explanations that correspond to the user's emotional state, reducing stress when entering contract information and completing cancellation procedures and enabling smoother procedures. Furthermore, by utilizing emotion analysis, appropriate support can be provided to surviving family members when they check the deceased's contract information and complete cancellation procedures.

[0608] "Contract information" refers to information in which a user provides details regarding the use of a service or product, including the name of the contractor, contract details, contract number, etc.

[0609] "User terminal" refers to an electronic device used by a user to input contract information, and includes smartphones, personal computers, tablets, etc.

[0610] An "emotion engine" refers to software or a system that analyzes audio and image data to recognize a user's emotional state.

[0611] "Character recognition technology" refers to the technology of identifying characters in an image and converting them into text data, and OCR (Optical Character Recognition) is one such technology.

[0612] "Speech recognition technology" refers to the technology that analyzes voice data and converts it into text data, and ASR (Automatic Speech Recognition) is one such technology.

[0613] "Server" refers to a computer system that processes, stores, and operates generative AI models.

[0614] A "generative AI model" refers to an artificial intelligence model that generates recommended procedures or solutions based on given data.

[0615] "Feedback" refers to information and guidance provided to users based on the results of emotion analysis.

[0616] "Identity verification" refers to the process of verifying that a system user is the person registered, and is carried out to protect access rights.

[0617] A "database" refers to a collection of data that organizes and stores contract information and user information so that it can be efficiently accessed as needed.

[0618] This invention provides a system that recognizes the user's emotions throughout the process, from entering contract information to completing cancellation procedures, and adjusts the process accordingly. This system consists of a user terminal, a server, a generative AI model, and an emotion engine. The details and operation of each component are explained below.

[0619] Enter contract information

[0620] Users enter contract information using devices such as smartphones or PCs. They can either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input. The user device sends the input data (image data or voice data) to the emotion engine.

[0621] Emotion analysis using an emotion engine

[0622] The device receives voice and image data sent by the user, and the emotion engine analyzes them. In the case of voice data, the emotion engine recognizes feelings of relief or anxiety from the speaking style and tone of voice. In the case of image data, the emotion engine recognizes feelings of happiness or sadness from facial expressions. This emotion information is sent to the server.

[0623] Data processing by the server

[0624] The server processes the data (voice data, image data, and emotional information) received from the emotion engine. Specifically, it converts the voice data into text data using automatic speech recognition (ASR) technology, and converts the image data into text data using optical character recognition (OCR) technology. It also provides appropriate feedback to the user based on the emotional information.

[0625] Analysis and storage of contract information

[0626] The server analyzes the converted text data and extracts necessary contract information such as the contract holder's name, contract details, and contract number. The extracted contract information is stored in a database. Based on the results of the sentiment analysis, the server also provides the user with an appropriate interface and explanation.

[0627] The server provides contract information to the generated AI model

[0628] The server provides the contract information to the generative AI model. The generative AI model updates the model to generate recommended procedures for contract cancellation, and also optimizes feedback to the user based on the generated procedures. For example, it may provide the user with specific procedures such as "Please follow the steps below to cancel the contract."

[0629] Family access and identity verification

[0630] When the deceased passes away, the user (survivor) accesses the system and logs in using a user terminal. At this stage, the system analyzes the emotional state of the bereaved and provides appropriate feedback as needed.

[0631] Providing cancellation procedure information to surviving family members

[0632] The server retrieves details of the cancellation procedure from the generative AI model and provides information while taking into consideration the emotional state of the bereaved, allowing the cancellation procedure to proceed smoothly and without stress for the bereaved.

[0633] Specific examples

[0634] Contract entry: The user (elderly person) launches the smartphone app and takes a photo of their electricity contract and uploads it. The emotion engine analyzes the user's facial expression and determines that they are in a relaxed state. The device then sends the contract image and emotion information to the server, which then converts the image into text data using OCR technology and analyzes and saves the contract information. Because the user is in a relaxed state, feedback is provided using the normal procedure.

[0635] Obtaining contract information by surviving family members: The user (surviving family member) logs in to the system and checks the cancellation information for the deceased person's electricity contract. The emotion engine analyzes the emotional state of the surviving family member and determines that they are experiencing high levels of stress. The server performs identity verification, and if successful, extracts the contract information and queries the generation AI model, which then provides details of the cancellation procedure in an easy-to-understand manner, taking into consideration the emotional state of the surviving family member.

[0636] Example prompts for generative AI models

[0637] "What is the best procedure for a user to cancel their electricity contract?"

[0638] "If the emotion analysis results indicate high stress, please suggest ways to provide appropriate feedback to the bereaved."

[0639] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0640] Step 1:

[0641] The user enters contract information using a smartphone or PC. Specifically, they take a photo of the contract and upload it, or explain the contract details using voice input. The input is image data or voice data of the contract, and the output is the initial stage in which this data is processed on the device.

[0642] Step 2:

[0643] The terminal sends image data or voice data of the contract information to the emotion engine. In the case of voice data, the emotion engine analyzes the speaking style and tone of voice, and in the case of image data, it analyzes facial expressions. The input is the image data or voice data of the contract information, and the output is emotional state information (e.g., relaxed, anxious, happy, sad) as the analysis result.

[0644] Step 3:

[0645] The emotion engine analyzes the user's emotional state and sends the results to the server. For example, it analyzes the tone of voice from audio data to recognize feelings of relief or anxiety, or recognizes facial expressions from image data to identify feelings of happiness or sadness. The input is audio or image data, and the output is the analysis result of the emotional state.

[0646] Step 4:

[0647] The server processes contract information based on the received emotional information. Specifically, it converts voice data into text using automatic speech recognition (ASR) technology, and converts image data into text using optical character recognition (OCR) technology. The input is voice data and image data, and the output is text data.

[0648] Step 5:

[0649] The server analyzes the converted text data and extracts necessary contract information such as the contract holder's name, contract details, and contract number. The extracted contract information is stored in a database. For example, OCR technology can be used to extract the contract holder's name and contract details from image data of a contract. The input is text data, and the output is the extracted contract information.

[0650] Step 6:

[0651] The server provides appropriate feedback to the user based on the emotion analysis results. For example, if the user feels anxious, it provides a detailed and friendly explanation, and if the user feels relaxed, it provides standard procedures. The input is emotion information and contract information, and the output is the feedback content.

[0652] Step 7:

[0653] The server provides the contract information to the generative AI model, which then updates the model to generate recommended procedures for contract cancellation. The input is the contract information, and the output is the generated recommended procedures for cancellation.

[0654] Step 8:

[0655] When the server starts the process for the user (survivor) to confirm the contract information of the deceased, it analyzes the emotional state of the survivor and provides appropriate feedback. At this stage, the server receives input information from the survivor and the emotion engine provides the analysis results. The input is the survivor's login information and emotional information, and the output is appropriate feedback.

[0656] Step 9:

[0657] The server obtains details of the cancellation procedure from the generative AI model and provides information while taking into consideration the emotional state of the bereaved. Specifically, this includes gentle language and detailed explanations based on emotion analysis. The input is the details of the cancellation procedure, which are the output of the generative AI model, and the emotional information of the bereaved, and the output is optimized feedback.

[0658] (Application example 2)

[0659] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0660] Conventional systems for entering contract information and for canceling contracts are unable to provide optimal feedback based on the user's emotional state, making it difficult to alleviate the psychological burden, especially for elderly people and bereaved families. Electronic payment services also face the issue of not responding appropriately to users' emotions, resulting in a decline in the quality of the user experience.

[0661] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0662] In this invention, the server includes emotion engine means for analyzing the user's emotions, user terminal means for inputting contract information, and server means for converting the input contract information into text data using character recognition or voice recognition technology. This makes it possible to provide optimal feedback based on the user's emotions and reduce the psychological burden, particularly on elderly people and bereaved families. Furthermore, this invention can improve the quality of the user experience by providing optimal support based on emotion analysis in electronic payment services.

[0663] The "emotion engine means" is a system for analyzing the user's voice data and image data and recognizing the user's emotional state.

[0664] "User terminal means" refers to a device used by a user to input contract information, and includes a smartphone, a personal computer, etc.

[0665] "Character recognition technology" is a technology that extracts character information from image data and converts it into text data, and OCR (optical character recognition) is a representative example.

[0666] "Speech recognition technology" is a technology that analyzes voice data and converts it into text data, and a representative technology is ASR (automatic speech recognition).

[0667] "Server means" means a computer system that processes data sent from a user terminal and stores and provides necessary information.

[0668] The "generative AI model means" is an artificial intelligence model for generating optimal cancellation procedure steps and feedback based on analyzed contract information.

[0669] The "database" is a system for storing processed contract information and emotion analysis results, and providing information as needed.

[0670] "Electronic payment service" means a service for sending and receiving funds over the Internet.

[0671] "Feedback" refers to providing real-time responses and support information based on the user's emotional state and behavior.

[0672] "Transaction information" refers to specific data related to a transaction, including information such as the amount, currency, and transaction details.

[0673] The system for realizing this invention comprises an emotion engine means, a user terminal means, character recognition technology, voice recognition technology, a server means, a generation AI model means, and a processing means for electronic payment services.

[0674] The user terminal means is used to input contract information, voice data, and image data. For example, a user can use a smartphone or PC to take a picture of the contract and explain the contract contents by voice. This data is transmitted to the server means in real time.

[0675] The server means converts the received image data into text data using OCR (Optical Character Recognition) technology, and converts the voice data into text using ASR (Automatic Speech Recognition) technology. The server also uses an emotion engine means to analyze emotions from the user's voice and facial expressions, and recognizes the user's emotional state.

[0676] The emotion engine means is a system for analyzing the user's emotions based on voice data and image data, and grasps the user's emotional state by analyzing the tone of voice, facial expressions, etc. For example, it can recognize emotions such as joy, sadness, and anxiety.

[0677] In particular, in the case of electronic payment services, appropriate feedback and support can be provided based on the user's emotional state based on the results of emotion analysis. For example, if a user feels anxious during the payment process, additional confirmation messages or support information can be displayed.

[0678] The generative AI model generates optimal steps for the cancellation and settlement procedures based on the analyzed contract information and emotional information, allowing the procedures to proceed smoothly and without stress for the user and their family members.

[0679] As a specific example, consider a case where a user takes a photo of an electricity contract through a smartphone app and uploads it to the app. At this time, the emotion engine means analyzes the user's facial expression and, if it determines that the user is relaxed, provides normal feedback. On the other hand, if it determines that the surviving family members are feeling high stress when checking the contract information of the deceased person, it takes the emotional information into consideration and provides the surviving family members with easy-to-understand information on cancellation procedures.

[0680] The following sentences are used as example prompts:

[0681] "Analyze the emotions users have when they hit the checkout button and provide additional feedback if they are feeling anxious or uncertain."

[0682] This allows for appropriate responses depending on the user's emotional state, improving the overall user experience.

[0683] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0684] Step 1:

[0685] The user terminal means inputs contract information. The user then takes a photo of the contract using a smartphone or PC and explains the contract contents in voice. This data is sent to the server in real time. The inputs are image data and voice data, and the server receives these data as output. Specific operations use the smartphone's camera function and voice recording function.

[0686] Step 2:

[0687] The server converts the received image data into text data using OCR technology, and simultaneously converts the voice data into text using ASR technology. The inputs are image data and voice data, and the output is text data. This data conversion uses OpenCV and Google's voice recognition API.

[0688] Step 3:

[0689] The server analyzes the converted text data, extracts contract information (e.g., contract holder name, contract details, contract number, etc.), and stores it in a database. The input is text data, and the output is contract information stored in the database. Specifically, the text data is analyzed using natural language processing technology.

[0690] Step 4:

[0691] The server uses an emotion engine to analyze the voice and image data sent by the user and recognize the user's emotional state. The input is voice and image data, and the output is emotional information. Specific operations include DeepFace and emotion recognition APIs.

[0692] Step 5:

[0693] The server provides feedback according to the user's emotional state based on the emotional information acquired from the emotion engine means. The emotional information is input, and a feedback message is sent to the user terminal as output. A specific operation is to display a customized message.

[0694] Step 6:

[0695] When a registered family member accesses the system from a user terminal, the server verifies the identity of the person. The input is authentication information, and the output is the authentication result. Specific security methods such as two-factor authentication are used.

[0696] Step 7:

[0697] The generative AI model generates detailed steps for the cancellation procedure based on emotional information and contract information, and provides them to the bereaved family. The inputs are emotional information and contract information, and the cancellation procedure is generated as the output. The specific operation uses natural language processing technology and machine learning models.

[0698] Step 8:

[0699] The server processes electronic payment services, confirms payment based on the results of emotion analysis, and displays additional support messages according to the user's emotional state. The inputs are emotion information and payment information, and the output is the payment result and feedback message. Specifically, communication with the payment gateway is performed using an HTTP request.

[0700] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0701] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0702] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0703] [Third embodiment]

[0704] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0705] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0706] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0707] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0708] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0709] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0710] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0711] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0712] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0713] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0714] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0715] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0716] As an embodiment of the invention, the system that centrally manages contract information and smoothly processes cancellations is shown below. This system consists of a user terminal, a server, and a generative AI model.

[0717] 1. Enter contract information

[0718] First, the user (elderly person) enters the contract information. To do this, the user uses a device such as a smartphone or PC. They either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input.

[0719] 2. Data processing by the server

[0720] The device then sends the captured image or audio data to the server, which processes the received data. The image data is converted into text using optical character recognition (OCR) technology, and the audio data is similarly converted into text using automatic speech recognition (ASR) technology.

[0721] 3. Analysis and storage of contract information

[0722] The server analyzes the converted text data and extracts the necessary contract information (contractor name, contract details, contract number, etc.). The extracted contract information is stored in a database. The server also provides this contract information to a generative AI model for future cancellation procedures.

[0723] 4. User Access and Identity Verification

[0724] When a user dies, the surviving family members access the system. At this stage, the surviving family members (users) log in using a user terminal and begin the process of verifying the contract information of the deceased. The server performs identity verification to verify the identity of the surviving family members, and provides information only if this is successful.

[0725] 5. Provision of cancellation procedure information

[0726] The server uses the contract information stored in the generative AI model to provide details of the necessary cancellation procedures, including the necessary documents, contact information, and specific cancellation steps. Survivors can use this information to smoothly complete the cancellation procedures for various contracts.

[0727] Specific examples

[0728] Example 1: Entering a contract

[0729] User: An elderly person launches the smartphone app and takes a photo of their electricity contract and uploads it.

[0730] Terminal: Sends the contract image to the server.

[0731] Server: The server receives the image and converts it into text data using OCR technology. It then analyzes the contract information and stores it in a database.

[0732] Example 2: Obtaining contract information by a surviving family member

[0733] User (surviving family member): The deceased person's surviving family member logs into the system and checks the cancellation information for the electricity contract.

[0734] Server: Performs identity verification, and if successful, extracts contract information and queries the generation AI model.

[0735] Generative AI model: Generates optimal cancellation details and provides them to the bereaved family, who then follow the cancellation process.

[0736] In this way, the system efficiently manages the contract information of elderly people and helps surviving family members to complete the procedures quickly and smoothly.

[0737] The processing flow will be explained below.

[0738] Step 1:

[0739] The user enters the contract information. Specifically, the user takes a picture of the contract using a device such as a smartphone or PC, or explains the contract details using voice input. The user's device then uploads this input data to the application.

[0740] Step 2:

[0741] The device sends the input data to the server. Specifically, image data or audio data of the photographed contract is sent to the server via the Internet. The data may be encrypted during transmission.

[0742] Step 3:

[0743] The server processes the received data. Specifically, image data is converted to text using OCR technology, and voice data is converted to text using automatic speech recognition (ASR). Through this process, the contents of the contract and the contract contents explained in the voice are extracted as text data.

[0744] Step 4:

[0745] The server analyzes the text data. Specifically, it extracts the necessary contract information (e.g., contract holder name, contract details, contract number, etc.) from the text data and organizes it as contract information. The analysis results are saved in a database.

[0746] Step 5:

[0747] The server provides the contract information to the generative AI model. Specifically, the extracted and organized contract information is passed to the generative AI model, and the model is updated based on that information to generate recommended cancellation procedures.

[0748] Step 6:

[0749] The user (surviving family member) accesses the system. Specifically, the surviving family member logs into the system using a user terminal and inputs that they would like to check the contract information of the deceased person.

[0750] Step 7:

[0751] The server performs identity verification. Specifically, it uses security questions, QR codes, passwords, etc. to verify the identity of the family member. Only if identity verification is successful is access to the contract information granted.

[0752] Step 8:

[0753] The server extracts contract information. Specifically, it searches the database for contract information of the deceased person and extracts the relevant contract information. It then queries the generative AI model to generate cancellation procedure information.

[0754] Step 9:

[0755] The server provides cancellation procedure information, specifically detailed cancellation procedures provided by the generative AI model, as well as necessary documents and contact information, to the bereaved family, helping them to complete the cancellation process smoothly.

[0756] This flow provides a system that enables centralized management of contract information and streamlines cancellation procedures.

[0757] Example 1

[0758] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0759] It is extremely complicated for elderly people to manage a large number of contract information, and the cumbersome procedures become a problem, especially when contract cancellation procedures are required. In addition, in order for surviving family members to smoothly handle the contract information of the deceased, it is necessary to collect appropriate contract information and quickly cancel the contracts, but these procedures are not simple.

[0760] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0761] In this invention, the server includes means for converting input contract information into text data using optical character recognition technology or voice recognition technology, means for analyzing the text data using natural language processing technology to extract contract information and store it in a database, means for verifying the identity of surviving family members who log in from user terminal means using multi-factor authentication, and generation AI model means for providing contract information and details of cancellation procedures to surviving family members. This makes it possible to centrally manage contract information and provide quick and smooth cancellation procedures.

[0762] "Contract information" refers to data related to the contracts and contract contents owned by the user, including the name of the contract holder, contract contents, contract number, etc.

[0763] "User terminal" refers to a device used by a user to enter contract information. This includes smartphones, personal computers, etc.

[0764] "Server" means a computer system that receives, processes, and manages data sent from user terminals.

[0765] Optical character recognition (OCR) is a technology that extracts character information from image data and converts it into text data.

[0766] "ASR (audio recognition)" is a technology that analyzes voice data and converts it into text data.

[0767] "Natural language processing technology (NLP)" is a technology for analyzing text data and extracting specific information.

[0768] "Database" refers to a system for storing and managing extracted contract information.

[0769] "Multi-factor authentication (MFA)" is an authentication method that uses multiple means of verification to authenticate a user.

[0770] A "generative AI model" is a model that uses artificial intelligence to automatically perform specific tasks. In this case, its role is to generate cancellation procedure details based on contract information.

[0771] This invention is a system for centrally managing contract information and smoothly carrying out cancellation procedures. This system consists of a user terminal, a server, and a generative AI model, and is implemented using the following hardware and software.

[0772] Use of user terminals

[0773] First, the user enters contract information using a device such as a smartphone or PC. The user can either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input. This allows the user to easily provide the contract information to the system.

[0774] Server Processing

[0775] The device then sends the captured image or audio data to a server. The server processes the received data. The image data is converted into text using optical character recognition (OCR) technology, and the audio data is similarly converted into text using automatic speech recognition (ASR) technology. To achieve this, services such as Google Cloud Vision API and IBM Watson Speech to Text are used. The converted text data is then analyzed using natural language processing (NLP) technology (such as spaCy or Natural Language Toolkit (NLTK)) to extract the necessary contract information.

[0776] Information storage and analysis

[0777] The server stores the extracted contract information in a database, typically an SQL database (e.g., MySQL or PostgreSQL), and provides the extracted contract information to a generative AI model for future cancellation procedures.

[0778] Access by family members

[0779] When a user dies, the surviving family members access the system. At this stage, the surviving family members log in to the system using the user's device and begin the process of verifying the deceased's contract information. The server then uses a multi-factor authentication (MFA) system (e.g., SMS authentication or email authentication) to verify the identity of the surviving family members. Only if the identity verification is successful will the deceased's contract information be provided.

[0780] Support for cancellation procedures

[0781] The server uses a generative AI model to generate cancellation procedure details based on the contract information. For example, by using OpenAI's GPT series, the server can provide optimal cancellation procedure details to the bereaved family.

[0782] Specific examples

[0783] Entering the contract

[0784] User: An elderly person launches the smartphone app, takes a photo of their electricity contract, and uploads it.

[0785] Terminal: Sends the contract image to the server.

[0786] Server: The server receives the image and converts it into text data using OCR technology (Google Cloud Vision API), then analyzes the contract information using natural language processing (NLP) technology and stores it in an SQL database.

[0787] Acquisition of contract information by surviving family members

[0788] User (surviving family member): The deceased person's surviving family member logs into the system and checks the cancellation information for the electricity contract.

[0789] Server: Performs identity verification, and if successful, extracts contract information and queries the generative AI model (OpenAI GPT).

[0790] Generative AI model: Generates optimal cancellation details and provides them to the bereaved family, who then follow the cancellation process.

[0791] Prompt Sentence Examples

[0792] Please tell me the procedure for canceling my electricity contract.

[0793] How do I cancel a deceased person's mobile phone contract?

[0794] "Please tell me the documents required to cancel the water contract."

[0795] In this way, the purpose of this system is to efficiently manage the contract information of elderly people and to support the surviving family members in completing the procedures quickly and smoothly.

[0796] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0797] Step 1:

[0798] Enter contract information

[0799] Specific operation: The user enters contract information from a smartphone or PC. The user uses a dedicated application to take a photo of a document such as a contract and upload it to the application. Alternatively, the user can dictate the contract details using the voice input function.

[0800] Input: Image data or audio recording of a contract uploaded by the user.

[0801] Output: Image or audio data sent by the device to the server.

[0802] Step 2:

[0803] Data reception and initial processing by the server

[0804] Specific operation: The server receives image data or audio data sent from the user terminal. The received data is temporarily stored in storage.

[0805] Input: Image or audio data sent from the device.

[0806] Output: Data saved in temporary storage.

[0807] Step 3:

[0808] OCR and ASR text conversion

[0809] Specific operation: The server applies optical character recognition (OCR) technology to the received image data and converts it into text data. It also applies automatic speech recognition (ASR) technology to the audio data and converts it into text in the same way. It uses Google Cloud Vision API, IBM Watson Speech to Text, etc.

[0810] Input: Image or audio data stored in temporary storage.

[0811] Output: Contract information converted to text data.

[0812] Step 4:

[0813] Analysis of text data and extraction of contract information

[0814] Specific operation: The server analyzes the text data using natural language processing (NLP) technology to extract necessary contract information such as the contract holder's name, contract details, contract number, etc. This analysis is performed using tools such as spaCy or the Natural Language Toolkit (NLTK).

[0815] Input: Text data.

[0816] Output: The extracted contract information.

[0817] Step 5:

[0818] Saving to a database

[0819] Specific operation: The server stores the extracted contract information in an SQL database (e.g., MySQL or PostgreSQL).

[0820] Input: The extracted contract information.

[0821] Output: Contract information stored in a database.

[0822] Step 6:

[0823] System access and identity verification by family members

[0824] Specific operation: The user (survivor) accesses the system and logs in using the user's device. The server performs multi-factor authentication (MFA) to verify the identity of the surviving family member, and performs SMS authentication and email authentication.

[0825] Input: Family member's credentials.

[0826] Output: Access granted if identity verification is successful.

[0827] Step 7:

[0828] Generative AI model generates cancellation procedures

[0829] How it works: The server sends the contract information to a generative AI model (e.g., OpenAI's GPT series) and inputs a prompt to generate the appropriate cancellation procedure details. The generative AI model then generates the cancellation procedure accordingly.

[0830] Input: Contract information and prompt text stored in the database.

[0831] Output: Detailed steps of the cancellation procedure generated.

[0832] Step 8:

[0833] Provision of cancellation procedure information

[0834] Specific operation: The server analyzes the cancellation procedure information received from the generated AI model and displays the details on the user's device.

[0835] Input: Cancellation process information from the generative AI model.

[0836] Output: Detailed cancellation information displayed on the user's device.

[0837] (Application example 1)

[0838] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0839] There is a lack of means for surviving family members to quickly and accurately grasp the contract information of the deceased and smoothly complete the cancellation procedures. As a result, the procedures take time and often cause stress for surviving family members. In addition, there are limited means for elderly people to properly manage contract information, which runs the risk of the information being lost. It is important to resolve these issues.

[0840] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0841] In this invention, the server includes a user terminal means for inputting contract information, a server means for converting the input contract information into text data using character recognition or voice recognition technology, a means for analyzing the text data to extract the contract information and saving it in a database, a means for verifying the identity of surviving family members registered from the user terminal means when they access the server, a generation AI model means for providing the surviving family members with contract information and a cancellation procedure method, and a means for generating optimal cancellation procedure details based on the contract information saved in the server means and providing them with the necessary documents, contact information, and cancellation procedure. This enables surviving family members to quickly and accurately check the contract information of the deceased person and easily complete the cancellation procedure.

[0842] "Contract information" is information related to the contractor, including the contractor's name, contract details, contract number, and the like.

[0843] The "user terminal means" is a device for a user to input contract information, and refers to an electronic terminal such as a smartphone or a personal computer.

[0844] "Character recognition technology" refers to technology that extracts text data from captured image data, and includes optical character recognition (OCR) technology.

[0845] "Speech recognition technology" refers to technology that extracts text data from voice data, and includes automatic speech recognition (ASR) technology.

[0846] "Server means" refers to the equipment and software used to process, analyze and store input data.

[0847] "Text data" refers to data obtained by converting image data and audio data into character information.

[0848] "Database" refers to the system and storage for systematically storing and managing analyzed contract information.

[0849] "Personal identification means" refers to means for verifying the identity of the user and their family members, and includes authentication information and biometric authentication technology.

[0850] "Generative AI model means" refers to an artificial intelligence model and software that executes it to generate optimal cancellation procedure details based on stored contract information.

[0851] "Contact Information" refers to the contact information required to complete the cancellation procedure, including, for example, contact information for the company's customer support or related departments.

[0852] "Cancellation Procedure" refers to the steps and procedures required to cancel a contract.

[0853] As an embodiment of this invention, we propose a system that centrally manages contract information and smoothly processes cancellation procedures. This system consists of a user terminal, a server, and a generative AI model.

[0854] 1. Enter contract information

[0855] First, the user enters the contract information. Using a device such as a smartphone or PC, the user can either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input. This process uses optical character recognition (OCR) and automatic speech recognition (ASR).

[0856] 2. Data processing by the server

[0857] Next, the image data or audio data captured by the user device is sent to the server. The server processes the received data, converting the image data into text data using optical character recognition (OCR) technology and the audio data into text using automatic speech recognition (ASR) technology. The server then analyzes the text data, extracts necessary information such as the contract holder's name, contract details, and contract number, and stores it in a database.

[0858] 3. Analysis and storage of contract information

[0859] The server analyzes the converted text data and extracts the necessary contract information, which is then stored in a database and provided to the generative AI model for future cancellation procedures.

[0860] 4. User Access and Identity Verification

[0861] When a user dies, the surviving family members access the system. At this stage, the surviving family members log in using the user terminal and begin the process of verifying the contract information of the deceased. The server performs identity verification to verify the identity of the surviving family members, and provides the information only if this is successful.

[0862] 5. Provision of cancellation procedure information

[0863] The server generates optimal cancellation procedure details based on the contract information stored in the generative AI model, and provides the necessary documents, contact information, and specific cancellation procedures to the bereaved family members, allowing them to smoothly complete the cancellation procedures for various contracts according to the information provided.

[0864] Specific examples

[0865] Example 1: Entering a contract

[0866] User: An elderly person launches the smartphone app and takes a photo of their electricity contract and uploads it.

[0867] Terminal: Sends the contract image to the server.

[0868] Server: The server receives the image and converts it into text data using OCR technology. It then analyzes the contract information and stores it in a database.

[0869] Example 2: Obtaining contract information by a surviving family member

[0870] User (surviving family member): The deceased person's surviving family member logs into the system and checks the cancellation information for the electricity contract.

[0871] Server: Performs identity verification, and if successful, extracts contract information and queries the generation AI model.

[0872] Generative AI model: Generates optimal cancellation details and provides them to the bereaved family, who then follow the cancellation process.

[0873] Prompt Sentence Examples

[0874] Example prompt 1: "Contract holder: Taro Yamada, Contract details: Mobile phone service contract. Please tell me the cancellation procedure."

[0875] Example prompt 2: "Contract holder: Ichiro Tanaka, Contract details: Electricity contract. Please tell me the cancellation procedure."

[0876] In this way, the system efficiently manages the contract information of elderly people and helps surviving family members to complete the procedures quickly and smoothly.

[0877] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0878] Step 1:

[0879] Users enter contract information using devices such as smartphones or PCs. Specifically, they either take a photo of a document such as a contract and upload it, or explain the contract details using voice input. The input data in this case is image data or voice data.

[0880] Step 2:

[0881] The device sends the captured image data or audio data to the server, which processes the data.

[0882] Step 3:

[0883] The server converts the received image data into text data using optical character recognition (OCR) technology. For audio data, it converts it into text using automatic speech recognition (ASR). This is the process of converting input data (image data or audio data) into text data.

[0884] Step 4:

[0885] The server analyzes the converted text data and extracts necessary information such as the contract holder's name, contract details, contract number, etc. The contract information obtained through this analysis process is stored in a database. This output data is the contract information text data.

[0886] Step 5:

[0887] When a user dies, the surviving family members access the system. They log in using the user terminal, and the server verifies the identity of the surviving family members. Information is provided only if the conditions for successful identity verification are met. The input data is the login credentials, and the output data is access authentication.

[0888] Step 6:

[0889] The server queries the generative AI model to generate cancellation procedure details based on the contract information stored in the database. The generative AI model generates optimal cancellation procedure details, required documents, contact information, and cancellation procedures based on the provided contract information. The output data obtained from this process is the cancellation procedure details.

[0890] Step 7:

[0891] The server provides the bereaved family with details of the cancellation procedures obtained from the generative AI model. Based on the information provided, the bereaved family can smoothly proceed with the cancellation procedures for various contracts. The output data is the specific cancellation procedures and related information.

[0892] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0893] As an embodiment of the invention, the system below recognizes the user's emotions during the process from entering contract information to the cancellation procedure and adjusts the process accordingly. This system is composed of a user terminal, a server, a generative AI model, and an emotion engine.

[0894] 1. Enter contract information

[0895] First, the user (elderly person) uses a device such as a smartphone or PC to enter contract information. They either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input. The user device then sends the input data to an emotion engine, which analyzes the user's emotional state.

[0896] 2. Emotion analysis using an emotion engine

[0897] The emotion engine analyzes the user's emotional state based on the voice and image data sent by the device. For example, it analyzes the speaking style and tone of voice from the voice data to recognize feelings of relief or anxiety. It also analyzes facial expressions from image data to recognize feelings of happiness or sadness. This emotional information is sent to the server.

[0898] 3. Data processing by the server

[0899] The server processes the received data. Specifically, it converts image data into text using OCR technology and voice data into text using ASR technology. It also takes into account the emotional information obtained from the emotion engine and provides appropriate feedback to the user.

[0900] 4. Analysis and storage of contract information

[0901] The server analyzes the converted text data and extracts the necessary contract information (e.g., contract holder name, contract details, contract number, etc.). The extracted contract information is stored in a database. It also provides an appropriate interface and explanation based on the user's emotional state based on the results of the emotion analysis.

[0902] 5. The server provides the contract information to the generated AI model

[0903] The server provides contract information to the generative AI model and updates the model to generate recommended procedures for contract cancellation. Feedback to users based on emotional information is also optimized.

[0904] 6. Access and identity verification by family members

[0905] When a user passes away, the surviving family members access the system. At this stage, they log in to the system using a user terminal and begin the process of checking the contract information of the deceased. At the same time, an emotion engine runs to analyze the emotional state of the surviving family members and provide appropriate feedback.

[0906] 7. Providing cancellation procedure information to surviving family members

[0907] The server retrieves details of the cancellation procedure from the generative AI model and provides them to the bereaved family after taking into account emotional information, allowing the bereaved family to complete the cancellation procedure smoothly and without stress.

[0908] Specific examples

[0909] Example 1: Entering a contract

[0910] User: An elderly person launches the smartphone app and takes a photo of their electricity contract and uploads it. The emotion engine analyzes the user's facial expression and determines that they are relaxed.

[0911] Terminal: Sends the contract image and emotion information to the server.

[0912] Server: Converts the image into text data using OCR technology, analyzes and saves the contract information, and provides feedback in the usual way since the subject is in a relaxed state.

[0913] Example 2: Obtaining contract information by a surviving family member

[0914] User (surviving family member): The family member of the deceased logs into the system and checks the cancellation information for the electricity contract. The emotion engine analyzes the family member's emotional state and determines that they are experiencing high levels of stress.

[0915] Server: After verifying the identity of the bereaved, the server extracts the contract information and queries the generative AI model. The generative AI model provides details of the cancellation procedure in an easy-to-understand manner, taking into consideration the emotional state of the bereaved.

[0916] In this way, the system combines an emotion engine to optimize the process from entering contract information to the cancellation procedure in a way that responds to the emotional state of the user and their family members, ensuring smooth operation.

[0917] The processing flow will be explained below.

[0918] Step 1:

[0919] The user enters the contract information. Specifically, the user takes a picture of the contract using a device such as a smartphone or PC, or explains the contract details using voice input. The user's device then uploads this input data to the application.

[0920] Step 2:

[0921] The device sends input data to the emotion engine, which receives image and audio data and analyzes the user's emotional state. For example, it recognizes feelings of relief, anxiety, happiness, sadness, etc. from the user's facial expressions and tone of voice.

[0922] Step 3:

[0923] The emotion engine sends the analysis results to the server. Specifically, if the user is relaxed while inputting, emotion data indicating that state is sent to the server.

[0924] Step 4:

[0925] The device sends the input data to the server. Specifically, image data or audio data of the photographed contract is sent to the server via the Internet. The data may be encrypted during transmission.

[0926] Step 5:

[0927] The server processes the received data. Specifically, image data is converted to text using OCR technology, and voice data is converted to text using ASR (audio-speech recognition) technology. Through this process, the contents of the contract and the contract details explained in the voice are extracted as text data.

[0928] Step 6:

[0929] The server analyzes the text data. Specifically, it extracts the necessary contract information (e.g., contract holder name, contract details, contract number, etc.) from the text data and organizes it as contract information. The analysis results are saved in a database.

[0930] Step 7:

[0931] The server provides feedback based on the emotional information: if the user is relaxed, it provides normal feedback, and if the user is anxious, it provides a reassuring message or additional support information.

[0932] Step 8:

[0933] The server provides the contract information to the generative AI model. Specifically, the extracted and organized contract information is passed to the generative AI model, and the model is updated based on that information to generate recommended cancellation procedures.

[0934] Step 9:

[0935] The user (surviving family member) accesses the system. Specifically, the family member logs into the system using a user terminal and requests to check the contract information of the deceased. The emotion engine analyzes the emotional state of the family member and recognizes high levels of stress and sadness.

[0936] Step 10:

[0937] The server performs identity verification. Specifically, it uses security questions, QR codes, passwords, etc. to verify the identity of the family member. Only if identity verification is successful is access to the contract information granted.

[0938] Step 11:

[0939] The server extracts contract information. Specifically, it searches the database for contract information of the deceased person and extracts the relevant contract information. It then queries the generative AI model to generate cancellation procedure information.

[0940] Step 12:

[0941] The server provides cancellation procedure information to the bereaved family. Specifically, it provides detailed cancellation procedure instructions, necessary documents, and contact information provided by the generative AI model. The server takes into consideration the emotional state of the bereaved family and provides the information in an easy-to-understand and stress-reducing format.

[0942] This flow enables centralized management of contract information, streamlines cancellation procedures, and provides support tailored to the emotional state of the user and their family members.

[0943] Example 2

[0944] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0945] Many users, including the elderly, find the process from entering contract information to the cancellation procedure complicated and stressful. The process of entering contract details and then completing the cancellation procedure can be particularly stressful for elderly users. Furthermore, when a user passes away, there are few ways for their surviving family members to smoothly cancel the contract. To address these issues, a system is needed that recognizes the user's emotional state and optimizes the process based on that information.

[0946] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0947] In this invention, the server includes a user terminal means for inputting contract information, an emotion engine means for analyzing emotion data and recognizing the user's emotional state, a server means for converting the input contract information into text data using character recognition or voice recognition technology, and a means for providing appropriate feedback to the user based on the emotion analysis results. This provides an interface and explanations that correspond to the user's emotional state, reducing stress when entering contract information and completing cancellation procedures and enabling smoother procedures. Furthermore, by utilizing emotion analysis, appropriate support can be provided to surviving family members when they check the deceased's contract information and complete cancellation procedures.

[0948] "Contract information" refers to information in which a user provides details regarding the use of a service or product, including the name of the contractor, contract details, contract number, etc.

[0949] "User terminal" refers to an electronic device used by a user to input contract information, and includes smartphones, personal computers, tablets, etc.

[0950] An "emotion engine" refers to software or a system that analyzes audio and image data to recognize a user's emotional state.

[0951] "Character recognition technology" refers to the technology of identifying characters in an image and converting them into text data, and OCR (Optical Character Recognition) is one such technology.

[0952] "Speech recognition technology" refers to the technology that analyzes voice data and converts it into text data, and ASR (Automatic Speech Recognition) is one such technology.

[0953] "Server" refers to a computer system that processes, stores, and operates generative AI models.

[0954] A "generative AI model" refers to an artificial intelligence model that generates recommended procedures or solutions based on given data.

[0955] "Feedback" refers to information and guidance provided to users based on the results of emotion analysis.

[0956] "Identity verification" refers to the process of verifying that a system user is the person registered, and is carried out to protect access rights.

[0957] A "database" refers to a collection of data that organizes and stores contract information and user information so that it can be efficiently accessed as needed.

[0958] This invention provides a system that recognizes the user's emotions throughout the process, from entering contract information to completing cancellation procedures, and adjusts the process accordingly. This system consists of a user terminal, a server, a generative AI model, and an emotion engine. The details and operation of each component are explained below.

[0959] Enter contract information

[0960] Users enter contract information using devices such as smartphones or PCs. They can either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input. The user device sends the input data (image data or voice data) to the emotion engine.

[0961] Emotion analysis using an emotion engine

[0962] The device receives voice and image data sent by the user, and the emotion engine analyzes them. In the case of voice data, the emotion engine recognizes feelings of relief or anxiety from the speaking style and tone of voice. In the case of image data, the emotion engine recognizes feelings of happiness or sadness from facial expressions. This emotion information is sent to the server.

[0963] Data processing by the server

[0964] The server processes the data (voice data, image data, and emotional information) received from the emotion engine. Specifically, it converts the voice data into text data using automatic speech recognition (ASR) technology, and converts the image data into text data using optical character recognition (OCR) technology. It also provides appropriate feedback to the user based on the emotional information.

[0965] Analysis and storage of contract information

[0966] The server analyzes the converted text data and extracts necessary contract information such as the contract holder's name, contract details, and contract number. The extracted contract information is stored in a database. Based on the results of the sentiment analysis, the server also provides the user with an appropriate interface and explanation.

[0967] The server provides contract information to the generated AI model

[0968] The server provides the contract information to the generative AI model. The generative AI model updates the model to generate recommended procedures for contract cancellation, and also optimizes feedback to the user based on the generated procedures. For example, it may provide the user with specific procedures such as "Please follow the steps below to cancel the contract."

[0969] Family access and identity verification

[0970] When the deceased passes away, the user (survivor) accesses the system and logs in using a user terminal. At this stage, the system analyzes the emotional state of the bereaved and provides appropriate feedback as needed.

[0971] Providing cancellation procedure information to surviving family members

[0972] The server retrieves details of the cancellation procedure from the generative AI model and provides information while taking into consideration the emotional state of the bereaved, allowing the cancellation procedure to proceed smoothly and without stress for the bereaved.

[0973] Specific examples

[0974] Contract entry: The user (elderly person) launches the smartphone app and takes a photo of their electricity contract and uploads it. The emotion engine analyzes the user's facial expression and determines that they are in a relaxed state. The device then sends the contract image and emotion information to the server, which then converts the image into text data using OCR technology and analyzes and saves the contract information. Because the user is in a relaxed state, feedback is provided using the normal procedure.

[0975] Obtaining contract information by surviving family members: The user (surviving family member) logs in to the system and checks the cancellation information for the deceased person's electricity contract. The emotion engine analyzes the emotional state of the surviving family member and determines that they are experiencing high levels of stress. The server performs identity verification, and if successful, extracts the contract information and queries the generation AI model, which then provides details of the cancellation procedure in an easy-to-understand manner, taking into consideration the emotional state of the surviving family member.

[0976] Example prompts for generative AI models

[0977] "What is the best procedure for a user to cancel their electricity contract?"

[0978] "If the emotion analysis results indicate high stress, please suggest ways to provide appropriate feedback to the bereaved."

[0979] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0980] Step 1:

[0981] The user enters contract information using a smartphone or PC. Specifically, they take a photo of the contract and upload it, or explain the contract details using voice input. The input is image data or voice data of the contract, and the output is the initial stage in which this data is processed on the device.

[0982] Step 2:

[0983] The terminal sends image data or voice data of the contract information to the emotion engine. In the case of voice data, the emotion engine analyzes the speaking style and tone of voice, and in the case of image data, it analyzes facial expressions. The input is the image data or voice data of the contract information, and the output is emotional state information (e.g., relaxed, anxious, happy, sad) as the analysis result.

[0984] Step 3:

[0985] The emotion engine analyzes the user's emotional state and sends the results to the server. For example, it analyzes the tone of voice from audio data to recognize feelings of relief or anxiety, or recognizes facial expressions from image data to identify feelings of happiness or sadness. The input is audio or image data, and the output is the analysis result of the emotional state.

[0986] Step 4:

[0987] The server processes contract information based on the received emotional information. Specifically, it converts voice data into text using automatic speech recognition (ASR) technology, and converts image data into text using optical character recognition (OCR) technology. The input is voice data and image data, and the output is text data.

[0988] Step 5:

[0989] The server analyzes the converted text data and extracts necessary contract information such as the contract holder's name, contract details, and contract number. The extracted contract information is stored in a database. For example, OCR technology can be used to extract the contract holder's name and contract details from image data of a contract. The input is text data, and the output is the extracted contract information.

[0990] Step 6:

[0991] The server provides appropriate feedback to the user based on the emotion analysis results. For example, if the user feels anxious, it provides a detailed and friendly explanation, and if the user feels relaxed, it provides standard procedures. The input is emotion information and contract information, and the output is the feedback content.

[0992] Step 7:

[0993] The server provides the contract information to the generative AI model, which then updates the model to generate recommended procedures for contract cancellation. The input is the contract information, and the output is the generated recommended procedures for cancellation.

[0994] Step 8:

[0995] When the server starts the process for the user (survivor) to confirm the contract information of the deceased, it analyzes the emotional state of the survivor and provides appropriate feedback. At this stage, the server receives input information from the survivor and the emotion engine provides the analysis results. The input is the survivor's login information and emotional information, and the output is appropriate feedback.

[0996] Step 9:

[0997] The server obtains details of the cancellation procedure from the generative AI model and provides information while taking into consideration the emotional state of the bereaved. Specifically, this includes gentle language and detailed explanations based on emotion analysis. The input is the details of the cancellation procedure, which are the output of the generative AI model, and the emotional information of the bereaved, and the output is optimized feedback.

[0998] (Application example 2)

[0999] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1000] Conventional systems for entering contract information and for canceling contracts are unable to provide optimal feedback based on the user's emotional state, making it difficult to alleviate the psychological burden, especially for elderly people and bereaved families. Electronic payment services also face the issue of not responding appropriately to users' emotions, resulting in a decline in the quality of the user experience.

[1001] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1002] In this invention, the server includes emotion engine means for analyzing the user's emotions, user terminal means for inputting contract information, and server means for converting the input contract information into text data using character recognition or voice recognition technology. This makes it possible to provide optimal feedback based on the user's emotions and reduce the psychological burden, particularly on elderly people and bereaved families. Furthermore, this invention can improve the quality of the user experience by providing optimal support based on emotion analysis in electronic payment services.

[1003] The "emotion engine means" is a system for analyzing the user's voice data and image data and recognizing the user's emotional state.

[1004] "User terminal means" refers to a device used by a user to input contract information, and includes a smartphone, a personal computer, etc.

[1005] "Character recognition technology" is a technology that extracts character information from image data and converts it into text data, and OCR (optical character recognition) is a representative example.

[1006] "Speech recognition technology" is a technology that analyzes voice data and converts it into text data, and a representative technology is ASR (automatic speech recognition).

[1007] "Server means" means a computer system that processes data sent from a user terminal and stores and provides necessary information.

[1008] The "generative AI model means" is an artificial intelligence model for generating optimal cancellation procedure steps and feedback based on analyzed contract information.

[1009] The "database" is a system for storing processed contract information and emotion analysis results, and providing information as needed.

[1010] "Electronic payment service" means a service for sending and receiving funds over the Internet.

[1011] "Feedback" refers to providing real-time responses and support information based on the user's emotional state and behavior.

[1012] "Transaction information" refers to specific data related to a transaction, including information such as the amount, currency, and transaction details.

[1013] The system for realizing this invention comprises an emotion engine means, a user terminal means, character recognition technology, voice recognition technology, a server means, a generation AI model means, and a processing means for electronic payment services.

[1014] The user terminal means is used to input contract information, voice data, and image data. For example, a user can use a smartphone or PC to take a picture of the contract and explain the contract contents by voice. This data is transmitted to the server means in real time.

[1015] The server means converts the received image data into text data using OCR (Optical Character Recognition) technology, and converts the voice data into text using ASR (Automatic Speech Recognition) technology. The server also uses an emotion engine means to analyze emotions from the user's voice and facial expressions, and recognizes the user's emotional state.

[1016] The emotion engine means is a system for analyzing the user's emotions based on voice data and image data, and grasps the user's emotional state by analyzing the tone of voice, facial expressions, etc. For example, it can recognize emotions such as joy, sadness, and anxiety.

[1017] In particular, in the case of electronic payment services, appropriate feedback and support can be provided based on the user's emotional state based on the results of emotion analysis. For example, if a user feels anxious during the payment process, additional confirmation messages or support information can be displayed.

[1018] The generative AI model generates optimal steps for the cancellation and settlement procedures based on the analyzed contract information and emotional information, allowing the procedures to proceed smoothly and without stress for the user and their family members.

[1019] As a specific example, consider a case where a user takes a photo of an electricity contract through a smartphone app and uploads it to the app. At this time, the emotion engine means analyzes the user's facial expression and, if it determines that the user is relaxed, provides normal feedback. On the other hand, if it determines that the surviving family members are feeling high stress when checking the contract information of the deceased person, it takes the emotional information into consideration and provides the surviving family members with easy-to-understand information on cancellation procedures.

[1020] The following sentences are used as example prompts:

[1021] "Analyze the emotions users have when they hit the checkout button and provide additional feedback if they are feeling anxious or uncertain."

[1022] This allows for appropriate responses depending on the user's emotional state, improving the overall user experience.

[1023] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1024] Step 1:

[1025] The user terminal means inputs contract information. The user then takes a photo of the contract using a smartphone or PC and explains the contract contents in voice. This data is sent to the server in real time. The inputs are image data and voice data, and the server receives these data as output. Specific operations use the smartphone's camera function and voice recording function.

[1026] Step 2:

[1027] The server converts the received image data into text data using OCR technology, and simultaneously converts the voice data into text using ASR technology. The inputs are image data and voice data, and the output is text data. This data conversion uses OpenCV and Google's voice recognition API.

[1028] Step 3:

[1029] The server analyzes the converted text data, extracts contract information (e.g., contract holder name, contract details, contract number, etc.), and stores it in a database. The input is text data, and the output is contract information stored in the database. Specifically, the text data is analyzed using natural language processing technology.

[1030] Step 4:

[1031] The server uses an emotion engine to analyze the voice and image data sent by the user and recognize the user's emotional state. The input is voice and image data, and the output is emotional information. Specific operations include DeepFace and emotion recognition APIs.

[1032] Step 5:

[1033] The server provides feedback according to the user's emotional state based on the emotional information acquired from the emotion engine means. The emotional information is input, and a feedback message is sent to the user terminal as output. A specific operation is to display a customized message.

[1034] Step 6:

[1035] When a registered family member accesses the system from a user terminal, the server verifies the identity of the person. The input is authentication information, and the output is the authentication result. Specific security methods such as two-factor authentication are used.

[1036] Step 7:

[1037] The generative AI model generates detailed steps for the cancellation procedure based on emotional information and contract information, and provides them to the bereaved family. The inputs are emotional information and contract information, and the cancellation procedure is generated as the output. The specific operation uses natural language processing technology and machine learning models.

[1038] Step 8:

[1039] The server processes electronic payment services, confirms payment based on the results of emotion analysis, and displays additional support messages according to the user's emotional state. The inputs are emotion information and payment information, and the output is the payment result and feedback message. Specifically, communication with the payment gateway is performed using an HTTP request.

[1040] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1041] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1042] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1043] [Fourth embodiment]

[1044] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1045] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1046] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1047] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1048] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1049] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1050] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1051] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1052] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1053] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1054] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1055] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1056] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1057] As an embodiment of the invention, the system that centrally manages contract information and smoothly processes cancellations is shown below. This system consists of a user terminal, a server, and a generative AI model.

[1058] 1. Enter contract information

[1059] First, the user (elderly person) enters the contract information. To do this, the user uses a device such as a smartphone or PC. They either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input.

[1060] 2. Data processing by the server

[1061] The device then sends the captured image or audio data to the server, which processes the received data. The image data is converted into text using optical character recognition (OCR) technology, and the audio data is similarly converted into text using automatic speech recognition (ASR) technology.

[1062] 3. Analysis and storage of contract information

[1063] The server analyzes the converted text data and extracts the necessary contract information (contractor name, contract details, contract number, etc.). The extracted contract information is stored in a database. The server also provides this contract information to a generative AI model for future cancellation procedures.

[1064] 4. User Access and Identity Verification

[1065] When a user dies, the surviving family members access the system. At this stage, the surviving family members (users) log in using a user terminal and begin the process of verifying the contract information of the deceased. The server performs identity verification to verify the identity of the surviving family members, and provides information only if this is successful.

[1066] 5. Provision of cancellation procedure information

[1067] The server uses the contract information stored in the generative AI model to provide details of the necessary cancellation procedures, including the necessary documents, contact information, and specific cancellation steps. Survivors can use this information to smoothly complete the cancellation procedures for various contracts.

[1068] Specific examples

[1069] Example 1: Entering a contract

[1070] User: An elderly person launches the smartphone app and takes a photo of their electricity contract and uploads it.

[1071] Terminal: Sends the contract image to the server.

[1072] Server: The server receives the image and converts it into text data using OCR technology. It then analyzes the contract information and stores it in a database.

[1073] Example 2: Obtaining contract information by a surviving family member

[1074] User (surviving family member): The deceased person's surviving family member logs into the system and checks the cancellation information for the electricity contract.

[1075] Server: Performs identity verification, and if successful, extracts contract information and queries the generation AI model.

[1076] Generative AI model: Generates optimal cancellation details and provides them to the bereaved family, who then follow the cancellation process.

[1077] In this way, the system efficiently manages the contract information of elderly people and helps surviving family members to complete the procedures quickly and smoothly.

[1078] The processing flow will be explained below.

[1079] Step 1:

[1080] The user enters the contract information. Specifically, the user takes a picture of the contract using a device such as a smartphone or PC, or explains the contract details using voice input. The user's device then uploads this input data to the application.

[1081] Step 2:

[1082] The device sends the input data to the server. Specifically, image data or audio data of the photographed contract is sent to the server via the Internet. The data may be encrypted during transmission.

[1083] Step 3:

[1084] The server processes the received data. Specifically, image data is converted to text using OCR technology, and voice data is converted to text using automatic speech recognition (ASR). Through this process, the contents of the contract and the contract contents explained in the voice are extracted as text data.

[1085] Step 4:

[1086] The server analyzes the text data. Specifically, it extracts the necessary contract information (e.g., contract holder name, contract details, contract number, etc.) from the text data and organizes it as contract information. The analysis results are saved in a database.

[1087] Step 5:

[1088] The server provides the contract information to the generative AI model. Specifically, the extracted and organized contract information is passed to the generative AI model, and the model is updated based on that information to generate recommended cancellation procedures.

[1089] Step 6:

[1090] The user (surviving family member) accesses the system. Specifically, the surviving family member logs into the system using a user terminal and inputs that they would like to check the contract information of the deceased person.

[1091] Step 7:

[1092] The server performs identity verification. Specifically, it uses security questions, QR codes, passwords, etc. to verify the identity of the family member. Only if identity verification is successful is access to the contract information granted.

[1093] Step 8:

[1094] The server extracts contract information. Specifically, it searches the database for contract information of the deceased person and extracts the relevant contract information. It then queries the generative AI model to generate cancellation procedure information.

[1095] Step 9:

[1096] The server provides cancellation procedure information, specifically detailed cancellation procedures provided by the generative AI model, as well as necessary documents and contact information, to the bereaved family, helping them to complete the cancellation process smoothly.

[1097] This flow provides a system that enables centralized management of contract information and streamlines cancellation procedures.

[1098] Example 1

[1099] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1100] It is extremely complicated for elderly people to manage a large number of contract information, and the cumbersome procedures become a problem, especially when contract cancellation procedures are required. In addition, in order for surviving family members to smoothly handle the contract information of the deceased, it is necessary to collect appropriate contract information and quickly cancel the contracts, but these procedures are not simple.

[1101] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1102] In this invention, the server includes means for converting input contract information into text data using optical character recognition technology or voice recognition technology, means for analyzing the text data using natural language processing technology to extract contract information and store it in a database, means for verifying the identity of surviving family members who log in from user terminal means using multi-factor authentication, and generation AI model means for providing contract information and details of cancellation procedures to surviving family members. This makes it possible to centrally manage contract information and provide quick and smooth cancellation procedures.

[1103] "Contract information" refers to data related to the contracts and contract contents owned by the user, including the name of the contract holder, contract contents, contract number, etc.

[1104] "User terminal" refers to a device used by a user to enter contract information. This includes smartphones, personal computers, etc.

[1105] "Server" means a computer system that receives, processes, and manages data sent from user terminals.

[1106] Optical character recognition (OCR) is a technology that extracts character information from image data and converts it into text data.

[1107] "ASR (audio recognition)" is a technology that analyzes voice data and converts it into text data.

[1108] "Natural language processing technology (NLP)" is a technology for analyzing text data and extracting specific information.

[1109] "Database" refers to a system for storing and managing extracted contract information.

[1110] "Multi-factor authentication (MFA)" is an authentication method that uses multiple means of verification to authenticate a user.

[1111] A "generative AI model" is a model that uses artificial intelligence to automatically perform specific tasks. In this case, its role is to generate cancellation procedure details based on contract information.

[1112] This invention is a system for centrally managing contract information and smoothly carrying out cancellation procedures. This system consists of a user terminal, a server, and a generative AI model, and is implemented using the following hardware and software.

[1113] Use of user terminals

[1114] First, the user enters contract information using a device such as a smartphone or PC. The user can either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input. This allows the user to easily provide the contract information to the system.

[1115] Server Processing

[1116] The device then sends the captured image or audio data to a server. The server processes the received data. The image data is converted into text using optical character recognition (OCR) technology, and the audio data is similarly converted into text using automatic speech recognition (ASR) technology. To achieve this, services such as Google Cloud Vision API and IBM Watson Speech to Text are used. The converted text data is then analyzed using natural language processing (NLP) technology (such as spaCy or Natural Language Toolkit (NLTK)) to extract the necessary contract information.

[1117] Information storage and analysis

[1118] The server stores the extracted contract information in a database, typically an SQL database (e.g., MySQL or PostgreSQL), and provides the extracted contract information to a generative AI model for future cancellation procedures.

[1119] Access by family members

[1120] When a user dies, the surviving family members access the system. At this stage, the surviving family members log in to the system using the user's device and begin the process of verifying the deceased's contract information. The server then uses a multi-factor authentication (MFA) system (e.g., SMS authentication or email authentication) to verify the identity of the surviving family members. Only if the identity verification is successful will the deceased's contract information be provided.

[1121] Support for cancellation procedures

[1122] The server uses a generative AI model to generate cancellation procedure details based on the contract information. For example, by using OpenAI's GPT series, the server can provide optimal cancellation procedure details to the bereaved family.

[1123] Specific examples

[1124] Entering the contract

[1125] User: An elderly person launches the smartphone app, takes a photo of their electricity contract, and uploads it.

[1126] Terminal: Sends the contract image to the server.

[1127] Server: The server receives the image and converts it into text data using OCR technology (Google Cloud Vision API), then analyzes the contract information using natural language processing (NLP) technology and stores it in an SQL database.

[1128] Acquisition of contract information by surviving family members

[1129] User (surviving family member): The deceased person's surviving family member logs into the system and checks the cancellation information for the electricity contract.

[1130] Server: Performs identity verification, and if successful, extracts contract information and queries the generative AI model (OpenAI GPT).

[1131] Generative AI model: Generates optimal cancellation details and provides them to the bereaved family, who then follow the cancellation process.

[1132] Prompt Sentence Examples

[1133] Please tell me the procedure for canceling my electricity contract.

[1134] How do I cancel a deceased person's mobile phone contract?

[1135] "Please tell me the documents required to cancel the water contract."

[1136] In this way, the purpose of this system is to efficiently manage the contract information of elderly people and to support the surviving family members in completing the procedures quickly and smoothly.

[1137] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1138] Step 1:

[1139] Enter contract information

[1140] Specific operation: The user enters contract information from a smartphone or PC. The user uses a dedicated application to take a photo of a document such as a contract and upload it to the application. Alternatively, the user can dictate the contract details using the voice input function.

[1141] Input: Image data or audio recording of a contract uploaded by the user.

[1142] Output: Image or audio data sent by the device to the server.

[1143] Step 2:

[1144] Data reception and initial processing by the server

[1145] Specific operation: The server receives image data or audio data sent from the user terminal. The received data is temporarily stored in storage.

[1146] Input: Image or audio data sent from the device.

[1147] Output: Data saved in temporary storage.

[1148] Step 3:

[1149] OCR and ASR text conversion

[1150] Specific operation: The server applies optical character recognition (OCR) technology to the received image data and converts it into text data. It also applies automatic speech recognition (ASR) technology to the audio data and converts it into text in the same way. It uses Google Cloud Vision API, IBM Watson Speech to Text, etc.

[1151] Input: Image or audio data stored in temporary storage.

[1152] Output: Contract information converted to text data.

[1153] Step 4:

[1154] Analysis of text data and extraction of contract information

[1155] Specific operation: The server analyzes the text data using natural language processing (NLP) technology to extract necessary contract information such as the contract holder's name, contract details, contract number, etc. This analysis is performed using tools such as spaCy or the Natural Language Toolkit (NLTK).

[1156] Input: Text data.

[1157] Output: The extracted contract information.

[1158] Step 5:

[1159] Saving to a database

[1160] Specific operation: The server stores the extracted contract information in an SQL database (e.g., MySQL or PostgreSQL).

[1161] Input: The extracted contract information.

[1162] Output: Contract information stored in a database.

[1163] Step 6:

[1164] System access and identity verification by family members

[1165] Specific operation: The user (survivor) accesses the system and logs in using the user's device. The server performs multi-factor authentication (MFA) to verify the identity of the surviving family member, and performs SMS authentication and email authentication.

[1166] Input: Family member's credentials.

[1167] Output: Access granted if identity verification is successful.

[1168] Step 7:

[1169] Generative AI model generates cancellation procedures

[1170] How it works: The server sends the contract information to a generative AI model (e.g., OpenAI's GPT series) and inputs a prompt to generate the appropriate cancellation procedure details. The generative AI model then generates the cancellation procedure accordingly.

[1171] Input: Contract information and prompt text stored in the database.

[1172] Output: Detailed steps of the cancellation procedure generated.

[1173] Step 8:

[1174] Provision of cancellation procedure information

[1175] Specific operation: The server analyzes the cancellation procedure information received from the generated AI model and displays the details on the user's device.

[1176] Input: Cancellation process information from the generative AI model.

[1177] Output: Detailed cancellation information displayed on the user's device.

[1178] (Application example 1)

[1179] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1180] There is a lack of means for surviving family members to quickly and accurately grasp the contract information of the deceased and smoothly complete the cancellation procedures. As a result, the procedures take time and often cause stress for surviving family members. In addition, there are limited means for elderly people to properly manage contract information, which runs the risk of the information being lost. It is important to resolve these issues.

[1181] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1182] In this invention, the server includes a user terminal means for inputting contract information, a server means for converting the input contract information into text data using character recognition or voice recognition technology, a means for analyzing the text data to extract the contract information and saving it in a database, a means for verifying the identity of surviving family members registered from the user terminal means when they access the server, a generation AI model means for providing the surviving family members with contract information and a cancellation procedure method, and a means for generating optimal cancellation procedure details based on the contract information saved in the server means and providing them with the necessary documents, contact information, and cancellation procedure. This enables surviving family members to quickly and accurately check the contract information of the deceased person and easily complete the cancellation procedure.

[1183] "Contract information" is information related to the contractor, including the contractor's name, contract details, contract number, and the like.

[1184] The "user terminal means" is a device for a user to input contract information, and refers to an electronic terminal such as a smartphone or a personal computer.

[1185] "Character recognition technology" refers to technology that extracts text data from captured image data, and includes optical character recognition (OCR) technology.

[1186] "Speech recognition technology" refers to technology that extracts text data from voice data, and includes automatic speech recognition (ASR) technology.

[1187] "Server means" refers to the equipment and software used to process, analyze and store input data.

[1188] "Text data" refers to data obtained by converting image data and audio data into character information.

[1189] "Database" refers to the system and storage for systematically storing and managing analyzed contract information.

[1190] "Personal identification means" refers to means for verifying the identity of the user and their family members, and includes authentication information and biometric authentication technology.

[1191] "Generative AI model means" refers to an artificial intelligence model and software that executes it to generate optimal cancellation procedure details based on stored contract information.

[1192] "Contact Information" refers to the contact information required to complete the cancellation procedure, including, for example, contact information for the company's customer support or related departments.

[1193] "Cancellation Procedure" refers to the steps and procedures required to cancel a contract.

[1194] As an embodiment of this invention, we propose a system that centrally manages contract information and smoothly processes cancellation procedures. This system consists of a user terminal, a server, and a generative AI model.

[1195] 1. Enter contract information

[1196] First, the user enters the contract information. Using a device such as a smartphone or PC, the user can either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input. This process uses optical character recognition (OCR) and automatic speech recognition (ASR).

[1197] 2. Data processing by the server

[1198] Next, the image data or audio data captured by the user device is sent to the server. The server processes the received data, converting the image data into text data using optical character recognition (OCR) technology and the audio data into text using automatic speech recognition (ASR) technology. The server then analyzes the text data, extracts necessary information such as the contract holder's name, contract details, and contract number, and stores it in a database.

[1199] 3. Analysis and storage of contract information

[1200] The server analyzes the converted text data and extracts the necessary contract information, which is then stored in a database and provided to the generative AI model for future cancellation procedures.

[1201] 4. User Access and Identity Verification

[1202] When a user dies, the surviving family members access the system. At this stage, the surviving family members log in using the user terminal and begin the process of verifying the contract information of the deceased. The server performs identity verification to verify the identity of the surviving family members, and provides the information only if this is successful.

[1203] 5. Provision of cancellation procedure information

[1204] The server generates optimal cancellation procedure details based on the contract information stored in the generative AI model, and provides the necessary documents, contact information, and specific cancellation procedures to the bereaved family members, allowing them to smoothly complete the cancellation procedures for various contracts according to the information provided.

[1205] Specific examples

[1206] Example 1: Entering a contract

[1207] User: An elderly person launches the smartphone app and takes a photo of their electricity contract and uploads it.

[1208] Terminal: Sends the contract image to the server.

[1209] Server: The server receives the image and converts it into text data using OCR technology. It then analyzes the contract information and stores it in a database.

[1210] Example 2: Obtaining contract information by a surviving family member

[1211] User (surviving family member): The deceased person's surviving family member logs into the system and checks the cancellation information for the electricity contract.

[1212] Server: Performs identity verification, and if successful, extracts contract information and queries the generation AI model.

[1213] Generative AI model: Generates optimal cancellation details and provides them to the bereaved family, who then follow the cancellation process.

[1214] Prompt Sentence Examples

[1215] Example prompt 1: "Contract holder: Taro Yamada, Contract details: Mobile phone service contract. Please tell me the cancellation procedure."

[1216] Example prompt 2: "Contract holder: Ichiro Tanaka, Contract details: Electricity contract. Please tell me the cancellation procedure."

[1217] In this way, the system efficiently manages the contract information of elderly people and helps surviving family members to complete the procedures quickly and smoothly.

[1218] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1219] Step 1:

[1220] Users enter contract information using devices such as smartphones or PCs. Specifically, they either take a photo of a document such as a contract and upload it, or explain the contract details using voice input. The input data in this case is image data or voice data.

[1221] Step 2:

[1222] The device sends the captured image data or audio data to the server, which processes the data.

[1223] Step 3:

[1224] The server converts the received image data into text data using optical character recognition (OCR) technology. For audio data, it converts it into text using automatic speech recognition (ASR). This is the process of converting input data (image data or audio data) into text data.

[1225] Step 4:

[1226] The server analyzes the converted text data and extracts necessary information such as the contract holder's name, contract details, contract number, etc. The contract information obtained through this analysis process is stored in a database. This output data is the contract information text data.

[1227] Step 5:

[1228] When a user dies, the surviving family members access the system. They log in using the user terminal, and the server verifies the identity of the surviving family members. Information is provided only if the conditions for successful identity verification are met. The input data is the login credentials, and the output data is access authentication.

[1229] Step 6:

[1230] The server queries the generative AI model to generate cancellation procedure details based on the contract information stored in the database. The generative AI model generates optimal cancellation procedure details, required documents, contact information, and cancellation procedures based on the provided contract information. The output data obtained from this process is the cancellation procedure details.

[1231] Step 7:

[1232] The server provides the bereaved family with details of the cancellation procedures obtained from the generative AI model. Based on the information provided, the bereaved family can smoothly proceed with the cancellation procedures for various contracts. The output data is the specific cancellation procedures and related information.

[1233] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1234] As an embodiment of the invention, the system below recognizes the user's emotions during the process from entering contract information to the cancellation procedure and adjusts the process accordingly. This system is composed of a user terminal, a server, a generative AI model, and an emotion engine.

[1235] 1. Enter contract information

[1236] First, the user (elderly person) uses a device such as a smartphone or PC to enter contract information. They either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input. The user device then sends the input data to an emotion engine, which analyzes the user's emotional state.

[1237] 2. Emotion analysis using an emotion engine

[1238] The emotion engine analyzes the user's emotional state based on the voice and image data sent by the device. For example, it analyzes the speaking style and tone of voice from the voice data to recognize feelings of relief or anxiety. It also analyzes facial expressions from image data to recognize feelings of happiness or sadness. This emotional information is sent to the server.

[1239] 3. Data processing by the server

[1240] The server processes the received data. Specifically, it converts image data into text using OCR technology and voice data into text using ASR technology. It also takes into account the emotional information obtained from the emotion engine and provides appropriate feedback to the user.

[1241] 4. Analysis and storage of contract information

[1242] The server analyzes the converted text data and extracts the necessary contract information (e.g., contract holder name, contract details, contract number, etc.). The extracted contract information is stored in a database. It also provides an appropriate interface and explanation based on the user's emotional state based on the results of the emotion analysis.

[1243] 5. The server provides the contract information to the generated AI model

[1244] The server provides contract information to the generative AI model and updates the model to generate recommended procedures for contract cancellation. Feedback to users based on emotional information is also optimized.

[1245] 6. Access and identity verification by family members

[1246] When a user passes away, the surviving family members access the system. At this stage, they log in to the system using a user terminal and begin the process of checking the contract information of the deceased. At the same time, an emotion engine runs to analyze the emotional state of the surviving family members and provide appropriate feedback.

[1247] 7. Providing cancellation procedure information to surviving family members

[1248] The server retrieves details of the cancellation procedure from the generative AI model and provides them to the bereaved family after taking into account emotional information, allowing the bereaved family to complete the cancellation procedure smoothly and without stress.

[1249] Specific examples

[1250] Example 1: Entering a contract

[1251] User: An elderly person launches the smartphone app and takes a photo of their electricity contract and uploads it. The emotion engine analyzes the user's facial expression and determines that they are relaxed.

[1252] Terminal: Sends the contract image and emotion information to the server.

[1253] Server: Converts the image into text data using OCR technology, analyzes and saves the contract information, and provides feedback in the usual way since the subject is in a relaxed state.

[1254] Example 2: Obtaining contract information by a surviving family member

[1255] User (surviving family member): The family member of the deceased logs into the system and checks the cancellation information for the electricity contract. The emotion engine analyzes the family member's emotional state and determines that they are experiencing high levels of stress.

[1256] Server: After verifying the identity of the bereaved, the server extracts the contract information and queries the generative AI model. The generative AI model provides details of the cancellation procedure in an easy-to-understand manner, taking into consideration the emotional state of the bereaved.

[1257] In this way, the system combines an emotion engine to optimize the process from entering contract information to the cancellation procedure in a way that responds to the emotional state of the user and their family members, ensuring smooth operation.

[1258] The processing flow will be explained below.

[1259] Step 1:

[1260] The user enters the contract information. Specifically, the user takes a picture of the contract using a device such as a smartphone or PC, or explains the contract details using voice input. The user's device then uploads this input data to the application.

[1261] Step 2:

[1262] The device sends input data to the emotion engine, which receives image and audio data and analyzes the user's emotional state. For example, it recognizes feelings of relief, anxiety, happiness, sadness, etc. from the user's facial expressions and tone of voice.

[1263] Step 3:

[1264] The emotion engine sends the analysis results to the server. Specifically, if the user is relaxed while inputting, emotion data indicating that state is sent to the server.

[1265] Step 4:

[1266] The device sends the input data to the server. Specifically, image data or audio data of the photographed contract is sent to the server via the Internet. The data may be encrypted during transmission.

[1267] Step 5:

[1268] The server processes the received data. Specifically, image data is converted to text using OCR technology, and voice data is converted to text using ASR (audio-speech recognition) technology. Through this process, the contents of the contract and the contract details explained in the voice are extracted as text data.

[1269] Step 6:

[1270] The server analyzes the text data. Specifically, it extracts the necessary contract information (e.g., contract holder name, contract details, contract number, etc.) from the text data and organizes it as contract information. The analysis results are saved in a database.

[1271] Step 7:

[1272] The server provides feedback based on the emotional information: if the user is relaxed, it provides normal feedback, and if the user is anxious, it provides a reassuring message or additional support information.

[1273] Step 8:

[1274] The server provides the contract information to the generative AI model. Specifically, the extracted and organized contract information is passed to the generative AI model, and the model is updated based on that information to generate recommended cancellation procedures.

[1275] Step 9:

[1276] The user (surviving family member) accesses the system. Specifically, the family member logs into the system using a user terminal and requests to check the contract information of the deceased. The emotion engine analyzes the emotional state of the family member and recognizes high levels of stress and sadness.

[1277] Step 10:

[1278] The server performs identity verification. Specifically, it uses security questions, QR codes, passwords, etc. to verify the identity of the family member. Only if identity verification is successful is access to the contract information granted.

[1279] Step 11:

[1280] The server extracts contract information. Specifically, it searches the database for contract information of the deceased person and extracts the relevant contract information. It then queries the generative AI model to generate cancellation procedure information.

[1281] Step 12:

[1282] The server provides cancellation procedure information to the bereaved family. Specifically, it provides detailed cancellation procedure instructions, necessary documents, and contact information provided by the generative AI model. The server takes into consideration the emotional state of the bereaved family and provides the information in an easy-to-understand and stress-reducing format.

[1283] This flow enables centralized management of contract information, streamlines cancellation procedures, and provides support tailored to the emotional state of the user and their family members.

[1284] Example 2

[1285] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1286] Many users, including the elderly, find the process from entering contract information to the cancellation procedure complicated and stressful. The process of entering contract details and then completing the cancellation procedure can be particularly stressful for elderly users. Furthermore, when a user passes away, there are few ways for their surviving family members to smoothly cancel the contract. To address these issues, a system is needed that recognizes the user's emotional state and optimizes the process based on that information.

[1287] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1288] In this invention, the server includes a user terminal means for inputting contract information, an emotion engine means for analyzing emotion data and recognizing the user's emotional state, a server means for converting the input contract information into text data using character recognition or voice recognition technology, and a means for providing appropriate feedback to the user based on the emotion analysis results. This provides an interface and explanations that correspond to the user's emotional state, reducing stress when entering contract information and completing cancellation procedures and enabling smoother procedures. Furthermore, by utilizing emotion analysis, appropriate support can be provided to surviving family members when they check the deceased's contract information and complete cancellation procedures.

[1289] "Contract information" refers to information in which a user provides details regarding the use of a service or product, including the name of the contractor, contract details, contract number, etc.

[1290] "User terminal" refers to an electronic device used by a user to input contract information, and includes smartphones, personal computers, tablets, etc.

[1291] An "emotion engine" refers to software or a system that analyzes audio and image data to recognize a user's emotional state.

[1292] "Character recognition technology" refers to the technology of identifying characters in an image and converting them into text data, and OCR (Optical Character Recognition) is one such technology.

[1293] "Speech recognition technology" refers to the technology that analyzes voice data and converts it into text data, and ASR (Automatic Speech Recognition) is one such technology.

[1294] "Server" refers to a computer system that processes, stores, and operates generative AI models.

[1295] A "generative AI model" refers to an artificial intelligence model that generates recommended procedures or solutions based on given data.

[1296] "Feedback" refers to information and guidance provided to users based on the results of emotion analysis.

[1297] "Identity verification" refers to the process of verifying that a system user is the person registered, and is carried out to protect access rights.

[1298] A "database" refers to a collection of data that organizes and stores contract information and user information so that it can be efficiently accessed as needed.

[1299] This invention provides a system that recognizes the user's emotions throughout the process, from entering contract information to completing cancellation procedures, and adjusts the process accordingly. This system consists of a user terminal, a server, a generative AI model, and an emotion engine. The details and operation of each component are explained below.

[1300] Enter contract information

[1301] Users enter contract information using devices such as smartphones or PCs. They can either take a photo of a document such as a contract and upload it to the application, or explain the contract details using voice input. The user device sends the input data (image data or voice data) to the emotion engine.

[1302] Emotion analysis using an emotion engine

[1303] The device receives voice and image data sent by the user, and the emotion engine analyzes them. In the case of voice data, the emotion engine recognizes feelings of relief or anxiety from the speaking style and tone of voice. In the case of image data, the emotion engine recognizes feelings of happiness or sadness from facial expressions. This emotion information is sent to the server.

[1304] Data processing by the server

[1305] The server processes the data (voice data, image data, and emotional information) received from the emotion engine. Specifically, it converts the voice data into text data using automatic speech recognition (ASR) technology, and converts the image data into text data using optical character recognition (OCR) technology. It also provides appropriate feedback to the user based on the emotional information.

[1306] Analysis and storage of contract information

[1307] The server analyzes the converted text data and extracts necessary contract information such as the contract holder's name, contract details, and contract number. The extracted contract information is stored in a database. Based on the results of the sentiment analysis, the server also provides the user with an appropriate interface and explanation.

[1308] The server provides contract information to the generated AI model

[1309] The server provides the contract information to the generative AI model. The generative AI model updates the model to generate recommended procedures for contract cancellation, and also optimizes feedback to the user based on the generated procedures. For example, it may provide the user with specific procedures such as "Please follow the steps below to cancel the contract."

[1310] Family access and identity verification

[1311] When the deceased passes away, the user (survivor) accesses the system and logs in using a user terminal. At this stage, the system analyzes the emotional state of the bereaved and provides appropriate feedback as needed.

[1312] Providing cancellation procedure information to surviving family members

[1313] The server retrieves details of the cancellation procedure from the generative AI model and provides information while taking into consideration the emotional state of the bereaved, allowing the cancellation procedure to proceed smoothly and without stress for the bereaved.

[1314] Specific examples

[1315] Contract entry: The user (elderly person) launches the smartphone app and takes a photo of their electricity contract and uploads it. The emotion engine analyzes the user's facial expression and determines that they are in a relaxed state. The device then sends the contract image and emotion information to the server, which then converts the image into text data using OCR technology and analyzes and saves the contract information. Because the user is in a relaxed state, feedback is provided using the normal procedure.

[1316] Obtaining contract information by surviving family members: The user (surviving family member) logs in to the system and checks the cancellation information for the deceased person's electricity contract. The emotion engine analyzes the emotional state of the surviving family member and determines that they are experiencing high levels of stress. The server performs identity verification, and if successful, extracts the contract information and queries the generation AI model, which then provides details of the cancellation procedure in an easy-to-understand manner, taking into consideration the emotional state of the surviving family member.

[1317] Example prompts for generative AI models

[1318] "What is the best procedure for a user to cancel their electricity contract?"

[1319] "If the emotion analysis results indicate high stress, please suggest ways to provide appropriate feedback to the bereaved."

[1320] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1321] Step 1:

[1322] The user enters contract information using a smartphone or PC. Specifically, they take a photo of the contract and upload it, or explain the contract details using voice input. The input is image data or voice data of the contract, and the output is the initial stage in which this data is processed on the device.

[1323] Step 2:

[1324] The terminal sends image data or voice data of the contract information to the emotion engine. In the case of voice data, the emotion engine analyzes the speaking style and tone of voice, and in the case of image data, it analyzes facial expressions. The input is the image data or voice data of the contract information, and the output is emotional state information (e.g., relaxed, anxious, happy, sad) as the analysis result.

[1325] Step 3:

[1326] The emotion engine analyzes the user's emotional state and sends the results to the server. For example, it analyzes the tone of voice from audio data to recognize feelings of relief or anxiety, or recognizes facial expressions from image data to identify feelings of happiness or sadness. The input is audio or image data, and the output is the analysis result of the emotional state.

[1327] Step 4:

[1328] The server processes contract information based on the received emotional information. Specifically, it converts voice data into text using automatic speech recognition (ASR) technology, and converts image data into text using optical character recognition (OCR) technology. The input is voice data and image data, and the output is text data.

[1329] Step 5:

[1330] The server analyzes the converted text data and extracts necessary contract information such as the contract holder's name, contract details, and contract number. The extracted contract information is stored in a database. For example, OCR technology can be used to extract the contract holder's name and contract details from image data of a contract. The input is text data, and the output is the extracted contract information.

[1331] Step 6:

[1332] The server provides appropriate feedback to the user based on the emotion analysis results. For example, if the user feels anxious, it provides a detailed and friendly explanation, and if the user feels relaxed, it provides standard procedures. The input is emotion information and contract information, and the output is the feedback content.

[1333] Step 7:

[1334] The server provides the contract information to the generative AI model, which then updates the model to generate recommended procedures for contract cancellation. The input is the contract information, and the output is the generated recommended procedures for cancellation.

[1335] Step 8:

[1336] When the server starts the process for the user (survivor) to confirm the contract information of the deceased, it analyzes the emotional state of the survivor and provides appropriate feedback. At this stage, the server receives input information from the survivor and the emotion engine provides the analysis results. The input is the survivor's login information and emotional information, and the output is appropriate feedback.

[1337] Step 9:

[1338] The server obtains details of the cancellation procedure from the generative AI model and provides information while taking into consideration the emotional state of the bereaved. Specifically, this includes gentle language and detailed explanations based on emotion analysis. The input is the details of the cancellation procedure, which are the output of the generative AI model, and the emotional information of the bereaved, and the output is optimized feedback.

[1339] (Application example 2)

[1340] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1341] Conventional systems for entering contract information and for canceling contracts are unable to provide optimal feedback based on the user's emotional state, making it difficult to alleviate the psychological burden, especially for elderly people and bereaved families. Electronic payment services also face the issue of not responding appropriately to users' emotions, resulting in a decline in the quality of the user experience.

[1342] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1343] In this invention, the server includes emotion engine means for analyzing the user's emotions, user terminal means for inputting contract information, and server means for converting the input contract information into text data using character recognition or voice recognition technology. This makes it possible to provide optimal feedback based on the user's emotions and reduce the psychological burden, particularly on elderly people and bereaved families. Furthermore, this invention can improve the quality of the user experience by providing optimal support based on emotion analysis in electronic payment services.

[1344] The "emotion engine means" is a system for analyzing the user's voice data and image data and recognizing the user's emotional state.

[1345] "User terminal means" refers to a device used by a user to input contract information, and includes a smartphone, a personal computer, etc.

[1346] "Character recognition technology" is a technology that extracts character information from image data and converts it into text data, and OCR (optical character recognition) is a representative example.

[1347] "Speech recognition technology" is a technology that analyzes voice data and converts it into text data, and a representative technology is ASR (automatic speech recognition).

[1348] "Server means" means a computer system that processes data sent from a user terminal and stores and provides necessary information.

[1349] The "generative AI model means" is an artificial intelligence model for generating optimal cancellation procedure steps and feedback based on analyzed contract information.

[1350] The "database" is a system for storing processed contract information and emotion analysis results, and providing information as needed.

[1351] "Electronic payment service" means a service for sending and receiving funds over the Internet.

[1352] "Feedback" refers to providing real-time responses and support information based on the user's emotional state and behavior.

[1353] "Transaction information" refers to specific data related to a transaction, including information such as the amount, currency, and transaction details.

[1354] The system for realizing this invention comprises an emotion engine means, a user terminal means, character recognition technology, voice recognition technology, a server means, a generation AI model means, and a processing means for electronic payment services.

[1355] The user terminal means is used to input contract information, voice data, and image data. For example, a user can use a smartphone or PC to take a picture of the contract and explain the contract contents by voice. This data is transmitted to the server means in real time.

[1356] The server means converts the received image data into text data using OCR (Optical Character Recognition) technology, and converts the voice data into text using ASR (Automatic Speech Recognition) technology. The server also uses an emotion engine means to analyze emotions from the user's voice and facial expressions, and recognizes the user's emotional state.

[1357] The emotion engine means is a system for analyzing the user's emotions based on voice data and image data, and grasps the user's emotional state by analyzing the tone of voice, facial expressions, etc. For example, it can recognize emotions such as joy, sadness, and anxiety.

[1358] In particular, in the case of electronic payment services, appropriate feedback and support can be provided based on the user's emotional state based on the results of emotion analysis. For example, if a user feels anxious during the payment process, additional confirmation messages or support information can be displayed.

[1359] The generative AI model generates optimal steps for the cancellation and settlement procedures based on the analyzed contract information and emotional information, allowing the procedures to proceed smoothly and without stress for the user and their family members.

[1360] As a specific example, consider a case where a user takes a photo of an electricity contract through a smartphone app and uploads it to the app. At this time, the emotion engine means analyzes the user's facial expression and, if it determines that the user is relaxed, provides normal feedback. On the other hand, if it determines that the surviving family members are feeling high stress when checking the contract information of the deceased person, it takes the emotional information into consideration and provides the surviving family members with easy-to-understand information on cancellation procedures.

[1361] The following sentences are used as example prompts:

[1362] "Analyze the emotions users have when they hit the checkout button and provide additional feedback if they are feeling anxious or uncertain."

[1363] This allows for appropriate responses depending on the user's emotional state, improving the overall user experience.

[1364] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1365] Step 1:

[1366] The user terminal means inputs contract information. The user then takes a photo of the contract using a smartphone or PC and explains the contract contents in voice. This data is sent to the server in real time. The inputs are image data and voice data, and the server receives these data as output. Specific operations use the smartphone's camera function and voice recording function.

[1367] Step 2:

[1368] The server converts the received image data into text data using OCR technology, and simultaneously converts the voice data into text using ASR technology. The inputs are image data and voice data, and the output is text data. This data conversion uses OpenCV and Google's voice recognition API.

[1369] Step 3:

[1370] The server analyzes the converted text data, extracts contract information (e.g., contract holder name, contract details, contract number, etc.), and stores it in a database. The input is text data, and the output is contract information stored in the database. Specifically, the text data is analyzed using natural language processing technology.

[1371] Step 4:

[1372] The server uses an emotion engine to analyze the voice and image data sent by the user and recognize the user's emotional state. The input is voice and image data, and the output is emotional information. Specific operations include DeepFace and emotion recognition APIs.

[1373] Step 5:

[1374] The server provides feedback according to the user's emotional state based on the emotional information acquired from the emotion engine means. The emotional information is input, and a feedback message is sent to the user terminal as output. A specific operation is to display a customized message.

[1375] Step 6:

[1376] When a registered family member accesses the system from a user terminal, the server verifies the identity of the person. The input is authentication information, and the output is the authentication result. Specific security methods such as two-factor authentication are used.

[1377] Step 7:

[1378] The generative AI model generates detailed steps for the cancellation procedure based on emotional information and contract information, and provides them to the bereaved family. The inputs are emotional information and contract information, and the cancellation procedure is generated as the output. The specific operation uses natural language processing technology and machine learning models.

[1379] Step 8:

[1380] The server processes electronic payment services, confirms payment based on the results of emotion analysis, and displays additional support messages according to the user's emotional state. The inputs are emotion information and payment information, and the output is the payment result and feedback message. Specifically, communication with the payment gateway is performed using an HTTP request.

[1381] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1382] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1383] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1384] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1385] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1386] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1387] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1388] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1389] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1390] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1391] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1392] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1393] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1394] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1395] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1396] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1397] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1398] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1399] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1400] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1401] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1402] The following is further disclosed regarding the above embodiment.

[1403] (Claim 1)

[1404] a user terminal means for inputting contract information;

[1405] a server means for converting input contract information into text data using character recognition technology or voice recognition technology;

[1406] A means for analyzing the text data to extract contract information and store it in a database;

[1407] means for verifying the identity of a registered bereaved family member when the bereaved family member accesses the system from a user terminal means;

[1408] A generating AI model means for providing contract information and cancellation procedures to the bereaved family;

[1409] A system including:

[1410] (Claim 2)

[1411] 2. The system according to claim 1, further comprising means for transmitting image data or audio data of the contract information to the server means.

[1412] (Claim 3)

[1413] 2. The system of claim 1, wherein the generating AI model means includes means for generating detailed steps of cancellation procedures based on the analyzed contract information.

[1414] "Example 1"

[1415] (Claim 1)

[1416] a user terminal means for inputting contract information;

[1417] a server means for converting input contract information into text data using optical character recognition technology or voice recognition technology;

[1418] A means of analyzing text data using natural language processing technology to extract contract information and store it in a database;

[1419] a means for performing identity verification using multi-factor authentication on a bereaved family member who logs in from a user terminal means;

[1420] a generative AI model means for providing the bereaved with policy information and cancellation details;

[1421] A system including:

[1422] (Claim 2)

[1423] 2. The system according to claim 1, further comprising means for transmitting image data or audio data of the contract information to the server means.

[1424] (Claim 3)

[1425] 2. The system of claim 1, wherein the generating AI model means includes means for generating detailed steps of cancellation procedures based on the analyzed contract information.

[1426] "Application Example 1"

[1427] (Claim 1)

[1428] a user terminal means for inputting contract information;

[1429] a server means for converting input contract information into text data using character recognition technology or voice recognition technology;

[1430] A means for analyzing the text data to extract contract information and store it in a database;

[1431] means for verifying the identity of a registered bereaved family member when the bereaved family member accesses the system from a user terminal means;

[1432] A generating AI model means for providing contract information and cancellation procedures to the bereaved family;

[1433] A means for generating optimal cancellation procedure details based on contract information stored in the server means and providing necessary documents, contact information and cancellation procedures;

[1434] A system including:

[1435] (Claim 2)

[1436] 2. The system according to claim 1, further comprising means for transmitting image data or audio data of the contract information to the server means.

[1437] (Claim 3)

[1438] The system of claim 1, wherein the generating AI model means includes means for generating detailed steps for cancellation procedures based on the analyzed contract information and providing the steps to the user.

[1439] "Example 2: Combining Emotion Engines"

[1440] (Claim 1)

[1441] a user terminal means for inputting contract information;

[1442] emotion engine means for analyzing emotion data and recognizing an emotional state of a user;

[1443] a server means for converting input contract information into text data using character recognition technology or voice recognition technology;

[1444] means for providing appropriate feedback to the user based on the emotion analysis results;

[1445] A means for analyzing the text data to extract contract information and store it in a database;

[1446] means for verifying the identity of a registered bereaved family member when the bereaved family member accesses the system from a user terminal means;

[1447] A generating AI model means for providing contract information and cancellation procedures to the bereaved family;

[1448] A system including:

[1449] (Claim 2)

[1450] 2. The system according to claim 1, further comprising means for transmitting image data or audio data of the contract information to the server means.

[1451] (Claim 3)

[1452] 2. The system of claim 1, wherein the generating AI model means includes means for generating detailed steps for cancellation procedures based on the analyzed contract information and emotional state.

[1453] "Application example 2 when combining emotion engines"

[1454] (Claim 1)

[1455] emotion engine means for analyzing the emotion of a user;

[1456] a user terminal means for inputting contract information;

[1457] a server means for converting input contract information into text data using character recognition technology or voice recognition technology;

[1458] A means for analyzing the text data to extract contract information and store it in a database;

[1459] means for providing feedback according to the emotional state of the user based on the emotional information acquired from the emotion engine means;

[1460] means for verifying the identity of a registered bereaved family member when the bereaved family member accesses the system from a user terminal means;

[1461] A generating AI model means for providing contract information and cancellation procedures to the bereaved family;

[1462] means for processing electronic payment services;

[1463] A system including:

[1464] (Claim 2)

[1465] 2. The system according to claim 1, further comprising means for transmitting image data or audio data of the contract information to the server means.

[1466] (Claim 3)

[1467] 2. The system of claim 1, wherein the generating AI model means includes means for generating detailed steps of cancellation procedures based on the analyzed contract information. [Explanation of symbols]

[1468] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a user terminal means for inputting contract information; a server means for converting input contract information into text data using character recognition technology or voice recognition technology; A means for analyzing the text data to extract contract information and store it in a database; means for verifying the identity of a registered bereaved family member when the bereaved family member accesses the system from a user terminal means; A generating AI model means for providing contract information and cancellation procedures to the bereaved family; A system including:

2. 2. The system according to claim 1, further comprising means for transmitting image data or audio data of the contract information to the server means.

3. 2. The system of claim 1, wherein the generating AI model means includes means for generating detailed steps of cancellation procedures based on the analyzed contract information.

Images