System

Abstract

A system is provided.SOLUTION: The system includes a means for uploading an image of a contract or a use agreement, a means for converting the uploaded image into text data by using optical character recognition technology, a means for extracting an important keyword from the converted text data, a means for evaluating a risk of giving a disadvantage to a user on the basis of the extracted keyword, and a means for notifying a user of an evaluation result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Contracts and terms of use tend to be long and complex, leading many users to enter into them without thoroughly checking the details. As a result, users are at a higher risk of agreeing to contracts that may be detrimental to them. Furthermore, users need specialized knowledge to check the contract contents, which also places a significant burden on users. The challenge is to provide a means to solve these problems, allowing users to easily and quickly understand the contents of contracts and reduce the risk of suffering disadvantages. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems with a system that includes a means for uploading an image of a contract or terms of use, a means for converting the uploaded image into text data using optical character recognition technology, a means for extracting important keywords from the converted text data, a means for assessing the risk of harm to the user based on the extracted keywords, and a means for notifying the user of the assessment results. In particular, the extracted keywords are evaluated using natural language processing technology to improve the accuracy of the risk assessment, thereby reducing the risk of harm to the user. Furthermore, by including a means for scraping the contents of the contract or terms of use from a URL to obtain text data, the system can also handle online contracts and terms of use. This allows users to quickly assess the risks of the contract contents simply by uploading an image or URL.

[0006] A "contract" is a legally binding document agreed upon by two or more parties.

[0007] "Terms of Use" means a document between a service provider and a user that defines the rights and obligations regarding the use of the service.

[0008] "Images" means any visual data format, such as a photograph or screenshot, that contains the content of the Agreement or Terms of Use.

[0009] "Upload" refers to the act of a user transferring data from their own device to a server.

[0010] Optical character recognition (OCR) is a technology that analyzes characters in an image and converts them into text data.

[0011] "Text data" refers to character string data that can be handled by a computer, including the contents of contracts and terms of use.

[0012] "Keywords" refer to words or phrases that have particular significance in contracts or terms of use and are subject to risk assessment.

[0013] "Extraction" refers to the act of extracting specific information from a vast amount of data.

[0014] "Risk" refers to any disadvantage or danger that the User may incur in fulfilling the Agreement.

[0015] "Evaluation" refers to the act of analyzing the extracted keywords and clauses to determine the degree of risk they pose to the user.

[0016] "Notification" refers to the act of informing the user of the evaluation results, which are provided in a format that is easy for the user to understand.

[0017] A "server" refers to a computer system that sends, receives, and processes data over a network.

[0018] "Terminal" refers to a device (smartphone, PC, etc.) that is directly operated by the user.

[0019] "Natural language processing technology (NLP)" refers to the technology that allows computers to understand and analyze human language.

[0020] "Scraping" refers to the technique of extracting data from websites. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] This invention is a system that makes contracts and terms of use easier for users to understand, reducing the risk of signing a disadvantageous contract. Next, we will create a program for this system and explain the program's processing in natural language.

[0043] First, users take a photo of the contract or terms of use or prepare a screenshot of an existing one, then upload the image to a dedicated application. Users can also upload online contracts or terms of use by entering the URL into the same application.

[0044] The device then sends the uploaded image or URL to a server, which uses optical character recognition (OCR) technology to extract character information from the image data and converts it into text data. If a URL is sent, the server uses web scraping technology to retrieve the text of the contract or terms of use from the webpage.

[0045] The converted text data is then analyzed by the server, which uses natural language processing (NLP) technology to extract important keywords and phrases from the text of the contract or terms of use. These keywords include words related to risk, such as "penalty fee" and "forced termination." Based on these keywords, the server evaluates whether the contract contents may be disadvantageous to the user.

[0046] The server sends the details of the assessed risks and the results of the assessment to the device. The device then notifies the user of the assessment results and highlights any parts of the contract or terms of use that require particular attention. Detailed risk assessment comments and advice are also displayed, allowing the user to specifically understand the contract contents.

[0047] As a concrete example, a user takes a photo of the terms of use for a new service with their smartphone and uploads the image to an app. The device then sends the image to a server, which extracts text from the image. The server then analyzes the extracted text for important keywords such as "penalty fee" and "service suspension," and evaluates the risks these keywords pose to the user. The evaluation results are then sent to the device, which then displays a warning to the user, such as "This contract has a high penalty fee. Please be careful."

[0048] In this way, users can quickly understand the contents of contracts and terms of use, reducing the risk of suffering disadvantages. This system significantly simplifies the process of checking complex contracts and terms of use, improving users' peace of mind.

[0049] The processing flow will be explained below.

[0050] Step 1:

[0051] Users can take a picture of the contract or terms of use, or upload an existing screenshot / URL to the application.

[0052] Step 2:

[0053] The device sends the uploaded image and URL to the server.

[0054] Step 3:

[0055] The server performs an optical character recognition (OCR) process on the received image data to convert the characters in the image into text data, or, if a URL is provided, uses web scraping techniques to retrieve the content of the web page and convert it into text data.

[0056] Step 4:

[0057] The server pre-processes the retrieved text data, removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs.

[0058] Step 5:

[0059] The server uses a predefined list of risk keywords to extract important keywords and phrases from the text data, including "penalty fee," "forced termination," and "venue."

[0060] Step 6:

[0061] The server uses natural language processing (NLP) technology to perform a risk assessment of clauses containing the extracted keywords, using an AI-powered predictive model to assign a risk score to each clause.

[0062] Step 7:

[0063] The server generates a risk report for the user based on the AI ​​risk assessment results. High-risk clauses are highlighted, and an assessment score and advice are attached. A warning message based on the overall risk status is also generated.

[0064] Step 8:

[0065] The server sends the generated risk report to the terminal.

[0066] Step 9:

[0067] The terminal displays the risk report received from the server to the user. High-risk clauses are highlighted in red, and specific risk details and warnings are displayed as pop-ups.

[0068] Example 1

[0069] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0070] Conventional contracts and terms of use are often complex and full of technical terms, making them difficult for average users to understand. This puts users at risk of overlooking content that could be detrimental to them. Furthermore, reviewing contracts and terms of use is time-consuming and inefficient. This invention aims to enable users to quickly and easily understand the contents of contracts and terms of use, thereby reducing the risk of suffering disadvantages.

[0071] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0072] In this invention, the server includes means for uploading a digital image of a contract or terms of use, means for converting the uploaded digital image into character string data using optical character recognition technology, means for extracting important keywords from the character string data, means for analyzing the extracted keywords using natural language processing technology and evaluating the risk of causing harm to the user, and means for notifying the user of the evaluation result and visually highlighting parts requiring particular attention. This enables the user to quickly understand the contents of the contract or terms of use and efficiently identify any disadvantageous conditions.

[0073] "Agreement or Terms of Use" means the legal document to which a User agrees when using a Service or purchasing a Product.

[0074] "Digital image" refers to image data obtained by photographing the contents of a contract or terms of use with a camera or scanner.

[0075] "Optical character recognition technology" is a technology that reads character information from digital images and converts it into text data.

[0076] "String data" means data that expresses the contents of a contract or terms of use in text format, converted using optical character recognition technology.

[0077] "Keywords" are words or phrases that are considered important to users in analyzing the content of a contract or terms of use.

[0078] "Natural language processing technology" refers to information processing technology used to understand and analyze human language.

[0079] A "generative AI model" is a model that uses artificial intelligence techniques to learn from training data and perform specific tasks.

[0080] A "web resource identifier" is an address (such as a URL) that uniquely identifies content on the Internet.

[0081] The "evaluation result" refers to the result of the risk assessment obtained by the server by analyzing keywords and text data.

[0082] A "warning message" is a message generated to notify a user of a risk based on the evaluation results.

[0083] "Visual emphasis" refers to the means of highlighting specific text or areas to help users intuitively understand important information.

[0084] This invention is a system that makes it easier for users to understand the contents of contracts and terms of use. In order to reduce the risk of users signing unfavorable contracts, the system analyzes uploaded contract and terms of use data and performs a series of processes to evaluate the risk.

[0085] Users can take a digital image of the contract or terms of use, or prepare an existing screenshot, or if the contract or terms of use is online, they can upload it by entering the URL into a dedicated application.

[0086] The device sends image data and URLs uploaded by the user to the server. The server applies optical character recognition (OCR) technology to the received image data and converts it into text data. Possible OCR technologies include Google® Cloud Vision and Tesseract. When a URL is sent, the server uses web scraping technology to retrieve the text of the contract or terms of use from the web page. This process uses BeautifulSoup and Selenium.

[0087] The server then analyzes the converted text data and uses natural language processing (NLP) techniques to extract important keywords and phrases from the text of the contract or terms of use. SpaCy and NLTK are used as NLP tools. These keywords include risk-related words such as "penalty fee" and "forced termination." The server then uses a generative AI model to perform a risk assessment based on the extracted keywords. The generative AI model is fed the analyzed text data and keywords along with a prompt like the following:

[0088] "This agreement contains the following keywords: penalty, termination. Please assess the risk these pose to you."

[0089] The server sends the evaluation results to the device. The device then notifies the user based on the evaluation results and visually highlights any areas requiring particular attention. For example, it highlights specific contract terms or comments regarding risks and displays a warning message to the user. This allows the user to quickly understand the contents of the contract or terms of use and reduce the risk of suffering disadvantages.

[0090] A specific example is a process in which a user takes a photo of the terms of use for a new service with their smartphone and uploads the image to an app. In this case, the device sends the image to a server, which extracts text data from the image. The server analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk they pose to the user. The evaluation results are sent to the device, which then displays a warning to the user, such as "This contract has a high penalty fee. Please be careful."

[0091] As described above, the system of the present invention allows users to understand the contents of complex contracts and terms of use in a short period of time, thereby improving the user's sense of security.

[0092] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0093] Step 1:

[0094] The user prepares digital images of the contract and terms of use.

[0095] Users can take a picture of the contract or terms of use using their smartphone's camera or scanner, or they can use an existing screenshot.If the contract or terms of use is online, they can also enter its URL into a dedicated application.

[0096] Input: Digital image or URL of the contract or terms of use

[0097] Output: Image file or URL of the contract or terms of use

[0098] Specific behavior:

[0099] The user takes a photo of the contract with their smartphone camera, saves the image, and then copies the URL of the online terms of use and pastes it into a dedicated application.

[0100] Step 2:

[0101] The device sends image data and URLs to the server.

[0102] The device sends the digital images and URLs uploaded by the user to the server.

[0103] Transfer this data to the system's API using an HTTP POST request.

[0104] Input: Image file or URL of the contract or terms of use

[0105] Output: Image file or URL sent to the server

[0106] Specific behavior:

[0107] The device sends a POST request to http: / / example.com / api / upload with the image file selected by the user, or sends a URL entered by the user to the same endpoint.

[0108] Step 3:

[0109] The server converts the image data into character string data.

[0110] The server applies optical character recognition (OCR) technology to the received image data, converting the text information in the image into string data. The technologies used are Google Cloud Vision and Tesseract. When a URL is submitted, the server uses web scraping technology, such as BeautifulSoup and Selenium, to extract the text of the contract or terms of use.

[0111] Input: Image file or URL

[0112] Output: Converted string data

[0113] Specific behavior:

[0114] The server sends the image to the Google Cloud Vision API for OCR processing, and also uses Selenium to access the URL and retrieve the contract and terms of use text.

[0115] Step 4:

[0116] The server analyzes the string data and extracts important keywords.

[0117] The server then analyzes the converted text data and extracts important keywords and phrases using natural language processing (NLP) techniques, such as SpaCy and NLTK.

[0118] Input: Converted string data

[0119] Output: Extracted important keywords

[0120] Specific behavior:

[0121] The server analyzes the string data using SpaCy to identify risk-related keywords (e.g., penalty fees, forced termination), and generates a list of important keywords as a result of the analysis.

[0122] Step 5:

[0123] The server performs risk assessment using a generative AI model

[0124] The server performs risk assessment using a generative AI model based on the extracted keywords. It passes input data along with prompt statements to the model and obtains the assessment results.

[0125] Input: Extracted important keywords, prompt sentence

[0126] Output: Risk assessment results

[0127] Specific behavior:

[0128] The server inputs the following prompt to the generative AI model: "This contract contains the following keywords: penalty fee, forced termination. Please evaluate what risks these pose to the user." The server then obtains the evaluation result returned by the AI ​​model.

[0129] Step 6:

[0130] The server sends the evaluation results to the terminal.

[0131] The server transmits the generated risk assessment results to the terminal.

[0132] Input: Risk assessment results

[0133] Output: Evaluation results sent to the device

[0134] Specific behavior:

[0135] The server packages the risk assessment results in JSON format and sends them to the terminal using the sendResponse method.

[0136] Step 7:

[0137] The device notifies the user of the evaluation results and highlights them visually.

[0138] The device notifies the user based on the evaluation results received from the server, visually highlighting any parts of the evaluation results that require special attention.

[0139] Input: Risk assessment results

[0140] Output: Evaluation results notified to the user, with visually highlighted notes

[0141] Specific behavior:

[0142] The device displays the evaluation results on the user interface, highlights specific keywords (e.g., penalty fees), and issues a warning to the user via a pop-up message.

[0143] Through the above processing steps, the system quickly and accurately analyzes the contents of contracts and terms of use, and provides users with a risk assessment in an easy-to-understand format.

[0144] (Application example 1)

[0145] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0146] There is a need to reduce the risk of users signing contracts and terms of use without fully understanding them. However, especially in the case of electronic payment services, terms of use are often complex and lengthy, making them difficult for users to understand. As a result, users may suffer disadvantages. To solve this problem, a system is needed that allows users to intuitively understand the contents of the contract and immediately notifies them of particularly high-risk parts.

[0147] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0148] In this invention, the server includes means for uploading an image of the contract or terms of use, means for converting the uploaded image into text data using optical character recognition technology, means for extracting important keywords from the converted text data, means for evaluating the risk of harm to the user based on the extracted keywords, means for notifying the user of the evaluation result, means for notifying the user of the evaluation result in real time, and means for integrating the harm risk evaluation into the electronic payment environment. This allows the user to quickly understand the risks in the contract content and reduce the risk of suffering harm.

[0149] "Agreement or Terms of Use" means a document between a user and a provider regarding the use of a service or purchase of a product, which defines the terms of use and rights and obligations.

[0150] "Images" are visual information converted into digital data, and refer to the contents of contracts and terms of use as photographs or scanned data.

[0151] "Uploading" is the operation by which a user sends digital data to a server or cloud service via the Internet.

[0152] Optical character recognition (OCR) is a technology that reads character information contained in image data and converts it into text data.

[0153] "Text data" is digital information written in text format, and is data in a form that is easy to search and analyze.

[0154] "Important keywords" are words or phrases in the contract that may have a significant impact on the user.

[0155] "Extraction" is the operation of extracting necessary information or elements from text data.

[0156] "Risk assessment" is a process of analyzing and determining the possibility that a user may suffer a disadvantage based on the extracted keywords.

[0157] "Real-time notification" refers to providing analysis results immediately after the user provides data.

[0158] "Integration into the electronic payment environment" refers to a system that automatically incorporates risk analysis functions when conducting online financial transactions and promptly notifies users.

[0159] This invention is a system that enables users to quickly understand the contents of contracts and terms of use, reducing the risk of suffering disadvantages. Specifically, the system uploads images or URLs of contracts and terms of use, converts them into text data, and analyzes them using natural language processing technology.

[0160] System Configuration

[0161] Hardware: Smartphones, smart glasses, servers

[0162] Software: OCR technology (e.g., Tesseract), web scraping tools (e.g., BeautifulSoup), natural language processing (e.g., spaCy)

[0163] Operation overview

[0164] 1. Obtaining contract and terms of use data:

[0165] The user takes a picture of the contract or terms of use using a smartphone or smart glasses and uploads it to the application.

[0166] The user enters the URL of the online contract or terms of use into the application and uploads it.

[0167] The terminal transmits the input image or URL to the server.

[0168] 2. Data conversion and text generation:

[0169] The server converts the uploaded image into text data using optical character recognition (OCR) technology, which uses Tesseract.

[0170] When the URL is uploaded, the server uses a web scraping tool to extract the text data of the contract or terms of use from the web page. For example, we use BeautifulSoup.

[0171] 3. Natural Language Processing and Risk Assessment:

[0172] The server then uses natural language processing (NLP) techniques on the converted text data to extract important keywords and phrases. This process is performed using spaCy.

[0173] Based on the extracted keywords, the risk that the contract contents will cause disadvantage to the user is evaluated.

[0174] Based on the results of the risk assessment, necessary precautions and warnings are generated.

[0175] 4. Real-time notifications:

[0176] The server transmits the evaluation results to the terminal in real time.

[0177] The device notifies the user of the evaluation results and highlights areas requiring attention.

[0178] Usage example

[0179] The user takes a photo of the terms of use for a new service with their smartphone and uploads the image to the application. The device sends the image to the server, which extracts text from the image. The server analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk these keywords pose to the user. The evaluation results are sent to the device, which then displays a warning to the user, such as "This contract has a high penalty fee. Please be careful."

[0180] Prompt Sentence Examples

[0181] "Do the new terms of service agreements include penalties? And what are the risks to users?"

[0182] By using such a system, users can quickly understand the contents of the contract and reduce the risk of suffering disadvantages.

[0183] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0184] Step 1:

[0185] Users upload images of contracts and terms of use

[0186] The user takes an image of the contract or terms of use using a smartphone or smart glasses. Then, the user uploads the image or screenshot to the application. The uploaded image is sent to the server.

[0187] Input: Image of contract or terms of use

[0188] Output: Image data sent to the server

[0189] Step 2:

[0190] User uploads URL of contract or terms of use

[0191] Users enter the URL of the online contract or terms of use into a dedicated application and upload it. The entered URL is sent to the server.

[0192] Input: URL of contract or terms of use

[0193] Output: URL data sent to the server

[0194] Step 3:

[0195] The server converts image data into text data

[0196] The server uses optical character recognition (OCR) technology to extract character information from uploaded images and convert them into text data. The OCR technology used is Tesseract.

[0197] Input: Uploaded image data

[0198] Output: Text data

[0199] Step 4:

[0200] The server scrapes the web page content from the URL.

[0201] The server retrieves the contract and terms of use from the uploaded URL using web scraping technology, which uses BeautifulSoup to extract the text from the web page.

[0202] Input: Uploaded URL

[0203] Output: Text data

[0204] Step 5:

[0205] The server uses natural language processing technology to extract important keywords

[0206] The server then uses natural language processing (NLP) technology on the converted or scraped text data to extract important keywords and phrases. The NLP technology used is spaCy.

[0207] Input: Text data

[0208] Output: Extracted keywords and phrases

[0209] Step 6:

[0210] The server assesses the risk

[0211] The server uses the extracted keywords and phrases to assess the risk that the contract content will cause disadvantage to the user, using a generative AI model for the assessment.

[0212] Input: Extracted keywords or phrases

[0213] Output: Risk assessment results

[0214] Step 7:

[0215] The server notifies the evaluation results in real time

[0216] The server transmits the risk assessment results in real time to the user's terminal, which then notifies the user of the results.

[0217] Input: Risk assessment results

[0218] Output: The evaluation results displayed to the user

[0219] Step 8:

[0220] The device displays risk information to the user

[0221] Based on the evaluation results received, the device highlights and displays parts of the contract or terms of use that require particular attention, and also provides risk assessment comments and advice.

[0222] Input: Risk assessment results

[0223] Output: Highlighted contracts and terms of use, risk assessment comments

[0224] Prompt Sentence Examples

[0225] "Do the new terms of service agreements include penalties? And what are the risks to users?"

[0226] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0227] This invention is a system that makes contracts and terms of use easier for users to understand, reducing the risk of signing a disadvantageous contract, and also combines it with an emotion engine that recognizes the user's emotions and optimizes the information provided based on those emotions. Below, we will explain the program of this system in detail, and explain each processing step in natural language.

[0228] First, the user takes a picture of the contract or terms of use, or uploads an existing screenshot or URL to the application. The device then sends the image or URL to the server. The server then performs optical character recognition (OCR) on the received image data and converts the characters in the image into text data. If a URL is provided, the server uses web scraping technology to convert the contents of the web page into text data.

[0229] The server then pre-processes the retrieved text data, removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs. The server then uses a pre-defined risk keyword list to extract important keywords and phrases from the text data, including risk-related words such as "penalty" and "forced termination."

[0230] The server then uses natural language processing (NLP) technology to perform a risk assessment of clauses containing the extracted keywords. An AI-based predictive model is used for the risk assessment, and a risk score is assigned to each clause. The system also incorporates an emotion engine that can recognize the user's emotions. The emotion engine analyzes emotions from the user's voice, facial expressions, and input text to determine how the user feels about the contract or terms of use.

[0231] The emotion information recognized by the emotion engine is reflected in the assessment results and notification content by the server. For example, if the user expresses anxiety or doubt, the server can provide additional advice or detailed explanations. The server can also adjust the display format of the risk assessment and the tone of the notification message based on the emotion information.

[0232] Based on the evaluation results and sentiment information, the server generates a risk report for the user. High-risk clauses are highlighted and an evaluation score and advice are attached. A warning message based on the overall risk status is also generated. The server sends the generated risk report to the terminal, which displays it to the user.

[0233] As a concrete example, a user takes a photo of the terms of use for a new service and uploads the image to the app. The device sends the image to a server, which extracts text from the image. The server analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk these pose to the user. The emotion engine recognizes anxious emotions from the user's facial expressions and voice, and the server displays a message warning them that "This contract has a high penalty fee. Please be careful," along with suggestions for additional details or support.

[0234] In this way, users can quickly understand the contents of contracts and terms of use, reducing the risk of suffering disadvantages. Furthermore, the introduction of an emotion engine provides optimal advice and support based on the user's emotions, allowing users to review contract details with greater peace of mind. This system significantly simplifies the process of reviewing complex contracts and terms of use, increasing users' sense of security.

[0235] The processing flow will be explained below.

[0236] Step 1:

[0237] Users can take a picture of the contract or terms of use, or upload an existing screenshot / URL to the application using a smartphone or tablet.

[0238] Step 2:

[0239] The device sends the uploaded image and URL to the server, where the data is encrypted and securely transferred to the server.

[0240] Step 3:

[0241] The server uses optical character recognition (OCR) technology to convert the characters in the image into text data, and if a URL is provided, uses web scraping technology to retrieve the text of the contract or terms of use from the web page.

[0242] Step 4:

[0243] The server preprocesses the converted text data, specifically removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs. This step improves the quality of the text data.

[0244] Step 5:

[0245] The server extracts important keywords and phrases from the text data, using a predefined list of risk keywords to detect specific words (e.g., "penalty fee," "forced termination," "venue," etc.).

[0246] Step 6:

[0247] The server uses natural language processing (NLP) techniques to perform a risk assessment on clauses containing the extracted keywords, using an AI model to assign a risk score to each clause.

[0248] Step 7:

[0249] The server recognizes the user's emotions through an emotion engine, analyzing facial expressions, voice data, and input text when the user operates the application. Emotions are classified as anxiety, doubt, relief, etc.

[0250] Step 8:

[0251] The server adjusts the evaluation results and notification content based on the emotional information recognized by the emotion engine. For example, if the user expresses anxiety, it generates a notification with a detailed explanation and additional support information.

[0252] Step 9:

[0253] The server generates a risk report based on the evaluation results and sentiment information. High-risk clauses are highlighted and accompanied by an evaluation score and advice. Warning messages are also generated based on the overall risk situation.

[0254] Step 10:

[0255] The server sends the generated risk report to the terminal.

[0256] Step 11:

[0257] The terminal displays the risk report received from the server to the user. High-risk clauses are highlighted in red, and specific risk details and warnings are displayed as pop-ups.

[0258] This process allows users to quickly understand the contents of the contract and terms of use, reducing the risk of suffering disadvantages. Furthermore, appropriate advice and support tailored to the user's feelings are provided, allowing users to review the contract contents with greater peace of mind.

[0259] Example 2

[0260] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0261] Conventional contracts and terms of use are difficult for ordinary users to understand, increasing the risk of them signing disadvantageous contracts. Furthermore, because they do not take into account the emotions users feel toward these documents, necessary support and advice are lacking. Therefore, there is a need for a system that makes contracts and terms of use easier to understand and provides appropriate information that takes users' emotions into account.

[0262] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0263] In this invention, the server includes means for uploading an image of a contract or terms of use, means for converting the uploaded image into text data using optical character recognition technology, means for extracting important keywords from the converted text data, means for evaluating the risk of causing harm to the user based on the extracted keywords, means for analyzing emotions from the user's voice, facial expression, input text, etc., means for adjusting the evaluation result and notification content based on the analyzed emotional information, and means for notifying the user of the evaluation result. This makes it possible to objectively evaluate the risks of the contract or terms of use and provide appropriate information and support according to the user's emotions.

[0264] "Agreement" or "Terms of Use" means a document that contains the formal terms and conditions governing the use of a service and that clarifies the rights and obligations of both parties.

[0265] "Means for uploading images" is a function that allows a user to use a terminal to send image data of a contract or terms of use to a server.

[0266] "Optical character recognition technology" refers to the technology that analyzes characters in image data and converts them into text data.

[0267] The "means for converting into text data" is a function that uses optical character recognition technology to extract character information from image data and convert it into digital text format.

[0268] "Means for extracting important keywords" refers to the ability to identify and extract words and phrases related to specific risks or important matters from text data.

[0269] The "means for assessing the risk of causing harm" is a function that analyzes and assesses the risks that the clauses of a contract or terms of use may pose to the user, based on the extracted keywords.

[0270] "Means for analyzing emotions from a user's voice, facial expressions, input text, etc." refers to technology that analyzes audio, video, and text data collected through input devices such as microphones and cameras to identify the user's emotional state.

[0271] The "means for adjusting evaluation results and notification content" is a function for adjusting the content and tone of evaluation results and notification messages provided to users based on the results of emotion analysis.

[0272] The "means for notifying the user of the evaluation results" is a function for sending the results of risk evaluation and advice to the terminal and displaying them to the user.

[0273] "Method of scraping the contents of a contract or terms of use from a URL to obtain text data" refers to a technology that analyzes the contents of a web page and extracts the text data of a contract or terms of use.

[0274] "Natural language processing technology" is a technology that allows computers to understand and analyze human language, and involves classifying text data and performing semantic analysis.

[0275] This invention is a system that makes contracts and terms of use easier for users to understand, reducing the risk of signing a disadvantageous contract, and also combines an emotion engine that recognizes the user's emotions and optimizes the information provided accordingly. The program processing of this system is explained in detail below.

[0276] First, the user takes a picture of the contract or terms of use, or uploads an existing screenshot or URL to the application. This is done using a device such as a smartphone or tablet. The device then sends the image and URL data to the server.

[0277] The server applies optical character recognition (OCR) technology to the received image data to convert the characters in the image into text data using an OCR engine such as Tesseract. If a URL is provided, the server uses a web scraping technology such as BeautifulSoup or Scrapy to convert the contents of the web page into text data.

[0278] The server then pre-processes the retrieved text data, removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs using a text cleaning algorithm based on regular expressions.

[0279] The server then extracts important keywords and phrases from the text data using a predefined list of risk keywords, including risk-related words such as "penalty" and "cancellation." This process is performed using a Python NLP library (e.g., NLTK or spaCy).

[0280] The server then uses natural language processing (NLP) technology to perform a risk assessment of clauses containing the extracted keywords. The risk assessment utilizes AI-based predictive models, such as Tensorflow (registered trademark) and PyTorch, to assign a risk score to each clause.

[0281] The system also incorporates an emotion engine that analyzes emotions from the user's voice, facial expressions, input text, etc. The emotion engine uses data collected from the webcam and smartphone microphone to recognize emotions using OpenCV and the Google Cloud Speech-to-Text API.

[0282] The emotion information recognized by the emotion engine is reflected in the assessment results and notification content by the server. For example, if the user expresses anxiety or doubt, the server will provide additional advice or detailed explanations. The server will also adjust the display format of the risk assessment and the tone of the notification message based on the emotion information.

[0283] Based on the evaluation results and sentiment information, the server generates a risk report for the user. High-risk clauses are highlighted and an evaluation score and advice are attached. A warning message based on the overall risk status is also generated. The server sends the generated risk report to the terminal, which displays it to the user.

[0284] As a concrete example, if a user takes a photo of the terms of use for a new service and uploads the image to the app, the system operates as follows:

[0285] 1. The device sends the image to the server.

[0286] 2. The server extracts text from the image using OCR technology.

[0287] 3. The server analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk they pose to the user.

[0288] 4. The emotion engine recognizes anxious emotions from the user's facial expressions and voice.

[0289] 5. The server displays a message warning you that "This contract has a high penalty. Be careful" and offering additional details or support.

[0290] An example of an input prompt for the generative AI model is, "Please analyze the image of the contract below and assess the risks. Also, if the user feels anxious, please provide advice appropriate to their feelings."

[0291] This system helps users understand the contents of contracts and terms of use in a short amount of time, reducing the risk of suffering disadvantages. In addition, the introduction of an emotion engine provides optimal advice and support based on the user's emotions, allowing users to review contract details with greater peace of mind. This greatly simplifies the process of reviewing complex contracts and terms of use, improving users' sense of security.

[0292] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0293] Step 1:

[0294] The user takes a picture of the contract or terms of use, or uploads an existing screenshot or URL to the application. Specifically, when the user launches the app and taps the "Upload contract" button, the camera on the smartphone or tablet is activated. The user then takes a picture of the contract with the camera and presses the "Upload" button. Similarly, in the case of a screenshot or URL, the user is presented with the option to "Select file" or "Enter URL," and selects one of the options.

[0295] Input: Image data, screenshots, or URLs of contracts or terms of use

[0296] Output: Image data and URLs are imported to the device

[0297] Step 2:

[0298] The device sends the captured image or the provided URL to the server. Specifically, the device obtains the image data or URL data captured or uploaded by the user and sends it to the server via the Internet. An HTTP POST request is used to send the data to the endpoint.

[0299] Input: Image data or URL (data imported to the device)

[0300] Output: Image data and URL are sent to the server.

[0301] Step 3:

[0302] The server applies optical character recognition (OCR) technology to the received image data to convert the characters in the image into text data. Specifically, the server runs an OCR engine such as Tesseract to convert the characters in the image data into text format. Also, if a URL is provided, the server uses a web scraping tool such as BeautifulSoup or Scrapy to extract the text data from the page.

[0303] Input: Image data or URL sent

[0304] Output: Extracted text data

[0305] Step 4:

[0306] The server pre-processes the retrieved text data, removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs. Specifically, text cleaning algorithms are run, regular expressions are used to remove whitespace and special characters, and natural line and paragraph breaks are implemented.

[0307] Input: Extracted text data

[0308] Output: Cleaned text data

[0309] Step 5:

[0310] The server extracts important keywords and phrases from the text data using a predefined list of risk keywords. Specifically, the server uses a Python NLP library (e.g., NLTK or spaCy) to compare the list of risk keywords with the text data and perform the keyword extraction process.

[0311] Input: Cleaned text data

[0312] Output: Extracted important keywords and phrases

[0313] Step 6:

[0314] The server performs a risk assessment of clauses containing the extracted keywords. Specifically, an AI predictive model (e.g., TensorFlow or PyTorch) analyzes text containing risk keywords and calculates and assigns a risk score to each clause. The assessment results are stored in a database.

[0315] Input: Extracted important keywords and phrases

[0316] Output: Risk assessment score and assessment result

[0317] Step 7:

[0318] The server is equipped with an emotion engine that analyzes emotions from the user's voice, facial expressions, and input text. Specifically, it collects audio data from the webcam or smartphone microphone and runs an emotion recognition algorithm to analyze it (e.g., OpenCV is used for facial recognition, and Google Cloud Speech-to-Text API is used for voice analysis). The user's emotions are identified and stored in a database.

[0319] Input: Audio data, image data, input text

[0320] Output: Parsed emotion data

[0321] Step 8:

[0322] The server reflects the emotional information recognized by the emotion engine in the evaluation results and notification content. Specifically, it adjusts the tone of the risk assessment and notification message based on the emotional data. For example, for a user who expresses anxiety, it generates a warning message such as, "This contract has a high penalty fee. Please be careful."

[0323] Input: Risk assessment score, parsed sentiment data

[0324] Output: Adjusted evaluation results and notification messages

[0325] Step 9:

[0326] The server generates a risk report based on the evaluation results and emotional information. Specifically, it generates a risk report template that integrates the evaluation scores and emotional data, and creates a report in PDF or HTML format that visually highlights high-risk areas.

[0327] Input: Adjusted evaluation results and notification messages

[0328] Output: Generated risk report

[0329] Step 10:

[0330] The server sends the generated risk report to the terminal, which then displays it to the user. Specifically, the generated risk report is delivered to the terminal and displayed so that the user can view it on the application. There is a notification function to notify the user of the receipt of the risk report.

[0331] Input: Generated risk report

[0332] Output: Risk report displayed to the user

[0333] (Application example 2)

[0334] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0335] In modern digital services, many users agree to contracts and terms of use without properly understanding them. This increases the risk of signing unfavorable contracts. However, the content of the terms of use is complex, requiring time and effort for users to understand them. Furthermore, there is a lack of mechanisms to concretely recognize abstract risks or to support emotional decision-making. Therefore, there is a need for a method to more easily understand the risks in terms of use and contracts and provide emotional feedback.

[0336] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading an image of a contract or terms of use, means for converting the uploaded image into text data using optical character recognition technology, means for extracting important keywords from the converted text data, means for evaluating risks that may cause harm to the user based on the extracted keywords, means for notifying the user of the evaluation results, means for analyzing the user's emotions and optimizing the notification content based on the determined emotions, and means for generating a risk report based on the evaluation results and emotion information and transmitting the report to the terminal. This enables the user to quickly understand the contents of the contract or terms of use and receive optimal advice tailored to their emotions.

[0337] A "contract" is a document that describes the terms and conditions of a legal agreement.

[0338] "Terms of Use" means a document that sets out the conditions and rules for the use of a digital service or product.

[0339] "Means" are the methods or techniques used to achieve a particular goal.

[0340] "Means for uploading images" refers to a function that allows a user to send a specific image file to the system through a device.

[0341] "Optical character recognition technology" is a technology that converts characters in an image into digital text.

[0342] "Text data" is data that represents character information in a digital format.

[0343] "Important keywords" are words that are particularly noteworthy because they indicate specific content or risks.

[0344] "Keyword extraction means" refers to a technique for selecting specific words or phrases from text data.

[0345] The "means for assessing risk" is a method for analyzing the extracted keywords and determining the potential risks associated with them.

[0346] The "means for notifying the user of the evaluation results" is a function for displaying or communicating the results of the risk evaluation to the user.

[0347] "Means for analyzing emotions" refers to technology that recognizes and evaluates emotions from the user's voice, facial expressions, text input, etc.

[0348] "Means for optimizing notification content" refers to a method for adjusting the content and tone of a message based on the user's emotions.

[0349] A "risk report" is a document that compiles assessed risk information and advice.

[0350] "Terminal" refers to a digital device operated by a user.

[0351] The following describes an embodiment of the present invention.

[0352] The server has a function that allows users to upload images of contracts or terms of use to the server using their devices. The system starts when a user takes a photo of a contract and uploads the image using a smartphone or tablet.

[0353] The server then applies optical character recognition (OCR) technology to the uploaded image, converting the textual information in the image into digital text data. The specific software used is "pytesseract." The text extracted from the image is then entered into the system.

[0354] In the stage of extracting important keywords from the converted text data, analysis is performed based on a predefined list of risk keywords. For example, risk-related words such as "penalty" and "forced termination" are listed and extracted from the text data. Natural language processing (NLP) technology is used to achieve this function.

[0355] The server then evaluates the risk of harm to the user based on the extracted keywords. This risk evaluation uses a predictive model using a generative AI model to assign a risk score to each clause, which clarifies the risk level that the user should be aware of.

[0356] After the evaluation results are obtained, a method is used to analyze the user's emotions. This emotion analysis utilizes an emotion engine that analyzes the user's voice and facial expressions. For example, software such as "EmotionRecognition" is used to analyze the user's emotions. The content of notifications is optimized based on the emotional information.

[0357] Finally, a risk report is generated based on the evaluation results and sentiment information. This risk report highlights high-risk clauses and includes an evaluation score and advice. It also generates a warning message based on the overall risk situation. The server sends this risk report to the terminal, which then displays it to the user.

[0358] As a concrete example, consider the case where a user takes a screenshot of an online shopping website's terms of use and uploads the image to an app. The device sends the image to a server, which extracts text from the image. The server then analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk they pose to the user. The emotion engine recognizes anxious emotions from the user's facial expressions and voice, and the server displays a message warning the user that "This contract has a high penalty fee. Please be careful," along with suggestions for additional details or support.

[0359] An example prompt for a generative AI model might look something like this:

[0360] > "Read the user's sentiment and if they are unsure about the contract, generate advice like this:

[0361] "These terms of service have significant penalties. Please see additional details."

[0362] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0363] Step 1:

[0364] A user takes a picture of a contract or terms of use and uploads it to a server using their device. At this point, the input is the captured image file and the output is the image sent to the server.

[0365] Step 2:

[0366] The server applies optical character recognition (OCR) technology to the uploaded image and converts the characters in the image into digital text data. Specifically, it uses the pytesseract library to analyze the image file. The input is the image file, and the output is the converted text data.

[0367] Step 3:

[0368] The server extracts important keywords from the converted text data based on a predefined risk keyword list. It uses NLP technology to identify risk-related words and phrases. The input is the text data converted by OCR, and the output is the extracted keyword list.

[0369] Step 4:

[0370] The server evaluates the risk of posing a risk to the user based on the extracted keywords. It uses a generative AI model to calculate a risk score and determine the risk level of each keyword. The input is the extracted keyword list, and the output is the analysis result with a risk score.

[0371] Step 5:

[0372] When the user reads the contract, the server uses an emotion analysis engine to analyze the user's emotions. Specifically, the server analyzes the user's facial expressions and voice using emotion recognition software (e.g., EmotionRecognition). The input is the user's facial expressions and voice data, and the output is the determined emotional information.

[0373] Step 6:

[0374] The server optimizes the notification content for the user based on the results of risk assessment and sentiment analysis. It uses a generative AI model to generate an appropriate message based on the sentiment. A specific example includes a message such as, "This contract has a high penalty fee. Please be careful." The input is the risk assessment result and sentiment information, and the output is an optimized message.

[0375] Step 7:

[0376] The server generates a risk report based on the evaluation results and sentiment information. The report highlights high-risk clauses and provides an evaluation score and advice. The input is the optimized message and the risk evaluation results, and the output is the generated risk report.

[0377] Step 8:

[0378] The server sends the generated risk report to the terminal, which displays it to the user, allowing the user to easily understand the contents of the contract and its potential risks. The input is the risk report, and the output is a detailed report displayed to the user.

[0379] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0380] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0381] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0382] [Second embodiment]

[0383] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0384] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0385] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0386] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0387] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0388] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0389] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0390] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0391] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0392] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0393] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0394] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0395] This invention is a system that makes contracts and terms of use easier for users to understand, reducing the risk of signing a disadvantageous contract. Next, we will create a program for this system and explain the program's processing in natural language.

[0396] First, users take a photo of the contract or terms of use or prepare a screenshot of an existing one, then upload the image to a dedicated application. Users can also upload online contracts or terms of use by entering the URL into the same application.

[0397] The device then sends the uploaded image or URL to a server, which uses optical character recognition (OCR) technology to extract character information from the image data and converts it into text data. If a URL is sent, the server uses web scraping technology to retrieve the text of the contract or terms of use from the webpage.

[0398] The converted text data is then analyzed by the server, which uses natural language processing (NLP) technology to extract important keywords and phrases from the text of the contract or terms of use. These keywords include words related to risk, such as "penalty fee" and "forced termination." Based on these keywords, the server evaluates whether the contract contents may be disadvantageous to the user.

[0399] The server sends the details of the assessed risks and the results of the assessment to the device. The device then notifies the user of the assessment results and highlights any parts of the contract or terms of use that require particular attention. Detailed risk assessment comments and advice are also displayed, allowing the user to specifically understand the contract contents.

[0400] As a concrete example, a user takes a photo of the terms of use for a new service with their smartphone and uploads the image to an app. The device then sends the image to a server, which extracts text from the image. The server then analyzes the extracted text for important keywords such as "penalty fee" and "service suspension," and evaluates the risks these keywords pose to the user. The evaluation results are then sent to the device, which then displays a warning to the user, such as "This contract has a high penalty fee. Please be careful."

[0401] In this way, users can quickly understand the contents of contracts and terms of use, reducing the risk of suffering disadvantages. This system significantly simplifies the process of checking complex contracts and terms of use, improving users' peace of mind.

[0402] The processing flow will be explained below.

[0403] Step 1:

[0404] Users can take a picture of the contract or terms of use, or upload an existing screenshot / URL to the application.

[0405] Step 2:

[0406] The device sends the uploaded image and URL to the server.

[0407] Step 3:

[0408] The server performs an optical character recognition (OCR) process on the received image data to convert the characters in the image into text data, or, if a URL is provided, uses web scraping techniques to retrieve the content of the web page and convert it into text data.

[0409] Step 4:

[0410] The server pre-processes the retrieved text data, removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs.

[0411] Step 5:

[0412] The server uses a predefined list of risk keywords to extract important keywords and phrases from the text data, including "penalty fee," "forced termination," and "venue."

[0413] Step 6:

[0414] The server uses natural language processing (NLP) technology to perform a risk assessment of clauses containing the extracted keywords, using an AI-powered predictive model to assign a risk score to each clause.

[0415] Step 7:

[0416] The server generates a risk report for the user based on the AI ​​risk assessment results. High-risk clauses are highlighted, and an assessment score and advice are attached. A warning message based on the overall risk status is also generated.

[0417] Step 8:

[0418] The server sends the generated risk report to the terminal.

[0419] Step 9:

[0420] The terminal displays the risk report received from the server to the user. High-risk clauses are highlighted in red, and specific risk details and warnings are displayed as pop-ups.

[0421] Example 1

[0422] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0423] Conventional contracts and terms of use are often complex and full of technical terms, making them difficult for average users to understand. This puts users at risk of overlooking content that could be detrimental to them. Furthermore, reviewing contracts and terms of use is time-consuming and inefficient. This invention aims to enable users to quickly and easily understand the contents of contracts and terms of use, thereby reducing the risk of suffering disadvantages.

[0424] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0425] In this invention, the server includes means for uploading a digital image of a contract or terms of use, means for converting the uploaded digital image into character string data using optical character recognition technology, means for extracting important keywords from the character string data, means for analyzing the extracted keywords using natural language processing technology and evaluating the risk of causing harm to the user, and means for notifying the user of the evaluation result and visually highlighting parts requiring particular attention. This enables the user to quickly understand the contents of the contract or terms of use and efficiently identify any disadvantageous conditions.

[0426] "Agreement or Terms of Use" means the legal document to which a User agrees when using a Service or purchasing a Product.

[0427] "Digital image" refers to image data obtained by photographing the contents of a contract or terms of use with a camera or scanner.

[0428] "Optical character recognition technology" is a technology that reads character information from digital images and converts it into text data.

[0429] "String data" means data that expresses the contents of a contract or terms of use in text format, converted using optical character recognition technology.

[0430] "Keywords" are words or phrases that are considered important to users in analyzing the content of a contract or terms of use.

[0431] "Natural language processing technology" refers to information processing technology used to understand and analyze human language.

[0432] A "generative AI model" is a model that uses artificial intelligence techniques to learn from training data and perform specific tasks.

[0433] A "web resource identifier" is an address (such as a URL) that uniquely identifies content on the Internet.

[0434] The "evaluation result" refers to the result of the risk assessment obtained by the server by analyzing keywords and text data.

[0435] A "warning message" is a message generated to notify a user of a risk based on the evaluation results.

[0436] "Visual emphasis" refers to the means of highlighting specific text or areas to help users intuitively understand important information.

[0437] This invention is a system that makes it easier for users to understand the contents of contracts and terms of use. In order to reduce the risk of users signing unfavorable contracts, the system analyzes uploaded contract and terms of use data and performs a series of processes to evaluate the risk.

[0438] Users can take a digital image of the contract or terms of use, or prepare an existing screenshot, or if the contract or terms of use is online, they can upload it by entering the URL into a dedicated application.

[0439] The device sends image data and URLs uploaded by the user to the server. The server applies optical character recognition (OCR) technology to the received image data and converts it into text data. Possible OCR technologies include Google Cloud Vision and Tesseract. If a URL is sent, the server uses web scraping technology to retrieve the text of the contract or terms of use from the webpage. BeautifulSoup and Selenium are used for this process.

[0440] The server then analyzes the converted text data and uses natural language processing (NLP) techniques to extract important keywords and phrases from the text of the contract or terms of use. SpaCy and NLTK are used as NLP tools. These keywords include risk-related words such as "penalty fee" and "forced termination." The server then uses a generative AI model to perform a risk assessment based on the extracted keywords. The generative AI model is fed the analyzed text data and keywords along with a prompt like the following:

[0441] "This agreement contains the following keywords: penalty, termination. Please assess the risk these pose to you."

[0442] The server sends the evaluation results to the device. The device then notifies the user based on the evaluation results and visually highlights any areas requiring particular attention. For example, it highlights specific contract terms or comments regarding risks and displays a warning message to the user. This allows the user to quickly understand the contents of the contract or terms of use and reduce the risk of suffering disadvantages.

[0443] A specific example is a process in which a user takes a photo of the terms of use for a new service with their smartphone and uploads the image to an app. In this case, the device sends the image to a server, which extracts text data from the image. The server analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk they pose to the user. The evaluation results are sent to the device, which then displays a warning to the user, such as "This contract has a high penalty fee. Please be careful."

[0444] As described above, the system of the present invention allows users to understand the contents of complex contracts and terms of use in a short period of time, thereby improving the user's sense of security.

[0445] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0446] Step 1:

[0447] The user prepares digital images of the contract and terms of use.

[0448] Users can take a picture of the contract or terms of use using their smartphone's camera or scanner, or they can use an existing screenshot.If the contract or terms of use is online, they can also enter its URL into a dedicated application.

[0449] Input: Digital image or URL of the contract or terms of use

[0450] Output: Image file or URL of the contract or terms of use

[0451] Specific behavior:

[0452] The user takes a photo of the contract with their smartphone camera, saves the image, and then copies the URL of the online terms of use and pastes it into a dedicated application.

[0453] Step 2:

[0454] The device sends image data and URLs to the server.

[0455] The device sends the digital images and URLs uploaded by the user to the server.

[0456] Transfer this data to the system's API using an HTTP POST request.

[0457] Input: Image file or URL of the contract or terms of use

[0458] Output: Image file or URL sent to the server

[0459] Specific behavior:

[0460] The device sends a POST request to http: / / example.com / api / upload with the image file selected by the user, or sends a URL entered by the user to the same endpoint.

[0461] Step 3:

[0462] The server converts the image data into character string data.

[0463] The server applies optical character recognition (OCR) technology to the received image data, converting the text information in the image into string data. The technologies used are Google Cloud Vision and Tesseract. When a URL is submitted, the server uses web scraping technology, such as BeautifulSoup and Selenium, to extract the text of the contract or terms of use.

[0464] Input: Image file or URL

[0465] Output: Converted string data

[0466] Specific behavior:

[0467] The server sends the image to the Google Cloud Vision API for OCR processing, and also uses Selenium to access the URL and retrieve the contract and terms of use text.

[0468] Step 4:

[0469] The server analyzes the string data and extracts important keywords.

[0470] The server then analyzes the converted text data and extracts important keywords and phrases using natural language processing (NLP) techniques, such as SpaCy and NLTK.

[0471] Input: Converted string data

[0472] Output: Extracted important keywords

[0473] Specific behavior:

[0474] The server analyzes the string data using SpaCy to identify risk-related keywords (e.g., penalty fees, forced termination), and generates a list of important keywords as a result of the analysis.

[0475] Step 5:

[0476] The server performs risk assessment using a generative AI model

[0477] The server performs risk assessment using a generative AI model based on the extracted keywords. It passes input data along with prompt statements to the model and obtains the assessment results.

[0478] Input: Extracted important keywords, prompt sentence

[0479] Output: Risk assessment results

[0480] Specific behavior:

[0481] The server inputs the following prompt to the generative AI model: "This contract contains the following keywords: penalty fee, forced termination. Please evaluate what risks these pose to the user." The server then obtains the evaluation result returned by the AI ​​model.

[0482] Step 6:

[0483] The server sends the evaluation results to the terminal.

[0484] The server transmits the generated risk assessment results to the terminal.

[0485] Input: Risk assessment results

[0486] Output: Evaluation results sent to the device

[0487] Specific behavior:

[0488] The server packages the risk assessment results in JSON format and sends them to the terminal using the sendResponse method.

[0489] Step 7:

[0490] The device notifies the user of the evaluation results and highlights them visually.

[0491] The device notifies the user based on the evaluation results received from the server, visually highlighting any parts of the evaluation results that require special attention.

[0492] Input: Risk assessment results

[0493] Output: Evaluation results notified to the user, with visually highlighted notes

[0494] Specific behavior:

[0495] The device displays the evaluation results on the user interface, highlights specific keywords (e.g., penalty fees), and issues a warning to the user via a pop-up message.

[0496] Through the above processing steps, the system quickly and accurately analyzes the contents of contracts and terms of use, and provides users with a risk assessment in an easy-to-understand format.

[0497] (Application example 1)

[0498] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0499] There is a need to reduce the risk of users signing contracts and terms of use without fully understanding them. However, especially in the case of electronic payment services, terms of use are often complex and lengthy, making them difficult for users to understand. As a result, users may suffer disadvantages. To solve this problem, a system is needed that allows users to intuitively understand the contents of the contract and immediately notifies them of particularly high-risk parts.

[0500] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0501] In this invention, the server includes means for uploading an image of the contract or terms of use, means for converting the uploaded image into text data using optical character recognition technology, means for extracting important keywords from the converted text data, means for evaluating the risk of harm to the user based on the extracted keywords, means for notifying the user of the evaluation result, means for notifying the user of the evaluation result in real time, and means for integrating the harm risk evaluation into the electronic payment environment. This allows the user to quickly understand the risks in the contract content and reduce the risk of suffering harm.

[0502] "Agreement or Terms of Use" means a document between a user and a provider regarding the use of a service or purchase of a product, which defines the terms of use and rights and obligations.

[0503] "Images" are visual information converted into digital data, and refer to the contents of contracts and terms of use as photographs or scanned data.

[0504] "Uploading" is the operation by which a user sends digital data to a server or cloud service via the Internet.

[0505] Optical character recognition (OCR) is a technology that reads character information contained in image data and converts it into text data.

[0506] "Text data" is digital information written in text format, and is data in a form that is easy to search and analyze.

[0507] "Important keywords" are words or phrases in the contract that may have a significant impact on the user.

[0508] "Extraction" is the operation of extracting necessary information or elements from text data.

[0509] "Risk assessment" is a process of analyzing and determining the possibility that a user may suffer a disadvantage based on the extracted keywords.

[0510] "Real-time notification" refers to providing analysis results immediately after the user provides data.

[0511] "Integration into the electronic payment environment" refers to a system that automatically incorporates risk analysis functions when conducting online financial transactions and promptly notifies users.

[0512] This invention is a system that enables users to quickly understand the contents of contracts and terms of use, reducing the risk of suffering disadvantages. Specifically, the system uploads images or URLs of contracts and terms of use, converts them into text data, and analyzes them using natural language processing technology.

[0513] System Configuration

[0514] Hardware: Smartphones, smart glasses, servers

[0515] Software: OCR technology (e.g., Tesseract), web scraping tools (e.g., BeautifulSoup), natural language processing (e.g., spaCy)

[0516] Operation overview

[0517] 1. Obtaining contract and terms of use data:

[0518] The user takes a picture of the contract or terms of use using a smartphone or smart glasses and uploads it to the application.

[0519] The user enters the URL of the online contract or terms of use into the application and uploads it.

[0520] The terminal transmits the input image or URL to the server.

[0521] 2. Data conversion and text generation:

[0522] The server converts the uploaded image into text data using optical character recognition (OCR) technology, which uses Tesseract.

[0523] When the URL is uploaded, the server uses a web scraping tool to extract the text data of the contract or terms of use from the web page. For example, we use BeautifulSoup.

[0524] 3. Natural Language Processing and Risk Assessment:

[0525] The server then uses natural language processing (NLP) techniques on the converted text data to extract important keywords and phrases. This process is performed using spaCy.

[0526] Based on the extracted keywords, the risk that the contract contents will cause disadvantage to the user is evaluated.

[0527] Based on the results of the risk assessment, necessary precautions and warnings are generated.

[0528] 4. Real-time notifications:

[0529] The server transmits the evaluation results to the terminal in real time.

[0530] The device notifies the user of the evaluation results and highlights areas requiring attention.

[0531] Usage example

[0532] The user takes a photo of the terms of use for a new service with their smartphone and uploads the image to the application. The device sends the image to the server, which extracts text from the image. The server analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk these keywords pose to the user. The evaluation results are sent to the device, which then displays a warning to the user, such as "This contract has a high penalty fee. Please be careful."

[0533] Prompt Sentence Examples

[0534] "Do the new terms of service agreements include penalties? And what are the risks to users?"

[0535] By using such a system, users can quickly understand the contents of the contract and reduce the risk of suffering disadvantages.

[0536] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0537] Step 1:

[0538] Users upload images of contracts and terms of use

[0539] The user takes an image of the contract or terms of use using a smartphone or smart glasses. Then, the user uploads the image or screenshot to the application. The uploaded image is sent to the server.

[0540] Input: Image of contract or terms of use

[0541] Output: Image data sent to the server

[0542] Step 2:

[0543] User uploads URL of contract or terms of use

[0544] Users enter the URL of the online contract or terms of use into a dedicated application and upload it. The entered URL is sent to the server.

[0545] Input: URL of contract or terms of use

[0546] Output: URL data sent to the server

[0547] Step 3:

[0548] The server converts image data into text data

[0549] The server uses optical character recognition (OCR) technology to extract character information from uploaded images and convert them into text data. The OCR technology used is Tesseract.

[0550] Input: Uploaded image data

[0551] Output: Text data

[0552] Step 4:

[0553] The server scrapes the web page content from the URL.

[0554] The server retrieves the contract and terms of use from the uploaded URL using web scraping technology, which uses BeautifulSoup to extract the text from the web page.

[0555] Input: Uploaded URL

[0556] Output: Text data

[0557] Step 5:

[0558] The server uses natural language processing technology to extract important keywords

[0559] The server then uses natural language processing (NLP) technology on the converted or scraped text data to extract important keywords and phrases. The NLP technology used is spaCy.

[0560] Input: Text data

[0561] Output: Extracted keywords and phrases

[0562] Step 6:

[0563] The server assesses the risk

[0564] The server uses the extracted keywords and phrases to assess the risk that the contract content will cause disadvantage to the user, using a generative AI model for the assessment.

[0565] Input: Extracted keywords or phrases

[0566] Output: Risk assessment results

[0567] Step 7:

[0568] The server notifies the evaluation results in real time

[0569] The server transmits the risk assessment results in real time to the user's terminal, which then notifies the user of the results.

[0570] Input: Risk assessment results

[0571] Output: The evaluation results displayed to the user

[0572] Step 8:

[0573] The device displays risk information to the user

[0574] Based on the evaluation results received, the device highlights and displays parts of the contract or terms of use that require particular attention, and also provides risk assessment comments and advice.

[0575] Input: Risk assessment results

[0576] Output: Highlighted contracts and terms of use, risk assessment comments

[0577] Prompt Sentence Examples

[0578] "Do the new terms of service agreements include penalties? And what are the risks to users?"

[0579] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0580] This invention is a system that makes contracts and terms of use easier for users to understand, reducing the risk of signing a disadvantageous contract, and also combines it with an emotion engine that recognizes the user's emotions and optimizes the information provided based on those emotions. Below, we will explain the program of this system in detail, and explain each processing step in natural language.

[0581] First, the user takes a picture of the contract or terms of use, or uploads an existing screenshot or URL to the application. The device then sends the image or URL to the server. The server then performs optical character recognition (OCR) on the received image data and converts the characters in the image into text data. If a URL is provided, the server uses web scraping technology to convert the contents of the web page into text data.

[0582] The server then pre-processes the retrieved text data, removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs. The server then uses a pre-defined risk keyword list to extract important keywords and phrases from the text data, including risk-related words such as "penalty" and "forced termination."

[0583] The server then uses natural language processing (NLP) technology to perform a risk assessment of clauses containing the extracted keywords. An AI-based predictive model is used for the risk assessment, and a risk score is assigned to each clause. The system also incorporates an emotion engine that can recognize the user's emotions. The emotion engine analyzes emotions from the user's voice, facial expressions, and input text to determine how the user feels about the contract or terms of use.

[0584] The emotion information recognized by the emotion engine is reflected in the assessment results and notification content by the server. For example, if the user expresses anxiety or doubt, the server can provide additional advice or detailed explanations. The server can also adjust the display format of the risk assessment and the tone of the notification message based on the emotion information.

[0585] Based on the evaluation results and sentiment information, the server generates a risk report for the user. High-risk clauses are highlighted and an evaluation score and advice are attached. A warning message based on the overall risk status is also generated. The server sends the generated risk report to the terminal, which displays it to the user.

[0586] As a concrete example, a user takes a photo of the terms of use for a new service and uploads the image to the app. The device sends the image to a server, which extracts text from the image. The server analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk these pose to the user. The emotion engine recognizes anxious emotions from the user's facial expressions and voice, and the server displays a message warning them that "This contract has a high penalty fee. Please be careful," along with suggestions for additional details or support.

[0587] In this way, users can quickly understand the contents of contracts and terms of use, reducing the risk of suffering disadvantages. Furthermore, the introduction of an emotion engine provides optimal advice and support based on the user's emotions, allowing users to review contract details with greater peace of mind. This system significantly simplifies the process of reviewing complex contracts and terms of use, increasing users' sense of security.

[0588] The processing flow will be explained below.

[0589] Step 1:

[0590] Users can take a picture of the contract or terms of use, or upload an existing screenshot / URL to the application using a smartphone or tablet.

[0591] Step 2:

[0592] The device sends the uploaded image and URL to the server, where the data is encrypted and securely transferred to the server.

[0593] Step 3:

[0594] The server uses optical character recognition (OCR) technology to convert the characters in the image into text data, and if a URL is provided, uses web scraping technology to retrieve the text of the contract or terms of use from the web page.

[0595] Step 4:

[0596] The server preprocesses the converted text data, specifically removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs. This step improves the quality of the text data.

[0597] Step 5:

[0598] The server extracts important keywords and phrases from the text data, using a predefined list of risk keywords to detect specific words (e.g., "penalty fee," "forced termination," "venue," etc.).

[0599] Step 6:

[0600] The server uses natural language processing (NLP) techniques to perform a risk assessment on clauses containing the extracted keywords, using an AI model to assign a risk score to each clause.

[0601] Step 7:

[0602] The server recognizes the user's emotions through an emotion engine, analyzing facial expressions, voice data, and input text when the user operates the application. Emotions are classified as anxiety, doubt, relief, etc.

[0603] Step 8:

[0604] The server adjusts the evaluation results and notification content based on the emotional information recognized by the emotion engine. For example, if the user expresses anxiety, it generates a notification with a detailed explanation and additional support information.

[0605] Step 9:

[0606] The server generates a risk report based on the evaluation results and sentiment information. High-risk clauses are highlighted and accompanied by an evaluation score and advice. Warning messages are also generated based on the overall risk situation.

[0607] Step 10:

[0608] The server sends the generated risk report to the terminal.

[0609] Step 11:

[0610] The terminal displays the risk report received from the server to the user. High-risk clauses are highlighted in red, and specific risk details and warnings are displayed as pop-ups.

[0611] This process allows users to quickly understand the contents of the contract and terms of use, reducing the risk of suffering disadvantages. Furthermore, appropriate advice and support tailored to the user's feelings are provided, allowing users to review the contract contents with greater peace of mind.

[0612] Example 2

[0613] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0614] Conventional contracts and terms of use are difficult for ordinary users to understand, increasing the risk of them signing disadvantageous contracts. Furthermore, because they do not take into account the emotions users feel toward these documents, necessary support and advice are lacking. Therefore, there is a need for a system that makes contracts and terms of use easier to understand and provides appropriate information that takes users' emotions into account.

[0615] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0616] In this invention, the server includes means for uploading an image of a contract or terms of use, means for converting the uploaded image into text data using optical character recognition technology, means for extracting important keywords from the converted text data, means for evaluating the risk of causing harm to the user based on the extracted keywords, means for analyzing emotions from the user's voice, facial expression, input text, etc., means for adjusting the evaluation result and notification content based on the analyzed emotional information, and means for notifying the user of the evaluation result. This makes it possible to objectively evaluate the risks of the contract or terms of use and provide appropriate information and support according to the user's emotions.

[0617] "Agreement" or "Terms of Use" means a document that contains the formal terms and conditions governing the use of a service and that clarifies the rights and obligations of both parties.

[0618] "Means for uploading images" is a function that allows a user to use a terminal to send image data of a contract or terms of use to a server.

[0619] "Optical character recognition technology" refers to the technology that analyzes characters in image data and converts them into text data.

[0620] The "means for converting into text data" is a function that uses optical character recognition technology to extract character information from image data and convert it into digital text format.

[0621] "Means for extracting important keywords" refers to the ability to identify and extract words and phrases related to specific risks or important matters from text data.

[0622] The "means for assessing the risk of causing harm" is a function that analyzes and assesses the risks that the clauses of a contract or terms of use may pose to the user, based on the extracted keywords.

[0623] "Means for analyzing emotions from a user's voice, facial expressions, input text, etc." refers to technology that analyzes audio, video, and text data collected through input devices such as microphones and cameras to identify the user's emotional state.

[0624] The "means for adjusting evaluation results and notification content" is a function for adjusting the content and tone of evaluation results and notification messages provided to users based on the results of emotion analysis.

[0625] The "means for notifying the user of the evaluation results" is a function for sending the results of risk evaluation and advice to the terminal and displaying them to the user.

[0626] "Method of scraping the contents of a contract or terms of use from a URL to obtain text data" refers to a technology that analyzes the contents of a web page and extracts the text data of a contract or terms of use.

[0627] "Natural language processing technology" is a technology that allows computers to understand and analyze human language, and involves classifying text data and performing semantic analysis.

[0628] This invention is a system that makes contracts and terms of use easier for users to understand, reducing the risk of signing a disadvantageous contract, and also combines an emotion engine that recognizes the user's emotions and optimizes the information provided accordingly. The program processing of this system is explained in detail below.

[0629] First, the user takes a picture of the contract or terms of use, or uploads an existing screenshot or URL to the application. This is done using a device such as a smartphone or tablet. The device then sends the image and URL data to the server.

[0630] The server applies optical character recognition (OCR) technology to the received image data to convert the characters in the image into text data using an OCR engine such as Tesseract. If a URL is provided, the server uses a web scraping technology such as BeautifulSoup or Scrapy to convert the contents of the web page into text data.

[0631] The server then pre-processes the retrieved text data, removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs using a text cleaning algorithm based on regular expressions.

[0632] The server then extracts important keywords and phrases from the text data using a predefined list of risk keywords, including risk-related words such as "penalty" and "cancellation." This process is performed using a Python NLP library (e.g., NLTK or spaCy).

[0633] The server then uses natural language processing (NLP) technology to perform a risk assessment of clauses containing the extracted keywords, using an AI-powered predictive model to assign a risk score to each clause using TensorFlow and PyTorch.

[0634] The system also incorporates an emotion engine that analyzes emotions from the user's voice, facial expressions, input text, etc. The emotion engine uses data collected from the webcam and smartphone microphone to recognize emotions using OpenCV and the Google Cloud Speech-to-Text API.

[0635] The emotion information recognized by the emotion engine is reflected in the assessment results and notification content by the server. For example, if the user expresses anxiety or doubt, the server will provide additional advice or detailed explanations. The server will also adjust the display format of the risk assessment and the tone of the notification message based on the emotion information.

[0636] Based on the evaluation results and sentiment information, the server generates a risk report for the user. High-risk clauses are highlighted and an evaluation score and advice are attached. A warning message based on the overall risk status is also generated. The server sends the generated risk report to the terminal, which displays it to the user.

[0637] As a concrete example, if a user takes a photo of the terms of use for a new service and uploads the image to the app, the system operates as follows:

[0638] 1. The device sends the image to the server.

[0639] 2. The server extracts text from the image using OCR technology.

[0640] 3. The server analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk they pose to the user.

[0641] 4. The emotion engine recognizes anxious emotions from the user's facial expressions and voice.

[0642] 5. The server displays a message warning you that "This contract has a high penalty. Be careful" and offering additional details or support.

[0643] An example of an input prompt for the generative AI model is, "Please analyze the image of the contract below and assess the risks. Also, if the user feels anxious, please provide advice appropriate to their feelings."

[0644] This system helps users understand the contents of contracts and terms of use in a short amount of time, reducing the risk of suffering disadvantages. In addition, the introduction of an emotion engine provides optimal advice and support based on the user's emotions, allowing users to review contract details with greater peace of mind. This greatly simplifies the process of reviewing complex contracts and terms of use, improving users' sense of security.

[0645] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0646] Step 1:

[0647] The user takes a picture of the contract or terms of use, or uploads an existing screenshot or URL to the application. Specifically, when the user launches the app and taps the "Upload contract" button, the camera on the smartphone or tablet is activated. The user then takes a picture of the contract with the camera and presses the "Upload" button. Similarly, in the case of a screenshot or URL, the user is presented with the option to "Select file" or "Enter URL," and selects one of the options.

[0648] Input: Image data, screenshots, or URLs of contracts or terms of use

[0649] Output: Image data and URLs are imported to the device

[0650] Step 2:

[0651] The device sends the captured image or the provided URL to the server. Specifically, the device obtains the image data or URL data captured or uploaded by the user and sends it to the server via the Internet. An HTTP POST request is used to send the data to the endpoint.

[0652] Input: Image data or URL (data imported to the device)

[0653] Output: Image data and URL are sent to the server.

[0654] Step 3:

[0655] The server applies optical character recognition (OCR) technology to the received image data to convert the characters in the image into text data. Specifically, the server runs an OCR engine such as Tesseract to convert the characters in the image data into text format. Also, if a URL is provided, the server uses a web scraping tool such as BeautifulSoup or Scrapy to extract the text data from the page.

[0656] Input: Image data or URL sent

[0657] Output: Extracted text data

[0658] Step 4:

[0659] The server pre-processes the retrieved text data, removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs. Specifically, text cleaning algorithms are run, regular expressions are used to remove whitespace and special characters, and natural line and paragraph breaks are implemented.

[0660] Input: Extracted text data

[0661] Output: Cleaned text data

[0662] Step 5:

[0663] The server extracts important keywords and phrases from the text data using a predefined list of risk keywords. Specifically, the server uses a Python NLP library (e.g., NLTK or spaCy) to compare the list of risk keywords with the text data and perform the keyword extraction process.

[0664] Input: Cleaned text data

[0665] Output: Extracted important keywords and phrases

[0666] Step 6:

[0667] The server performs a risk assessment of clauses containing the extracted keywords. Specifically, an AI predictive model (e.g., TensorFlow or PyTorch) analyzes text containing risk keywords and calculates and assigns a risk score to each clause. The assessment results are stored in a database.

[0668] Input: Extracted important keywords and phrases

[0669] Output: Risk assessment score and assessment result

[0670] Step 7:

[0671] The server is equipped with an emotion engine that analyzes emotions from the user's voice, facial expressions, and input text. Specifically, it collects audio data from the webcam or smartphone microphone and runs an emotion recognition algorithm to analyze it (e.g., OpenCV is used for facial recognition, and Google Cloud Speech-to-Text API is used for voice analysis). The user's emotions are identified and stored in a database.

[0672] Input: Audio data, image data, input text

[0673] Output: Parsed emotion data

[0674] Step 8:

[0675] The server reflects the emotional information recognized by the emotion engine in the evaluation results and notification content. Specifically, it adjusts the tone of the risk assessment and notification message based on the emotional data. For example, for a user who expresses anxiety, it generates a warning message such as, "This contract has a high penalty fee. Please be careful."

[0676] Input: Risk assessment score, parsed sentiment data

[0677] Output: Adjusted evaluation results and notification messages

[0678] Step 9:

[0679] The server generates a risk report based on the evaluation results and emotional information. Specifically, it generates a risk report template that integrates the evaluation scores and emotional data, and creates a report in PDF or HTML format that visually highlights high-risk areas.

[0680] Input: Adjusted evaluation results and notification messages

[0681] Output: Generated risk report

[0682] Step 10:

[0683] The server sends the generated risk report to the terminal, which then displays it to the user. Specifically, the generated risk report is delivered to the terminal and displayed so that the user can view it on the application. There is a notification function to notify the user of the receipt of the risk report.

[0684] Input: Generated risk report

[0685] Output: Risk report displayed to the user

[0686] (Application example 2)

[0687] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0688] In modern digital services, many users agree to contracts and terms of use without properly understanding them. This increases the risk of signing unfavorable contracts. However, the content of the terms of use is complex, requiring time and effort for users to understand them. Furthermore, there is a lack of mechanisms to concretely recognize abstract risks or to support emotional decision-making. Therefore, there is a need for a method to more easily understand the risks in terms of use and contracts and provide emotional feedback.

[0689] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading an image of a contract or terms of use, means for converting the uploaded image into text data using optical character recognition technology, means for extracting important keywords from the converted text data, means for evaluating risks that may cause harm to the user based on the extracted keywords, means for notifying the user of the evaluation results, means for analyzing the user's emotions and optimizing the notification content based on the determined emotions, and means for generating a risk report based on the evaluation results and emotion information and transmitting the report to the terminal. This enables the user to quickly understand the contents of the contract or terms of use and receive optimal advice tailored to their emotions.

[0690] A "contract" is a document that describes the terms and conditions of a legal agreement.

[0691] "Terms of Use" means a document that sets out the conditions and rules for the use of a digital service or product.

[0692] "Means" are the methods or techniques used to achieve a particular goal.

[0693] "Means for uploading images" refers to a function that allows a user to send a specific image file to the system through a device.

[0694] "Optical character recognition technology" is a technology that converts characters in an image into digital text.

[0695] "Text data" is data that represents character information in a digital format.

[0696] "Important keywords" are words that are particularly noteworthy because they indicate specific content or risks.

[0697] "Keyword extraction means" refers to a technique for selecting specific words or phrases from text data.

[0698] The "means for assessing risk" is a method for analyzing the extracted keywords and determining the potential risks associated with them.

[0699] The "means for notifying the user of the evaluation results" is a function for displaying or communicating the results of the risk evaluation to the user.

[0700] "Means for analyzing emotions" refers to technology that recognizes and evaluates emotions from the user's voice, facial expressions, text input, etc.

[0701] "Means for optimizing notification content" refers to a method for adjusting the content and tone of a message based on the user's emotions.

[0702] A "risk report" is a document that compiles assessed risk information and advice.

[0703] "Terminal" refers to a digital device operated by a user.

[0704] The following describes an embodiment of the present invention.

[0705] The server has a function that allows users to upload images of contracts or terms of use to the server using their devices. The system starts when a user takes a photo of a contract and uploads the image using a smartphone or tablet.

[0706] The server then applies optical character recognition (OCR) technology to the uploaded image, converting the textual information in the image into digital text data. The specific software used is "pytesseract." The text extracted from the image is then entered into the system.

[0707] In the stage of extracting important keywords from the converted text data, analysis is performed based on a predefined list of risk keywords. For example, risk-related words such as "penalty" and "forced termination" are listed and extracted from the text data. Natural language processing (NLP) technology is used to achieve this function.

[0708] The server then evaluates the risk of harm to the user based on the extracted keywords. This risk evaluation uses a predictive model using a generative AI model to assign a risk score to each clause, which clarifies the risk level that the user should be aware of.

[0709] After the evaluation results are obtained, a method is used to analyze the user's emotions. This emotion analysis utilizes an emotion engine that analyzes the user's voice and facial expressions. For example, software such as "EmotionRecognition" is used to analyze the user's emotions. The content of notifications is optimized based on the emotional information.

[0710] Finally, a risk report is generated based on the evaluation results and sentiment information. This risk report highlights high-risk clauses and includes an evaluation score and advice. It also generates a warning message based on the overall risk situation. The server sends this risk report to the terminal, which then displays it to the user.

[0711] As a concrete example, consider the case where a user takes a screenshot of an online shopping website's terms of use and uploads the image to an app. The device sends the image to a server, which extracts text from the image. The server then analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk they pose to the user. The emotion engine recognizes anxious emotions from the user's facial expressions and voice, and the server displays a message warning the user that "This contract has a high penalty fee. Please be careful," along with suggestions for additional details or support.

[0712] An example prompt for a generative AI model might look something like this:

[0713] > "Read the user's sentiment and if they are unsure about the contract, generate advice like this:

[0714] "These terms of service have significant penalties. Please see additional details."

[0715] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0716] Step 1:

[0717] A user takes a picture of a contract or terms of use and uploads it to a server using their device. At this point, the input is the captured image file and the output is the image sent to the server.

[0718] Step 2:

[0719] The server applies optical character recognition (OCR) technology to the uploaded image and converts the characters in the image into digital text data. Specifically, it uses the pytesseract library to analyze the image file. The input is the image file, and the output is the converted text data.

[0720] Step 3:

[0721] The server extracts important keywords from the converted text data based on a predefined risk keyword list. It uses NLP technology to identify risk-related words and phrases. The input is the text data converted by OCR, and the output is the extracted keyword list.

[0722] Step 4:

[0723] The server evaluates the risk of posing a risk to the user based on the extracted keywords. It uses a generative AI model to calculate a risk score and determine the risk level of each keyword. The input is the extracted keyword list, and the output is the analysis result with a risk score.

[0724] Step 5:

[0725] When the user reads the contract, the server uses an emotion analysis engine to analyze the user's emotions. Specifically, the server analyzes the user's facial expressions and voice using emotion recognition software (e.g., EmotionRecognition). The input is the user's facial expressions and voice data, and the output is the determined emotional information.

[0726] Step 6:

[0727] The server optimizes the notification content for the user based on the results of risk assessment and sentiment analysis. It uses a generative AI model to generate an appropriate message based on the sentiment. A specific example includes a message such as, "This contract has a high penalty fee. Please be careful." The input is the risk assessment result and sentiment information, and the output is an optimized message.

[0728] Step 7:

[0729] The server generates a risk report based on the evaluation results and sentiment information. The report highlights high-risk clauses and provides an evaluation score and advice. The input is the optimized message and the risk evaluation results, and the output is the generated risk report.

[0730] Step 8:

[0731] The server sends the generated risk report to the terminal, which displays it to the user, allowing the user to easily understand the contents of the contract and its potential risks. The input is the risk report, and the output is a detailed report displayed to the user.

[0732] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0733] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0734] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0735] [Third embodiment]

[0736] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0737] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0738] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0739] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0740] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0741] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0742] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0743] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0744] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0745] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0746] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0747] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0748] This invention is a system that makes contracts and terms of use easier for users to understand, reducing the risk of signing a disadvantageous contract. Next, we will create a program for this system and explain the program's processing in natural language.

[0749] First, users take a photo of the contract or terms of use or prepare a screenshot of an existing one, then upload the image to a dedicated application. Users can also upload online contracts or terms of use by entering the URL into the same application.

[0750] The device then sends the uploaded image or URL to a server, which uses optical character recognition (OCR) technology to extract character information from the image data and converts it into text data. If a URL is sent, the server uses web scraping technology to retrieve the text of the contract or terms of use from the webpage.

[0751] The converted text data is then analyzed by the server, which uses natural language processing (NLP) technology to extract important keywords and phrases from the text of the contract or terms of use. These keywords include words related to risk, such as "penalty fee" and "forced termination." Based on these keywords, the server evaluates whether the contract contents may be disadvantageous to the user.

[0752] The server sends the details of the assessed risks and the results of the assessment to the device. The device then notifies the user of the assessment results and highlights any parts of the contract or terms of use that require particular attention. Detailed risk assessment comments and advice are also displayed, allowing the user to specifically understand the contract contents.

[0753] As a concrete example, a user takes a photo of the terms of use for a new service with their smartphone and uploads the image to an app. The device then sends the image to a server, which extracts text from the image. The server then analyzes the extracted text for important keywords such as "penalty fee" and "service suspension," and evaluates the risks these keywords pose to the user. The evaluation results are then sent to the device, which then displays a warning to the user, such as "This contract has a high penalty fee. Please be careful."

[0754] In this way, users can quickly understand the contents of contracts and terms of use, reducing the risk of suffering disadvantages. This system significantly simplifies the process of checking complex contracts and terms of use, improving users' peace of mind.

[0755] The processing flow will be explained below.

[0756] Step 1:

[0757] Users can take a picture of the contract or terms of use, or upload an existing screenshot / URL to the application.

[0758] Step 2:

[0759] The device sends the uploaded image and URL to the server.

[0760] Step 3:

[0761] The server performs an optical character recognition (OCR) process on the received image data to convert the characters in the image into text data, or, if a URL is provided, uses web scraping techniques to retrieve the content of the web page and convert it into text data.

[0762] Step 4:

[0763] The server pre-processes the retrieved text data, removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs.

[0764] Step 5:

[0765] The server uses a predefined list of risk keywords to extract important keywords and phrases from the text data, including "penalty fee," "forced termination," and "venue."

[0766] Step 6:

[0767] The server uses natural language processing (NLP) technology to perform a risk assessment of clauses containing the extracted keywords, using an AI-powered predictive model to assign a risk score to each clause.

[0768] Step 7:

[0769] The server generates a risk report for the user based on the AI ​​risk assessment results. High-risk clauses are highlighted, and an assessment score and advice are attached. A warning message based on the overall risk status is also generated.

[0770] Step 8:

[0771] The server sends the generated risk report to the terminal.

[0772] Step 9:

[0773] The terminal displays the risk report received from the server to the user. High-risk clauses are highlighted in red, and specific risk details and warnings are displayed as pop-ups.

[0774] Example 1

[0775] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0776] Conventional contracts and terms of use are often complex and full of technical terms, making them difficult for average users to understand. This puts users at risk of overlooking content that could be detrimental to them. Furthermore, reviewing contracts and terms of use is time-consuming and inefficient. This invention aims to enable users to quickly and easily understand the contents of contracts and terms of use, thereby reducing the risk of suffering disadvantages.

[0777] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0778] In this invention, the server includes means for uploading a digital image of a contract or terms of use, means for converting the uploaded digital image into character string data using optical character recognition technology, means for extracting important keywords from the character string data, means for analyzing the extracted keywords using natural language processing technology and evaluating the risk of causing harm to the user, and means for notifying the user of the evaluation result and visually highlighting parts requiring particular attention. This enables the user to quickly understand the contents of the contract or terms of use and efficiently identify any disadvantageous conditions.

[0779] "Agreement or Terms of Use" means the legal document to which a User agrees when using a Service or purchasing a Product.

[0780] "Digital image" refers to image data obtained by photographing the contents of a contract or terms of use with a camera or scanner.

[0781] "Optical character recognition technology" is a technology that reads character information from digital images and converts it into text data.

[0782] "String data" means data that expresses the contents of a contract or terms of use in text format, converted using optical character recognition technology.

[0783] "Keywords" are words or phrases that are considered important to users in analyzing the content of a contract or terms of use.

[0784] "Natural language processing technology" refers to information processing technology used to understand and analyze human language.

[0785] A "generative AI model" is a model that uses artificial intelligence techniques to learn from training data and perform specific tasks.

[0786] A "web resource identifier" is an address (such as a URL) that uniquely identifies content on the Internet.

[0787] The "evaluation result" refers to the result of the risk assessment obtained by the server by analyzing keywords and text data.

[0788] A "warning message" is a message generated to notify a user of a risk based on the evaluation results.

[0789] "Visual emphasis" refers to the means of highlighting specific text or areas to help users intuitively understand important information.

[0790] This invention is a system that makes it easier for users to understand the contents of contracts and terms of use. In order to reduce the risk of users signing unfavorable contracts, the system analyzes uploaded contract and terms of use data and performs a series of processes to evaluate the risk.

[0791] Users can take a digital image of the contract or terms of use, or prepare an existing screenshot, or if the contract or terms of use is online, they can upload it by entering the URL into a dedicated application.

[0792] The device sends image data and URLs uploaded by the user to the server. The server applies optical character recognition (OCR) technology to the received image data and converts it into text data. Possible OCR technologies include Google Cloud Vision and Tesseract. If a URL is sent, the server uses web scraping technology to retrieve the text of the contract or terms of use from the webpage. BeautifulSoup and Selenium are used for this process.

[0793] The server then analyzes the converted text data and uses natural language processing (NLP) techniques to extract important keywords and phrases from the text of the contract or terms of use. SpaCy and NLTK are used as NLP tools. These keywords include risk-related words such as "penalty fee" and "forced termination." The server then uses a generative AI model to perform a risk assessment based on the extracted keywords. The generative AI model is fed the analyzed text data and keywords along with a prompt like the following:

[0794] "This agreement contains the following keywords: penalty, termination. Please assess the risk these pose to you."

[0795] The server sends the evaluation results to the device. The device then notifies the user based on the evaluation results and visually highlights any areas requiring particular attention. For example, it highlights specific contract terms or comments regarding risks and displays a warning message to the user. This allows the user to quickly understand the contents of the contract or terms of use and reduce the risk of suffering disadvantages.

[0796] A specific example is a process in which a user takes a photo of the terms of use for a new service with their smartphone and uploads the image to an app. In this case, the device sends the image to a server, which extracts text data from the image. The server analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk they pose to the user. The evaluation results are sent to the device, which then displays a warning to the user, such as "This contract has a high penalty fee. Please be careful."

[0797] As described above, the system of the present invention allows users to understand the contents of complex contracts and terms of use in a short period of time, thereby improving the user's sense of security.

[0798] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0799] Step 1:

[0800] The user prepares digital images of the contract and terms of use.

[0801] Users can take a picture of the contract or terms of use using their smartphone's camera or scanner, or they can use an existing screenshot.If the contract or terms of use is online, they can also enter its URL into a dedicated application.

[0802] Input: Digital image or URL of the contract or terms of use

[0803] Output: Image file or URL of the contract or terms of use

[0804] Specific behavior:

[0805] The user takes a photo of the contract with their smartphone camera, saves the image, and then copies the URL of the online terms of use and pastes it into a dedicated application.

[0806] Step 2:

[0807] The device sends image data and URLs to the server.

[0808] The device sends the digital images and URLs uploaded by the user to the server.

[0809] Transfer this data to the system's API using an HTTP POST request.

[0810] Input: Image file or URL of the contract or terms of use

[0811] Output: Image file or URL sent to the server

[0812] Specific behavior:

[0813] The device sends a POST request to http: / / example.com / api / upload with the image file selected by the user, or sends a URL entered by the user to the same endpoint.

[0814] Step 3:

[0815] The server converts the image data into character string data.

[0816] The server applies optical character recognition (OCR) technology to the received image data, converting the text information in the image into string data. The technologies used are Google Cloud Vision and Tesseract. When a URL is submitted, the server uses web scraping technology, such as BeautifulSoup and Selenium, to extract the text of the contract or terms of use.

[0817] Input: Image file or URL

[0818] Output: Converted string data

[0819] Specific behavior:

[0820] The server sends the image to the Google Cloud Vision API for OCR processing, and also uses Selenium to access the URL and retrieve the contract and terms of use text.

[0821] Step 4:

[0822] The server analyzes the string data and extracts important keywords.

[0823] The server then analyzes the converted text data and extracts important keywords and phrases using natural language processing (NLP) techniques, such as SpaCy and NLTK.

[0824] Input: Converted string data

[0825] Output: Extracted important keywords

[0826] Specific behavior:

[0827] The server analyzes the string data using SpaCy to identify risk-related keywords (e.g., penalty fees, forced termination), and generates a list of important keywords as a result of the analysis.

[0828] Step 5:

[0829] The server performs risk assessment using a generative AI model

[0830] The server performs risk assessment using a generative AI model based on the extracted keywords. It passes input data along with prompt statements to the model and obtains the assessment results.

[0831] Input: Extracted important keywords, prompt sentence

[0832] Output: Risk assessment results

[0833] Specific behavior:

[0834] The server inputs the following prompt to the generative AI model: "This contract contains the following keywords: penalty fee, forced termination. Please evaluate what risks these pose to the user." The server then obtains the evaluation result returned by the AI ​​model.

[0835] Step 6:

[0836] The server sends the evaluation results to the terminal.

[0837] The server transmits the generated risk assessment results to the terminal.

[0838] Input: Risk assessment results

[0839] Output: Evaluation results sent to the device

[0840] Specific behavior:

[0841] The server packages the risk assessment results in JSON format and sends them to the terminal using the sendResponse method.

[0842] Step 7:

[0843] The device notifies the user of the evaluation results and highlights them visually.

[0844] The device notifies the user based on the evaluation results received from the server, visually highlighting any parts of the evaluation results that require special attention.

[0845] Input: Risk assessment results

[0846] Output: Evaluation results notified to the user, with visually highlighted notes

[0847] Specific behavior:

[0848] The device displays the evaluation results on the user interface, highlights specific keywords (e.g., penalty fees), and issues a warning to the user via a pop-up message.

[0849] Through the above processing steps, the system quickly and accurately analyzes the contents of contracts and terms of use, and provides users with a risk assessment in an easy-to-understand format.

[0850] (Application example 1)

[0851] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0852] There is a need to reduce the risk of users signing contracts and terms of use without fully understanding them. However, especially in the case of electronic payment services, terms of use are often complex and lengthy, making them difficult for users to understand. As a result, users may suffer disadvantages. To solve this problem, a system is needed that allows users to intuitively understand the contents of the contract and immediately notifies them of particularly high-risk parts.

[0853] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0854] In this invention, the server includes means for uploading an image of the contract or terms of use, means for converting the uploaded image into text data using optical character recognition technology, means for extracting important keywords from the converted text data, means for evaluating the risk of harm to the user based on the extracted keywords, means for notifying the user of the evaluation result, means for notifying the user of the evaluation result in real time, and means for integrating the harm risk evaluation into the electronic payment environment. This allows the user to quickly understand the risks in the contract content and reduce the risk of suffering harm.

[0855] "Agreement or Terms of Use" means a document between a user and a provider regarding the use of a service or purchase of a product, which defines the terms of use and rights and obligations.

[0856] "Images" are visual information converted into digital data, and refer to the contents of contracts and terms of use as photographs or scanned data.

[0857] "Uploading" is the operation by which a user sends digital data to a server or cloud service via the Internet.

[0858] Optical character recognition (OCR) is a technology that reads character information contained in image data and converts it into text data.

[0859] "Text data" is digital information written in text format, and is data in a form that is easy to search and analyze.

[0860] "Important keywords" are words or phrases in the contract that may have a significant impact on the user.

[0861] "Extraction" is the operation of extracting necessary information or elements from text data.

[0862] "Risk assessment" is a process of analyzing and determining the possibility that a user may suffer a disadvantage based on the extracted keywords.

[0863] "Real-time notification" refers to providing analysis results immediately after the user provides data.

[0864] "Integration into the electronic payment environment" refers to a system that automatically incorporates risk analysis functions when conducting online financial transactions and promptly notifies users.

[0865] This invention is a system that enables users to quickly understand the contents of contracts and terms of use, reducing the risk of suffering disadvantages. Specifically, the system uploads images or URLs of contracts and terms of use, converts them into text data, and analyzes them using natural language processing technology.

[0866] System Configuration

[0867] Hardware: Smartphones, smart glasses, servers

[0868] Software: OCR technology (e.g., Tesseract), web scraping tools (e.g., BeautifulSoup), natural language processing (e.g., spaCy)

[0869] Operation overview

[0870] 1. Obtaining contract and terms of use data:

[0871] The user takes a picture of the contract or terms of use using a smartphone or smart glasses and uploads it to the application.

[0872] The user enters the URL of the online contract or terms of use into the application and uploads it.

[0873] The terminal transmits the input image or URL to the server.

[0874] 2. Data conversion and text generation:

[0875] The server converts the uploaded image into text data using optical character recognition (OCR) technology, which uses Tesseract.

[0876] When the URL is uploaded, the server uses a web scraping tool to extract the text data of the contract or terms of use from the web page. For example, we use BeautifulSoup.

[0877] 3. Natural Language Processing and Risk Assessment:

[0878] The server then uses natural language processing (NLP) techniques on the converted text data to extract important keywords and phrases. This process is performed using spaCy.

[0879] Based on the extracted keywords, the risk that the contract contents will cause disadvantage to the user is evaluated.

[0880] Based on the results of the risk assessment, necessary precautions and warnings are generated.

[0881] 4. Real-time notifications:

[0882] The server transmits the evaluation results to the terminal in real time.

[0883] The device notifies the user of the evaluation results and highlights areas requiring attention.

[0884] Usage example

[0885] The user takes a photo of the terms of use for a new service with their smartphone and uploads the image to the application. The device sends the image to the server, which extracts text from the image. The server analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk these keywords pose to the user. The evaluation results are sent to the device, which then displays a warning to the user, such as "This contract has a high penalty fee. Please be careful."

[0886] Prompt Sentence Examples

[0887] "Do the new terms of service agreements include penalties? And what are the risks to users?"

[0888] By using such a system, users can quickly understand the contents of the contract and reduce the risk of suffering disadvantages.

[0889] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0890] Step 1:

[0891] Users upload images of contracts and terms of use

[0892] The user takes an image of the contract or terms of use using a smartphone or smart glasses. Then, the user uploads the image or screenshot to the application. The uploaded image is sent to the server.

[0893] Input: Image of contract or terms of use

[0894] Output: Image data sent to the server

[0895] Step 2:

[0896] User uploads URL of contract or terms of use

[0897] Users enter the URL of the online contract or terms of use into a dedicated application and upload it. The entered URL is sent to the server.

[0898] Input: URL of contract or terms of use

[0899] Output: URL data sent to the server

[0900] Step 3:

[0901] The server converts image data into text data

[0902] The server uses optical character recognition (OCR) technology to extract character information from uploaded images and convert them into text data. The OCR technology used is Tesseract.

[0903] Input: Uploaded image data

[0904] Output: Text data

[0905] Step 4:

[0906] The server scrapes the web page content from the URL.

[0907] The server retrieves the contract and terms of use from the uploaded URL using web scraping technology, which uses BeautifulSoup to extract the text from the web page.

[0908] Input: Uploaded URL

[0909] Output: Text data

[0910] Step 5:

[0911] The server uses natural language processing technology to extract important keywords

[0912] The server then uses natural language processing (NLP) technology on the converted or scraped text data to extract important keywords and phrases. The NLP technology used is spaCy.

[0913] Input: Text data

[0914] Output: Extracted keywords and phrases

[0915] Step 6:

[0916] The server assesses the risk

[0917] The server uses the extracted keywords and phrases to assess the risk that the contract content will cause disadvantage to the user, using a generative AI model for the assessment.

[0918] Input: Extracted keywords or phrases

[0919] Output: Risk assessment results

[0920] Step 7:

[0921] The server notifies the evaluation results in real time

[0922] The server transmits the risk assessment results in real time to the user's terminal, which then notifies the user of the results.

[0923] Input: Risk assessment results

[0924] Output: The evaluation results displayed to the user

[0925] Step 8:

[0926] The device displays risk information to the user

[0927] Based on the evaluation results received, the device highlights and displays parts of the contract or terms of use that require particular attention, and also provides risk assessment comments and advice.

[0928] Input: Risk assessment results

[0929] Output: Highlighted contracts and terms of use, risk assessment comments

[0930] Prompt Sentence Examples

[0931] "Do the new terms of service agreements include penalties? And what are the risks to users?"

[0932] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0933] This invention is a system that makes contracts and terms of use easier for users to understand, reducing the risk of signing a disadvantageous contract, and also combines it with an emotion engine that recognizes the user's emotions and optimizes the information provided based on those emotions. Below, we will explain the program of this system in detail, and explain each processing step in natural language.

[0934] First, the user takes a picture of the contract or terms of use, or uploads an existing screenshot or URL to the application. The device then sends the image or URL to the server. The server then performs optical character recognition (OCR) on the received image data and converts the characters in the image into text data. If a URL is provided, the server uses web scraping technology to convert the contents of the web page into text data.

[0935] The server then pre-processes the retrieved text data, removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs. The server then uses a pre-defined risk keyword list to extract important keywords and phrases from the text data, including risk-related words such as "penalty" and "forced termination."

[0936] The server then uses natural language processing (NLP) technology to perform a risk assessment of clauses containing the extracted keywords. An AI-based predictive model is used for the risk assessment, and a risk score is assigned to each clause. The system also incorporates an emotion engine that can recognize the user's emotions. The emotion engine analyzes emotions from the user's voice, facial expressions, and input text to determine how the user feels about the contract or terms of use.

[0937] The emotion information recognized by the emotion engine is reflected in the assessment results and notification content by the server. For example, if the user expresses anxiety or doubt, the server can provide additional advice or detailed explanations. The server can also adjust the display format of the risk assessment and the tone of the notification message based on the emotion information.

[0938] Based on the evaluation results and sentiment information, the server generates a risk report for the user. High-risk clauses are highlighted and an evaluation score and advice are attached. A warning message based on the overall risk status is also generated. The server sends the generated risk report to the terminal, which displays it to the user.

[0939] As a concrete example, a user takes a photo of the terms of use for a new service and uploads the image to the app. The device sends the image to a server, which extracts text from the image. The server analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk these pose to the user. The emotion engine recognizes anxious emotions from the user's facial expressions and voice, and the server displays a message warning them that "This contract has a high penalty fee. Please be careful," along with suggestions for additional details or support.

[0940] In this way, users can quickly understand the contents of contracts and terms of use, reducing the risk of suffering disadvantages. Furthermore, the introduction of an emotion engine provides optimal advice and support based on the user's emotions, allowing users to review contract details with greater peace of mind. This system significantly simplifies the process of reviewing complex contracts and terms of use, increasing users' sense of security.

[0941] The processing flow will be explained below.

[0942] Step 1:

[0943] Users can take a picture of the contract or terms of use, or upload an existing screenshot / URL to the application using a smartphone or tablet.

[0944] Step 2:

[0945] The device sends the uploaded image and URL to the server, where the data is encrypted and securely transferred to the server.

[0946] Step 3:

[0947] The server uses optical character recognition (OCR) technology to convert the characters in the image into text data, and if a URL is provided, uses web scraping technology to retrieve the text of the contract or terms of use from the web page.

[0948] Step 4:

[0949] The server preprocesses the converted text data, specifically removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs. This step improves the quality of the text data.

[0950] Step 5:

[0951] The server extracts important keywords and phrases from the text data, using a predefined list of risk keywords to detect specific words (e.g., "penalty fee," "forced termination," "venue," etc.).

[0952] Step 6:

[0953] The server uses natural language processing (NLP) techniques to perform a risk assessment on clauses containing the extracted keywords, using an AI model to assign a risk score to each clause.

[0954] Step 7:

[0955] The server recognizes the user's emotions through an emotion engine, analyzing facial expressions, voice data, and input text when the user operates the application. Emotions are classified as anxiety, doubt, relief, etc.

[0956] Step 8:

[0957] The server adjusts the evaluation results and notification content based on the emotional information recognized by the emotion engine. For example, if the user expresses anxiety, it generates a notification with a detailed explanation and additional support information.

[0958] Step 9:

[0959] The server generates a risk report based on the evaluation results and sentiment information. High-risk clauses are highlighted and accompanied by an evaluation score and advice. Warning messages are also generated based on the overall risk situation.

[0960] Step 10:

[0961] The server sends the generated risk report to the terminal.

[0962] Step 11:

[0963] The terminal displays the risk report received from the server to the user. High-risk clauses are highlighted in red, and specific risk details and warnings are displayed as pop-ups.

[0964] This process allows users to quickly understand the contents of the contract and terms of use, reducing the risk of suffering disadvantages. Furthermore, appropriate advice and support tailored to the user's feelings are provided, allowing users to review the contract contents with greater peace of mind.

[0965] Example 2

[0966] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0967] Conventional contracts and terms of use are difficult for ordinary users to understand, increasing the risk of them signing disadvantageous contracts. Furthermore, because they do not take into account the emotions users feel toward these documents, necessary support and advice are lacking. Therefore, there is a need for a system that makes contracts and terms of use easier to understand and provides appropriate information that takes users' emotions into account.

[0968] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0969] In this invention, the server includes means for uploading an image of a contract or terms of use, means for converting the uploaded image into text data using optical character recognition technology, means for extracting important keywords from the converted text data, means for evaluating the risk of causing harm to the user based on the extracted keywords, means for analyzing emotions from the user's voice, facial expression, input text, etc., means for adjusting the evaluation result and notification content based on the analyzed emotional information, and means for notifying the user of the evaluation result. This makes it possible to objectively evaluate the risks of the contract or terms of use and provide appropriate information and support according to the user's emotions.

[0970] "Agreement" or "Terms of Use" means a document that contains the formal terms and conditions governing the use of a service and that clarifies the rights and obligations of both parties.

[0971] "Means for uploading images" is a function that allows a user to use a terminal to send image data of a contract or terms of use to a server.

[0972] "Optical character recognition technology" refers to the technology that analyzes characters in image data and converts them into text data.

[0973] The "means for converting into text data" is a function that uses optical character recognition technology to extract character information from image data and convert it into digital text format.

[0974] "Means for extracting important keywords" refers to the ability to identify and extract words and phrases related to specific risks or important matters from text data.

[0975] The "means for assessing the risk of causing harm" is a function that analyzes and assesses the risks that the clauses of a contract or terms of use may pose to the user, based on the extracted keywords.

[0976] "Means for analyzing emotions from a user's voice, facial expressions, input text, etc." refers to technology that analyzes audio, video, and text data collected through input devices such as microphones and cameras to identify the user's emotional state.

[0977] The "means for adjusting evaluation results and notification content" is a function for adjusting the content and tone of evaluation results and notification messages provided to users based on the results of emotion analysis.

[0978] The "means for notifying the user of the evaluation results" is a function for sending the results of risk evaluation and advice to the terminal and displaying them to the user.

[0979] "Method of scraping the contents of a contract or terms of use from a URL to obtain text data" refers to a technology that analyzes the contents of a web page and extracts the text data of a contract or terms of use.

[0980] "Natural language processing technology" is a technology that allows computers to understand and analyze human language, and involves classifying text data and performing semantic analysis.

[0981] This invention is a system that makes contracts and terms of use easier for users to understand, reducing the risk of signing a disadvantageous contract, and also combines an emotion engine that recognizes the user's emotions and optimizes the information provided accordingly. The program processing of this system is explained in detail below.

[0982] First, the user takes a picture of the contract or terms of use, or uploads an existing screenshot or URL to the application. This is done using a device such as a smartphone or tablet. The device then sends the image and URL data to the server.

[0983] The server applies optical character recognition (OCR) technology to the received image data to convert the characters in the image into text data using an OCR engine such as Tesseract. If a URL is provided, the server uses a web scraping technology such as BeautifulSoup or Scrapy to convert the contents of the web page into text data.

[0984] The server then pre-processes the retrieved text data, removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs using a text cleaning algorithm based on regular expressions.

[0985] The server then extracts important keywords and phrases from the text data using a predefined list of risk keywords, including risk-related words such as "penalty" and "cancellation." This process is performed using a Python NLP library (e.g., NLTK or spaCy).

[0986] The server then uses natural language processing (NLP) technology to perform a risk assessment of clauses containing the extracted keywords, using an AI-powered predictive model to assign a risk score to each clause using TensorFlow and PyTorch.

[0987] The system also incorporates an emotion engine that analyzes emotions from the user's voice, facial expressions, input text, etc. The emotion engine uses data collected from the webcam and smartphone microphone to recognize emotions using OpenCV and the Google Cloud Speech-to-Text API.

[0988] The emotion information recognized by the emotion engine is reflected in the assessment results and notification content by the server. For example, if the user expresses anxiety or doubt, the server will provide additional advice or detailed explanations. The server will also adjust the display format of the risk assessment and the tone of the notification message based on the emotion information.

[0989] Based on the evaluation results and sentiment information, the server generates a risk report for the user. High-risk clauses are highlighted and an evaluation score and advice are attached. A warning message based on the overall risk status is also generated. The server sends the generated risk report to the terminal, which displays it to the user.

[0990] As a concrete example, if a user takes a photo of the terms of use for a new service and uploads the image to the app, the system operates as follows:

[0991] 1. The device sends the image to the server.

[0992] 2. The server extracts text from the image using OCR technology.

[0993] 3. The server analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk they pose to the user.

[0994] 4. The emotion engine recognizes anxious emotions from the user's facial expressions and voice.

[0995] 5. The server displays a message warning you that "This contract has a high penalty. Be careful" and offering additional details or support.

[0996] An example of an input prompt for the generative AI model is, "Please analyze the image of the contract below and assess the risks. Also, if the user feels anxious, please provide advice appropriate to their feelings."

[0997] This system helps users understand the contents of contracts and terms of use in a short amount of time, reducing the risk of suffering disadvantages. In addition, the introduction of an emotion engine provides optimal advice and support based on the user's emotions, allowing users to review contract details with greater peace of mind. This greatly simplifies the process of reviewing complex contracts and terms of use, improving users' sense of security.

[0998] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0999] Step 1:

[1000] The user takes a picture of the contract or terms of use, or uploads an existing screenshot or URL to the application. Specifically, when the user launches the app and taps the "Upload contract" button, the camera on the smartphone or tablet is activated. The user then takes a picture of the contract with the camera and presses the "Upload" button. Similarly, in the case of a screenshot or URL, the user is presented with the option to "Select file" or "Enter URL," and selects one of the options.

[1001] Input: Image data, screenshots, or URLs of contracts or terms of use

[1002] Output: Image data and URLs are imported to the device

[1003] Step 2:

[1004] The device sends the captured image or the provided URL to the server. Specifically, the device obtains the image data or URL data captured or uploaded by the user and sends it to the server via the Internet. An HTTP POST request is used to send the data to the endpoint.

[1005] Input: Image data or URL (data imported to the device)

[1006] Output: Image data and URL are sent to the server.

[1007] Step 3:

[1008] The server applies optical character recognition (OCR) technology to the received image data to convert the characters in the image into text data. Specifically, the server runs an OCR engine such as Tesseract to convert the characters in the image data into text format. Also, if a URL is provided, the server uses a web scraping tool such as BeautifulSoup or Scrapy to extract the text data from the page.

[1009] Input: Image data or URL sent

[1010] Output: Extracted text data

[1011] Step 4:

[1012] The server pre-processes the retrieved text data, removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs. Specifically, text cleaning algorithms are run, regular expressions are used to remove whitespace and special characters, and natural line and paragraph breaks are implemented.

[1013] Input: Extracted text data

[1014] Output: Cleaned text data

[1015] Step 5:

[1016] The server extracts important keywords and phrases from the text data using a predefined list of risk keywords. Specifically, the server uses a Python NLP library (e.g., NLTK or spaCy) to compare the list of risk keywords with the text data and perform the keyword extraction process.

[1017] Input: Cleaned text data

[1018] Output: Extracted important keywords and phrases

[1019] Step 6:

[1020] The server performs a risk assessment of clauses containing the extracted keywords. Specifically, an AI predictive model (e.g., TensorFlow or PyTorch) analyzes text containing risk keywords and calculates and assigns a risk score to each clause. The assessment results are stored in a database.

[1021] Input: Extracted important keywords and phrases

[1022] Output: Risk assessment score and assessment result

[1023] Step 7:

[1024] The server is equipped with an emotion engine that analyzes emotions from the user's voice, facial expressions, and input text. Specifically, it collects audio data from the webcam or smartphone microphone and runs an emotion recognition algorithm to analyze it (e.g., OpenCV is used for facial recognition, and Google Cloud Speech-to-Text API is used for voice analysis). The user's emotions are identified and stored in a database.

[1025] Input: Audio data, image data, input text

[1026] Output: Parsed emotion data

[1027] Step 8:

[1028] The server reflects the emotional information recognized by the emotion engine in the evaluation results and notification content. Specifically, it adjusts the tone of the risk assessment and notification message based on the emotional data. For example, for a user who expresses anxiety, it generates a warning message such as, "This contract has a high penalty fee. Please be careful."

[1029] Input: Risk assessment score, parsed sentiment data

[1030] Output: Adjusted evaluation results and notification messages

[1031] Step 9:

[1032] The server generates a risk report based on the evaluation results and emotional information. Specifically, it generates a risk report template that integrates the evaluation scores and emotional data, and creates a report in PDF or HTML format that visually highlights high-risk areas.

[1033] Input: Adjusted evaluation results and notification messages

[1034] Output: Generated risk report

[1035] Step 10:

[1036] The server sends the generated risk report to the terminal, which then displays it to the user. Specifically, the generated risk report is delivered to the terminal and displayed so that the user can view it on the application. There is a notification function to notify the user of the receipt of the risk report.

[1037] Input: Generated risk report

[1038] Output: Risk report displayed to the user

[1039] (Application example 2)

[1040] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1041] In modern digital services, many users agree to contracts and terms of use without properly understanding them. This increases the risk of signing unfavorable contracts. However, the content of the terms of use is complex, requiring time and effort for users to understand them. Furthermore, there is a lack of mechanisms to concretely recognize abstract risks or to support emotional decision-making. Therefore, there is a need for a method to more easily understand the risks in terms of use and contracts and provide emotional feedback.

[1042] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading an image of a contract or terms of use, means for converting the uploaded image into text data using optical character recognition technology, means for extracting important keywords from the converted text data, means for evaluating risks that may cause harm to the user based on the extracted keywords, means for notifying the user of the evaluation results, means for analyzing the user's emotions and optimizing the notification content based on the determined emotions, and means for generating a risk report based on the evaluation results and emotion information and transmitting the report to the terminal. This enables the user to quickly understand the contents of the contract or terms of use and receive optimal advice tailored to their emotions.

[1043] A "contract" is a document that describes the terms and conditions of a legal agreement.

[1044] "Terms of Use" means a document that sets out the conditions and rules for the use of a digital service or product.

[1045] "Means" are the methods or techniques used to achieve a particular goal.

[1046] "Means for uploading images" refers to a function that allows a user to send a specific image file to the system through a device.

[1047] "Optical character recognition technology" is a technology that converts characters in an image into digital text.

[1048] "Text data" is data that represents character information in a digital format.

[1049] "Important keywords" are words that are particularly noteworthy because they indicate specific content or risks.

[1050] "Keyword extraction means" refers to a technique for selecting specific words or phrases from text data.

[1051] The "means for assessing risk" is a method for analyzing the extracted keywords and determining the potential risks associated with them.

[1052] The "means for notifying the user of the evaluation results" is a function for displaying or communicating the results of the risk evaluation to the user.

[1053] "Means for analyzing emotions" refers to technology that recognizes and evaluates emotions from the user's voice, facial expressions, text input, etc.

[1054] "Means for optimizing notification content" refers to a method for adjusting the content and tone of a message based on the user's emotions.

[1055] A "risk report" is a document that compiles assessed risk information and advice.

[1056] "Terminal" refers to a digital device operated by a user.

[1057] The following describes an embodiment of the present invention.

[1058] The server has a function that allows users to upload images of contracts or terms of use to the server using their devices. The system starts when a user takes a photo of a contract and uploads the image using a smartphone or tablet.

[1059] The server then applies optical character recognition (OCR) technology to the uploaded image, converting the textual information in the image into digital text data. The specific software used is "pytesseract." The text extracted from the image is then entered into the system.

[1060] In the stage of extracting important keywords from the converted text data, analysis is performed based on a predefined list of risk keywords. For example, risk-related words such as "penalty" and "forced termination" are listed and extracted from the text data. Natural language processing (NLP) technology is used to achieve this function.

[1061] The server then evaluates the risk of harm to the user based on the extracted keywords. This risk evaluation uses a predictive model using a generative AI model to assign a risk score to each clause, which clarifies the risk level that the user should be aware of.

[1062] After the evaluation results are obtained, a method is used to analyze the user's emotions. This emotion analysis utilizes an emotion engine that analyzes the user's voice and facial expressions. For example, software such as "EmotionRecognition" is used to analyze the user's emotions. The content of notifications is optimized based on the emotional information.

[1063] Finally, a risk report is generated based on the evaluation results and sentiment information. This risk report highlights high-risk clauses and includes an evaluation score and advice. It also generates a warning message based on the overall risk situation. The server sends this risk report to the terminal, which then displays it to the user.

[1064] As a concrete example, consider the case where a user takes a screenshot of an online shopping website's terms of use and uploads the image to an app. The device sends the image to a server, which extracts text from the image. The server then analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk they pose to the user. The emotion engine recognizes anxious emotions from the user's facial expressions and voice, and the server displays a message warning the user that "This contract has a high penalty fee. Please be careful," along with suggestions for additional details or support.

[1065] An example prompt for a generative AI model might look something like this:

[1066] > "Read the user's sentiment and if they are unsure about the contract, generate advice like this:

[1067] "These terms of service have significant penalties. Please see additional details."

[1068] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1069] Step 1:

[1070] A user takes a picture of a contract or terms of use and uploads it to a server using their device. At this point, the input is the captured image file and the output is the image sent to the server.

[1071] Step 2:

[1072] The server applies optical character recognition (OCR) technology to the uploaded image and converts the characters in the image into digital text data. Specifically, it uses the pytesseract library to analyze the image file. The input is the image file, and the output is the converted text data.

[1073] Step 3:

[1074] The server extracts important keywords from the converted text data based on a predefined risk keyword list. It uses NLP technology to identify risk-related words and phrases. The input is the text data converted by OCR, and the output is the extracted keyword list.

[1075] Step 4:

[1076] The server evaluates the risk of posing a risk to the user based on the extracted keywords. It uses a generative AI model to calculate a risk score and determine the risk level of each keyword. The input is the extracted keyword list, and the output is the analysis result with a risk score.

[1077] Step 5:

[1078] When the user reads the contract, the server uses an emotion analysis engine to analyze the user's emotions. Specifically, the server analyzes the user's facial expressions and voice using emotion recognition software (e.g., EmotionRecognition). The input is the user's facial expressions and voice data, and the output is the determined emotional information.

[1079] Step 6:

[1080] The server optimizes the notification content for the user based on the results of risk assessment and sentiment analysis. It uses a generative AI model to generate an appropriate message based on the sentiment. A specific example includes a message such as, "This contract has a high penalty fee. Please be careful." The input is the risk assessment result and sentiment information, and the output is an optimized message.

[1081] Step 7:

[1082] The server generates a risk report based on the evaluation results and sentiment information. The report highlights high-risk clauses and provides an evaluation score and advice. The input is the optimized message and the risk evaluation results, and the output is the generated risk report.

[1083] Step 8:

[1084] The server sends the generated risk report to the terminal, which displays it to the user, allowing the user to easily understand the contents of the contract and its potential risks. The input is the risk report, and the output is a detailed report displayed to the user.

[1085] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1086] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1087] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1088] [Fourth embodiment]

[1089] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1090] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1091] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1092] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1093] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1094] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1095] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1096] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1097] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1098] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1099] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1100] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1101] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1102] This invention is a system that makes contracts and terms of use easier for users to understand, reducing the risk of signing a disadvantageous contract. Next, we will create a program for this system and explain the program's processing in natural language.

[1103] First, users take a photo of the contract or terms of use or prepare a screenshot of an existing one, then upload the image to a dedicated application. Users can also upload online contracts or terms of use by entering the URL into the same application.

[1104] The device then sends the uploaded image or URL to a server, which uses optical character recognition (OCR) technology to extract character information from the image data and converts it into text data. If a URL is sent, the server uses web scraping technology to retrieve the text of the contract or terms of use from the webpage.

[1105] The converted text data is then analyzed by the server, which uses natural language processing (NLP) technology to extract important keywords and phrases from the text of the contract or terms of use. These keywords include words related to risk, such as "penalty fee" and "forced termination." Based on these keywords, the server evaluates whether the contract contents may be disadvantageous to the user.

[1106] The server sends the details of the assessed risks and the results of the assessment to the device. The device then notifies the user of the assessment results and highlights any parts of the contract or terms of use that require particular attention. Detailed risk assessment comments and advice are also displayed, allowing the user to specifically understand the contract contents.

[1107] As a concrete example, a user takes a photo of the terms of use for a new service with their smartphone and uploads the image to an app. The device then sends the image to a server, which extracts text from the image. The server then analyzes the extracted text for important keywords such as "penalty fee" and "service suspension," and evaluates the risks these keywords pose to the user. The evaluation results are then sent to the device, which then displays a warning to the user, such as "This contract has a high penalty fee. Please be careful."

[1108] In this way, users can quickly understand the contents of contracts and terms of use, reducing the risk of suffering disadvantages. This system significantly simplifies the process of checking complex contracts and terms of use, improving users' peace of mind.

[1109] The processing flow will be explained below.

[1110] Step 1:

[1111] Users can take a picture of the contract or terms of use, or upload an existing screenshot / URL to the application.

[1112] Step 2:

[1113] The device sends the uploaded image and URL to the server.

[1114] Step 3:

[1115] The server performs an optical character recognition (OCR) process on the received image data to convert the characters in the image into text data, or, if a URL is provided, uses web scraping techniques to retrieve the content of the web page and convert it into text data.

[1116] Step 4:

[1117] The server pre-processes the retrieved text data, removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs.

[1118] Step 5:

[1119] The server uses a predefined list of risk keywords to extract important keywords and phrases from the text data, including "penalty fee," "forced termination," and "venue."

[1120] Step 6:

[1121] The server uses natural language processing (NLP) technology to perform a risk assessment of clauses containing the extracted keywords, using an AI-powered predictive model to assign a risk score to each clause.

[1122] Step 7:

[1123] The server generates a risk report for the user based on the AI ​​risk assessment results. High-risk clauses are highlighted, and an assessment score and advice are attached. A warning message based on the overall risk status is also generated.

[1124] Step 8:

[1125] The server sends the generated risk report to the terminal.

[1126] Step 9:

[1127] The terminal displays the risk report received from the server to the user. High-risk clauses are highlighted in red, and specific risk details and warnings are displayed as pop-ups.

[1128] Example 1

[1129] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1130] Conventional contracts and terms of use are often complex and full of technical terms, making them difficult for average users to understand. This puts users at risk of overlooking content that could be detrimental to them. Furthermore, reviewing contracts and terms of use is time-consuming and inefficient. This invention aims to enable users to quickly and easily understand the contents of contracts and terms of use, thereby reducing the risk of suffering disadvantages.

[1131] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1132] In this invention, the server includes means for uploading a digital image of a contract or terms of use, means for converting the uploaded digital image into character string data using optical character recognition technology, means for extracting important keywords from the character string data, means for analyzing the extracted keywords using natural language processing technology and evaluating the risk of causing harm to the user, and means for notifying the user of the evaluation result and visually highlighting parts requiring particular attention. This enables the user to quickly understand the contents of the contract or terms of use and efficiently identify any disadvantageous conditions.

[1133] "Agreement or Terms of Use" means the legal document to which a User agrees when using a Service or purchasing a Product.

[1134] "Digital image" refers to image data obtained by photographing the contents of a contract or terms of use with a camera or scanner.

[1135] "Optical character recognition technology" is a technology that reads character information from digital images and converts it into text data.

[1136] "String data" means data that expresses the contents of a contract or terms of use in text format, converted using optical character recognition technology.

[1137] "Keywords" are words or phrases that are considered important to users in analyzing the content of a contract or terms of use.

[1138] "Natural language processing technology" refers to information processing technology used to understand and analyze human language.

[1139] A "generative AI model" is a model that uses artificial intelligence techniques to learn from training data and perform specific tasks.

[1140] A "web resource identifier" is an address (such as a URL) that uniquely identifies content on the Internet.

[1141] The "evaluation result" refers to the result of the risk assessment obtained by the server by analyzing keywords and text data.

[1142] A "warning message" is a message generated to notify a user of a risk based on the evaluation results.

[1143] "Visual emphasis" refers to the means of highlighting specific text or areas to help users intuitively understand important information.

[1144] This invention is a system that makes it easier for users to understand the contents of contracts and terms of use. In order to reduce the risk of users signing unfavorable contracts, the system analyzes uploaded contract and terms of use data and performs a series of processes to evaluate the risk.

[1145] Users can take a digital image of the contract or terms of use, or prepare an existing screenshot, or if the contract or terms of use is online, they can upload it by entering the URL into a dedicated application.

[1146] The device sends image data and URLs uploaded by the user to the server. The server applies optical character recognition (OCR) technology to the received image data and converts it into text data. Possible OCR technologies include Google Cloud Vision and Tesseract. If a URL is sent, the server uses web scraping technology to retrieve the text of the contract or terms of use from the webpage. BeautifulSoup and Selenium are used for this process.

[1147] The server then analyzes the converted text data and uses natural language processing (NLP) techniques to extract important keywords and phrases from the text of the contract or terms of use. SpaCy and NLTK are used as NLP tools. These keywords include risk-related words such as "penalty fee" and "forced termination." The server then uses a generative AI model to perform a risk assessment based on the extracted keywords. The generative AI model is fed the analyzed text data and keywords along with a prompt like the following:

[1148] "This agreement contains the following keywords: penalty, termination. Please assess the risk these pose to you."

[1149] The server sends the evaluation results to the device. The device then notifies the user based on the evaluation results and visually highlights any areas requiring particular attention. For example, it highlights specific contract terms or comments regarding risks and displays a warning message to the user. This allows the user to quickly understand the contents of the contract or terms of use and reduce the risk of suffering disadvantages.

[1150] A specific example is a process in which a user takes a photo of the terms of use for a new service with their smartphone and uploads the image to an app. In this case, the device sends the image to a server, which extracts text data from the image. The server analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk they pose to the user. The evaluation results are sent to the device, which then displays a warning to the user, such as "This contract has a high penalty fee. Please be careful."

[1151] As described above, the system of the present invention allows users to understand the contents of complex contracts and terms of use in a short period of time, thereby improving the user's sense of security.

[1152] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1153] Step 1:

[1154] The user prepares digital images of the contract and terms of use.

[1155] Users can take a picture of the contract or terms of use using their smartphone's camera or scanner, or they can use an existing screenshot.If the contract or terms of use is online, they can also enter its URL into a dedicated application.

[1156] Input: Digital image or URL of the contract or terms of use

[1157] Output: Image file or URL of the contract or terms of use

[1158] Specific behavior:

[1159] The user takes a photo of the contract with their smartphone camera, saves the image, and then copies the URL of the online terms of use and pastes it into a dedicated application.

[1160] Step 2:

[1161] The device sends image data and URLs to the server.

[1162] The device sends the digital images and URLs uploaded by the user to the server.

[1163] Transfer this data to the system's API using an HTTP POST request.

[1164] Input: Image file or URL of the contract or terms of use

[1165] Output: Image file or URL sent to the server

[1166] Specific behavior:

[1167] The device sends a POST request to http: / / example.com / api / upload with the image file selected by the user, or sends a URL entered by the user to the same endpoint.

[1168] Step 3:

[1169] The server converts the image data into character string data.

[1170] The server applies optical character recognition (OCR) technology to the received image data, converting the text information in the image into string data. The technologies used are Google Cloud Vision and Tesseract. When a URL is submitted, the server uses web scraping technology, such as BeautifulSoup and Selenium, to extract the text of the contract or terms of use.

[1171] Input: Image file or URL

[1172] Output: Converted string data

[1173] Specific behavior:

[1174] The server sends the image to the Google Cloud Vision API for OCR processing, and also uses Selenium to access the URL and retrieve the contract and terms of use text.

[1175] Step 4:

[1176] The server analyzes the string data and extracts important keywords.

[1177] The server then analyzes the converted text data and extracts important keywords and phrases using natural language processing (NLP) techniques, such as SpaCy and NLTK.

[1178] Input: Converted string data

[1179] Output: Extracted important keywords

[1180] Specific behavior:

[1181] The server analyzes the string data using SpaCy to identify risk-related keywords (e.g., penalty fees, forced termination), and generates a list of important keywords as a result of the analysis.

[1182] Step 5:

[1183] The server performs risk assessment using a generative AI model

[1184] The server performs risk assessment using a generative AI model based on the extracted keywords. It passes input data along with prompt statements to the model and obtains the assessment results.

[1185] Input: Extracted important keywords, prompt sentence

[1186] Output: Risk assessment results

[1187] Specific behavior:

[1188] The server inputs the following prompt to the generative AI model: "This contract contains the following keywords: penalty fee, forced termination. Please evaluate what risks these pose to the user." The server then obtains the evaluation result returned by the AI ​​model.

[1189] Step 6:

[1190] The server sends the evaluation results to the terminal.

[1191] The server transmits the generated risk assessment results to the terminal.

[1192] Input: Risk assessment results

[1193] Output: Evaluation results sent to the device

[1194] Specific behavior:

[1195] The server packages the risk assessment results in JSON format and sends them to the terminal using the sendResponse method.

[1196] Step 7:

[1197] The device notifies the user of the evaluation results and highlights them visually.

[1198] The device notifies the user based on the evaluation results received from the server, visually highlighting any parts of the evaluation results that require special attention.

[1199] Input: Risk assessment results

[1200] Output: Evaluation results notified to the user, with visually highlighted notes

[1201] Specific behavior:

[1202] The device displays the evaluation results on the user interface, highlights specific keywords (e.g., penalty fees), and issues a warning to the user via a pop-up message.

[1203] Through the above processing steps, the system quickly and accurately analyzes the contents of contracts and terms of use, and provides users with a risk assessment in an easy-to-understand format.

[1204] (Application example 1)

[1205] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1206] There is a need to reduce the risk of users signing contracts and terms of use without fully understanding them. However, especially in the case of electronic payment services, terms of use are often complex and lengthy, making them difficult for users to understand. As a result, users may suffer disadvantages. To solve this problem, a system is needed that allows users to intuitively understand the contents of the contract and immediately notifies them of particularly high-risk parts.

[1207] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1208] In this invention, the server includes means for uploading an image of the contract or terms of use, means for converting the uploaded image into text data using optical character recognition technology, means for extracting important keywords from the converted text data, means for evaluating the risk of harm to the user based on the extracted keywords, means for notifying the user of the evaluation result, means for notifying the user of the evaluation result in real time, and means for integrating the harm risk evaluation into the electronic payment environment. This allows the user to quickly understand the risks in the contract content and reduce the risk of suffering harm.

[1209] "Agreement or Terms of Use" means a document between a user and a provider regarding the use of a service or purchase of a product, which defines the terms of use and rights and obligations.

[1210] "Images" are visual information converted into digital data, and refer to the contents of contracts and terms of use as photographs or scanned data.

[1211] "Uploading" is the operation by which a user sends digital data to a server or cloud service via the Internet.

[1212] Optical character recognition (OCR) is a technology that reads character information contained in image data and converts it into text data.

[1213] "Text data" is digital information written in text format, and is data in a form that is easy to search and analyze.

[1214] "Important keywords" are words or phrases in the contract that may have a significant impact on the user.

[1215] "Extraction" is the operation of extracting necessary information or elements from text data.

[1216] "Risk assessment" is a process of analyzing and determining the possibility that a user may suffer a disadvantage based on the extracted keywords.

[1217] "Real-time notification" refers to providing analysis results immediately after the user provides data.

[1218] "Integration into the electronic payment environment" refers to a system that automatically incorporates risk analysis functions when conducting online financial transactions and promptly notifies users.

[1219] This invention is a system that enables users to quickly understand the contents of contracts and terms of use, reducing the risk of suffering disadvantages. Specifically, the system uploads images or URLs of contracts and terms of use, converts them into text data, and analyzes them using natural language processing technology.

[1220] System Configuration

[1221] Hardware: Smartphones, smart glasses, servers

[1222] Software: OCR technology (e.g., Tesseract), web scraping tools (e.g., BeautifulSoup), natural language processing (e.g., spaCy)

[1223] Operation overview

[1224] 1. Obtaining contract and terms of use data:

[1225] The user takes a picture of the contract or terms of use using a smartphone or smart glasses and uploads it to the application.

[1226] The user enters the URL of the online contract or terms of use into the application and uploads it.

[1227] The terminal transmits the input image or URL to the server.

[1228] 2. Data conversion and text generation:

[1229] The server converts the uploaded image into text data using optical character recognition (OCR) technology, which uses Tesseract.

[1230] When the URL is uploaded, the server uses a web scraping tool to extract the text data of the contract or terms of use from the web page. For example, we use BeautifulSoup.

[1231] 3. Natural Language Processing and Risk Assessment:

[1232] The server then uses natural language processing (NLP) techniques on the converted text data to extract important keywords and phrases. This process is performed using spaCy.

[1233] Based on the extracted keywords, the risk that the contract contents will cause disadvantage to the user is evaluated.

[1234] Based on the results of the risk assessment, necessary precautions and warnings are generated.

[1235] 4. Real-time notifications:

[1236] The server transmits the evaluation results to the terminal in real time.

[1237] The device notifies the user of the evaluation results and highlights areas requiring attention.

[1238] Usage example

[1239] The user takes a photo of the terms of use for a new service with their smartphone and uploads the image to the application. The device sends the image to the server, which extracts text from the image. The server analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk these keywords pose to the user. The evaluation results are sent to the device, which then displays a warning to the user, such as "This contract has a high penalty fee. Please be careful."

[1240] Prompt Sentence Examples

[1241] "Do the new terms of service agreements include penalties? And what are the risks to users?"

[1242] By using such a system, users can quickly understand the contents of the contract and reduce the risk of suffering disadvantages.

[1243] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1244] Step 1:

[1245] Users upload images of contracts and terms of use

[1246] The user takes an image of the contract or terms of use using a smartphone or smart glasses. Then, the user uploads the image or screenshot to the application. The uploaded image is sent to the server.

[1247] Input: Image of contract or terms of use

[1248] Output: Image data sent to the server

[1249] Step 2:

[1250] User uploads URL of contract or terms of use

[1251] Users enter the URL of the online contract or terms of use into a dedicated application and upload it. The entered URL is sent to the server.

[1252] Input: URL of contract or terms of use

[1253] Output: URL data sent to the server

[1254] Step 3:

[1255] The server converts image data into text data

[1256] The server uses optical character recognition (OCR) technology to extract character information from uploaded images and convert them into text data. The OCR technology used is Tesseract.

[1257] Input: Uploaded image data

[1258] Output: Text data

[1259] Step 4:

[1260] The server scrapes the web page content from the URL.

[1261] The server retrieves the contract and terms of use from the uploaded URL using web scraping technology, which uses BeautifulSoup to extract the text from the web page.

[1262] Input: Uploaded URL

[1263] Output: Text data

[1264] Step 5:

[1265] The server uses natural language processing technology to extract important keywords

[1266] The server then uses natural language processing (NLP) technology on the converted or scraped text data to extract important keywords and phrases. The NLP technology used is spaCy.

[1267] Input: Text data

[1268] Output: Extracted keywords and phrases

[1269] Step 6:

[1270] The server assesses the risk

[1271] The server uses the extracted keywords and phrases to assess the risk that the contract content will cause disadvantage to the user, using a generative AI model for the assessment.

[1272] Input: Extracted keywords or phrases

[1273] Output: Risk assessment results

[1274] Step 7:

[1275] The server notifies the evaluation results in real time

[1276] The server transmits the risk assessment results in real time to the user's terminal, which then notifies the user of the results.

[1277] Input: Risk assessment results

[1278] Output: The evaluation results displayed to the user

[1279] Step 8:

[1280] The device displays risk information to the user

[1281] Based on the evaluation results received, the device highlights and displays parts of the contract or terms of use that require particular attention, and also provides risk assessment comments and advice.

[1282] Input: Risk assessment results

[1283] Output: Highlighted contracts and terms of use, risk assessment comments

[1284] Prompt Sentence Examples

[1285] "Do the new terms of service agreements include penalties? And what are the risks to users?"

[1286] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1287] This invention is a system that makes contracts and terms of use easier for users to understand, reducing the risk of signing a disadvantageous contract, and also combines it with an emotion engine that recognizes the user's emotions and optimizes the information provided based on those emotions. Below, we will explain the program of this system in detail, and explain each processing step in natural language.

[1288] First, the user takes a picture of the contract or terms of use, or uploads an existing screenshot or URL to the application. The device then sends the image or URL to the server. The server then performs optical character recognition (OCR) on the received image data and converts the characters in the image into text data. If a URL is provided, the server uses web scraping technology to convert the contents of the web page into text data.

[1289] The server then pre-processes the retrieved text data, removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs. The server then uses a pre-defined risk keyword list to extract important keywords and phrases from the text data, including risk-related words such as "penalty" and "forced termination."

[1290] The server then uses natural language processing (NLP) technology to perform a risk assessment of clauses containing the extracted keywords. An AI-based predictive model is used for the risk assessment, and a risk score is assigned to each clause. The system also incorporates an emotion engine that can recognize the user's emotions. The emotion engine analyzes emotions from the user's voice, facial expressions, and input text to determine how the user feels about the contract or terms of use.

[1291] The emotion information recognized by the emotion engine is reflected in the assessment results and notification content by the server. For example, if the user expresses anxiety or doubt, the server can provide additional advice or detailed explanations. The server can also adjust the display format of the risk assessment and the tone of the notification message based on the emotion information.

[1292] Based on the evaluation results and sentiment information, the server generates a risk report for the user. High-risk clauses are highlighted and an evaluation score and advice are attached. A warning message based on the overall risk status is also generated. The server sends the generated risk report to the terminal, which displays it to the user.

[1293] As a concrete example, a user takes a photo of the terms of use for a new service and uploads the image to the app. The device sends the image to a server, which extracts text from the image. The server analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk these pose to the user. The emotion engine recognizes anxious emotions from the user's facial expressions and voice, and the server displays a message warning them that "This contract has a high penalty fee. Please be careful," along with suggestions for additional details or support.

[1294] In this way, users can quickly understand the contents of contracts and terms of use, reducing the risk of suffering disadvantages. Furthermore, the introduction of an emotion engine provides optimal advice and support based on the user's emotions, allowing users to review contract details with greater peace of mind. This system significantly simplifies the process of reviewing complex contracts and terms of use, increasing users' sense of security.

[1295] The processing flow will be explained below.

[1296] Step 1:

[1297] Users can take a picture of the contract or terms of use, or upload an existing screenshot / URL to the application using a smartphone or tablet.

[1298] Step 2:

[1299] The device sends the uploaded image and URL to the server, where the data is encrypted and securely transferred to the server.

[1300] Step 3:

[1301] The server uses optical character recognition (OCR) technology to convert the characters in the image into text data, and if a URL is provided, uses web scraping technology to retrieve the text of the contract or terms of use from the web page.

[1302] Step 4:

[1303] The server preprocesses the converted text data, specifically removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs. This step improves the quality of the text data.

[1304] Step 5:

[1305] The server extracts important keywords and phrases from the text data, using a predefined list of risk keywords to detect specific words (e.g., "penalty fee," "forced termination," "venue," etc.).

[1306] Step 6:

[1307] The server uses natural language processing (NLP) techniques to perform a risk assessment on clauses containing the extracted keywords, using an AI model to assign a risk score to each clause.

[1308] Step 7:

[1309] The server recognizes the user's emotions through an emotion engine, analyzing facial expressions, voice data, and input text when the user operates the application. Emotions are classified as anxiety, doubt, relief, etc.

[1310] Step 8:

[1311] The server adjusts the evaluation results and notification content based on the emotional information recognized by the emotion engine. For example, if the user expresses anxiety, it generates a notification with a detailed explanation and additional support information.

[1312] Step 9:

[1313] The server generates a risk report based on the evaluation results and sentiment information. High-risk clauses are highlighted and accompanied by an evaluation score and advice. Warning messages are also generated based on the overall risk situation.

[1314] Step 10:

[1315] The server sends the generated risk report to the terminal.

[1316] Step 11:

[1317] The terminal displays the risk report received from the server to the user. High-risk clauses are highlighted in red, and specific risk details and warnings are displayed as pop-ups.

[1318] This process allows users to quickly understand the contents of the contract and terms of use, reducing the risk of suffering disadvantages. Furthermore, appropriate advice and support tailored to the user's feelings are provided, allowing users to review the contract contents with greater peace of mind.

[1319] Example 2

[1320] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1321] Conventional contracts and terms of use are difficult for ordinary users to understand, increasing the risk of them signing disadvantageous contracts. Furthermore, because they do not take into account the emotions users feel toward these documents, necessary support and advice are lacking. Therefore, there is a need for a system that makes contracts and terms of use easier to understand and provides appropriate information that takes users' emotions into account.

[1322] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1323] In this invention, the server includes means for uploading an image of a contract or terms of use, means for converting the uploaded image into text data using optical character recognition technology, means for extracting important keywords from the converted text data, means for evaluating the risk of causing harm to the user based on the extracted keywords, means for analyzing emotions from the user's voice, facial expression, input text, etc., means for adjusting the evaluation result and notification content based on the analyzed emotional information, and means for notifying the user of the evaluation result. This makes it possible to objectively evaluate the risks of the contract or terms of use and provide appropriate information and support according to the user's emotions.

[1324] "Agreement" or "Terms of Use" means a document that contains the formal terms and conditions governing the use of a service and that clarifies the rights and obligations of both parties.

[1325] "Means for uploading images" is a function that allows a user to use a terminal to send image data of a contract or terms of use to a server.

[1326] "Optical character recognition technology" refers to the technology that analyzes characters in image data and converts them into text data.

[1327] The "means for converting into text data" is a function that uses optical character recognition technology to extract character information from image data and convert it into digital text format.

[1328] "Means for extracting important keywords" refers to the ability to identify and extract words and phrases related to specific risks or important matters from text data.

[1329] The "means for assessing the risk of causing harm" is a function that analyzes and assesses the risks that the clauses of a contract or terms of use may pose to the user, based on the extracted keywords.

[1330] "Means for analyzing emotions from a user's voice, facial expressions, input text, etc." refers to technology that analyzes audio, video, and text data collected through input devices such as microphones and cameras to identify the user's emotional state.

[1331] The "means for adjusting evaluation results and notification content" is a function for adjusting the content and tone of evaluation results and notification messages provided to users based on the results of emotion analysis.

[1332] The "means for notifying the user of the evaluation results" is a function for sending the results of risk evaluation and advice to the terminal and displaying them to the user.

[1333] "Method of scraping the contents of a contract or terms of use from a URL to obtain text data" refers to a technology that analyzes the contents of a web page and extracts the text data of a contract or terms of use.

[1334] "Natural language processing technology" is a technology that allows computers to understand and analyze human language, and involves classifying text data and performing semantic analysis.

[1335] This invention is a system that makes contracts and terms of use easier for users to understand, reducing the risk of signing a disadvantageous contract, and also combines an emotion engine that recognizes the user's emotions and optimizes the information provided accordingly. The program processing of this system is explained in detail below.

[1336] First, the user takes a picture of the contract or terms of use, or uploads an existing screenshot or URL to the application. This is done using a device such as a smartphone or tablet. The device then sends the image and URL data to the server.

[1337] The server applies optical character recognition (OCR) technology to the received image data to convert the characters in the image into text data using an OCR engine such as Tesseract. If a URL is provided, the server uses a web scraping technology such as BeautifulSoup or Scrapy to convert the contents of the web page into text data.

[1338] The server then pre-processes the retrieved text data, removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs using a text cleaning algorithm based on regular expressions.

[1339] The server then extracts important keywords and phrases from the text data using a predefined list of risk keywords, including risk-related words such as "penalty" and "cancellation." This process is performed using a Python NLP library (e.g., NLTK or spaCy).

[1340] The server then uses natural language processing (NLP) technology to perform a risk assessment of clauses containing the extracted keywords, using an AI-powered predictive model to assign a risk score to each clause using TensorFlow and PyTorch.

[1341] The system also incorporates an emotion engine that analyzes emotions from the user's voice, facial expressions, input text, etc. The emotion engine uses data collected from the webcam and smartphone microphone to recognize emotions using OpenCV and the Google Cloud Speech-to-Text API.

[1342] The emotion information recognized by the emotion engine is reflected in the assessment results and notification content by the server. For example, if the user expresses anxiety or doubt, the server will provide additional advice or detailed explanations. The server will also adjust the display format of the risk assessment and the tone of the notification message based on the emotion information.

[1343] Based on the evaluation results and sentiment information, the server generates a risk report for the user. High-risk clauses are highlighted and an evaluation score and advice are attached. A warning message based on the overall risk status is also generated. The server sends the generated risk report to the terminal, which displays it to the user.

[1344] As a concrete example, if a user takes a photo of the terms of use for a new service and uploads the image to the app, the system operates as follows:

[1345] 1. The device sends the image to the server.

[1346] 2. The server extracts text from the image using OCR technology.

[1347] 3. The server analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk they pose to the user.

[1348] 4. The emotion engine recognizes anxious emotions from the user's facial expressions and voice.

[1349] 5. The server displays a message warning you that "This contract has a high penalty. Be careful" and offering additional details or support.

[1350] An example of an input prompt for the generative AI model is, "Please analyze the image of the contract below and assess the risks. Also, if the user feels anxious, please provide advice appropriate to their feelings."

[1351] This system helps users understand the contents of contracts and terms of use in a short amount of time, reducing the risk of suffering disadvantages. In addition, the introduction of an emotion engine provides optimal advice and support based on the user's emotions, allowing users to review contract details with greater peace of mind. This greatly simplifies the process of reviewing complex contracts and terms of use, improving users' sense of security.

[1352] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1353] Step 1:

[1354] The user takes a picture of the contract or terms of use, or uploads an existing screenshot or URL to the application. Specifically, when the user launches the app and taps the "Upload contract" button, the camera on the smartphone or tablet is activated. The user then takes a picture of the contract with the camera and presses the "Upload" button. Similarly, in the case of a screenshot or URL, the user is presented with the option to "Select file" or "Enter URL," and selects one of the options.

[1355] Input: Image data, screenshots, or URLs of contracts or terms of use

[1356] Output: Image data and URLs are imported to the device

[1357] Step 2:

[1358] The device sends the captured image or the provided URL to the server. Specifically, the device obtains the image data or URL data captured or uploaded by the user and sends it to the server via the Internet. An HTTP POST request is used to send the data to the endpoint.

[1359] Input: Image data or URL (data imported to the device)

[1360] Output: Image data and URL are sent to the server.

[1361] Step 3:

[1362] The server applies optical character recognition (OCR) technology to the received image data to convert the characters in the image into text data. Specifically, the server runs an OCR engine such as Tesseract to convert the characters in the image data into text format. Also, if a URL is provided, the server uses a web scraping tool such as BeautifulSoup or Scrapy to extract the text data from the page.

[1363] Input: Image data or URL sent

[1364] Output: Extracted text data

[1365] Step 4:

[1366] The server pre-processes the retrieved text data, removing unnecessary whitespace and special characters, and properly formatting line breaks and paragraphs. Specifically, text cleaning algorithms are run, regular expressions are used to remove whitespace and special characters, and natural line and paragraph breaks are implemented.

[1367] Input: Extracted text data

[1368] Output: Cleaned text data

[1369] Step 5:

[1370] The server extracts important keywords and phrases from the text data using a predefined list of risk keywords. Specifically, the server uses a Python NLP library (e.g., NLTK or spaCy) to compare the list of risk keywords with the text data and perform the keyword extraction process.

[1371] Input: Cleaned text data

[1372] Output: Extracted important keywords and phrases

[1373] Step 6:

[1374] The server performs a risk assessment of clauses containing the extracted keywords. Specifically, an AI predictive model (e.g., TensorFlow or PyTorch) analyzes text containing risk keywords and calculates and assigns a risk score to each clause. The assessment results are stored in a database.

[1375] Input: Extracted important keywords and phrases

[1376] Output: Risk assessment score and assessment result

[1377] Step 7:

[1378] The server is equipped with an emotion engine that analyzes emotions from the user's voice, facial expressions, and input text. Specifically, it collects audio data from the webcam or smartphone microphone and runs an emotion recognition algorithm to analyze it (e.g., OpenCV is used for facial recognition, and Google Cloud Speech-to-Text API is used for voice analysis). The user's emotions are identified and stored in a database.

[1379] Input: Audio data, image data, input text

[1380] Output: Parsed emotion data

[1381] Step 8:

[1382] The server reflects the emotional information recognized by the emotion engine in the evaluation results and notification content. Specifically, it adjusts the tone of the risk assessment and notification message based on the emotional data. For example, for a user who expresses anxiety, it generates a warning message such as, "This contract has a high penalty fee. Please be careful."

[1383] Input: Risk assessment score, parsed sentiment data

[1384] Output: Adjusted evaluation results and notification messages

[1385] Step 9:

[1386] The server generates a risk report based on the evaluation results and emotional information. Specifically, it generates a risk report template that integrates the evaluation scores and emotional data, and creates a report in PDF or HTML format that visually highlights high-risk areas.

[1387] Input: Adjusted evaluation results and notification messages

[1388] Output: Generated risk report

[1389] Step 10:

[1390] The server sends the generated risk report to the terminal, which then displays it to the user. Specifically, the generated risk report is delivered to the terminal and displayed so that the user can view it on the application. There is a notification function to notify the user of the receipt of the risk report.

[1391] Input: Generated risk report

[1392] Output: Risk report displayed to the user

[1393] (Application example 2)

[1394] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1395] In modern digital services, many users agree to contracts and terms of use without properly understanding them. This increases the risk of signing unfavorable contracts. However, the content of the terms of use is complex, requiring time and effort for users to understand them. Furthermore, there is a lack of mechanisms to concretely recognize abstract risks or to support emotional decision-making. Therefore, there is a need for a method to more easily understand the risks in terms of use and contracts and provide emotional feedback.

[1396] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for uploading an image of a contract or terms of use, means for converting the uploaded image into text data using optical character recognition technology, means for extracting important keywords from the converted text data, means for evaluating risks that may cause harm to the user based on the extracted keywords, means for notifying the user of the evaluation results, means for analyzing the user's emotions and optimizing the notification content based on the determined emotions, and means for generating a risk report based on the evaluation results and emotion information and transmitting the report to the terminal. This enables the user to quickly understand the contents of the contract or terms of use and receive optimal advice tailored to their emotions.

[1397] A "contract" is a document that describes the terms and conditions of a legal agreement.

[1398] "Terms of Use" means a document that sets out the conditions and rules for the use of a digital service or product.

[1399] "Means" are the methods or techniques used to achieve a particular goal.

[1400] "Means for uploading images" refers to a function that allows a user to send a specific image file to the system through a device.

[1401] "Optical character recognition technology" is a technology that converts characters in an image into digital text.

[1402] "Text data" is data that represents character information in a digital format.

[1403] "Important keywords" are words that are particularly noteworthy because they indicate specific content or risks.

[1404] "Keyword extraction means" refers to a technique for selecting specific words or phrases from text data.

[1405] The "means for assessing risk" is a method for analyzing the extracted keywords and determining the potential risks associated with them.

[1406] The "means for notifying the user of the evaluation results" is a function for displaying or communicating the results of the risk evaluation to the user.

[1407] "Means for analyzing emotions" refers to technology that recognizes and evaluates emotions from the user's voice, facial expressions, text input, etc.

[1408] "Means for optimizing notification content" refers to a method for adjusting the content and tone of a message based on the user's emotions.

[1409] A "risk report" is a document that compiles assessed risk information and advice.

[1410] "Terminal" refers to a digital device operated by a user.

[1411] The following describes an embodiment of the present invention.

[1412] The server has a function that allows users to upload images of contracts or terms of use to the server using their devices. The system starts when a user takes a photo of a contract and uploads the image using a smartphone or tablet.

[1413] The server then applies optical character recognition (OCR) technology to the uploaded image, converting the textual information in the image into digital text data. The specific software used is "pytesseract." The text extracted from the image is then entered into the system.

[1414] In the stage of extracting important keywords from the converted text data, analysis is performed based on a predefined list of risk keywords. For example, risk-related words such as "penalty" and "forced termination" are listed and extracted from the text data. Natural language processing (NLP) technology is used to achieve this function.

[1415] The server then evaluates the risk of harm to the user based on the extracted keywords. This risk evaluation uses a predictive model using a generative AI model to assign a risk score to each clause, which clarifies the risk level that the user should be aware of.

[1416] After the evaluation results are obtained, a method is used to analyze the user's emotions. This emotion analysis utilizes an emotion engine that analyzes the user's voice and facial expressions. For example, software such as "EmotionRecognition" is used to analyze the user's emotions. The content of notifications is optimized based on the emotional information.

[1417] Finally, a risk report is generated based on the evaluation results and sentiment information. This risk report highlights high-risk clauses and includes an evaluation score and advice. It also generates a warning message based on the overall risk situation. The server sends this risk report to the terminal, which then displays it to the user.

[1418] As a concrete example, consider the case where a user takes a screenshot of an online shopping website's terms of use and uploads the image to an app. The device sends the image to a server, which extracts text from the image. The server then analyzes the extracted text for important keywords such as "penalty fee" and "service suspension" and evaluates the risk they pose to the user. The emotion engine recognizes anxious emotions from the user's facial expressions and voice, and the server displays a message warning the user that "This contract has a high penalty fee. Please be careful," along with suggestions for additional details or support.

[1419] An example prompt for a generative AI model might look something like this:

[1420] > "Read the user's sentiment and if they are unsure about the contract, generate advice like this:

[1421] "These terms of service have significant penalties. Please see additional details."

[1422] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1423] Step 1:

[1424] A user takes a picture of a contract or terms of use and uploads it to a server using their device. At this point, the input is the captured image file and the output is the image sent to the server.

[1425] Step 2:

[1426] The server applies optical character recognition (OCR) technology to the uploaded image and converts the characters in the image into digital text data. Specifically, it uses the pytesseract library to analyze the image file. The input is the image file, and the output is the converted text data.

[1427] Step 3:

[1428] The server extracts important keywords from the converted text data based on a predefined risk keyword list. It uses NLP technology to identify risk-related words and phrases. The input is the text data converted by OCR, and the output is the extracted keyword list.

[1429] Step 4:

[1430] The server evaluates the risk of posing a risk to the user based on the extracted keywords. It uses a generative AI model to calculate a risk score and determine the risk level of each keyword. The input is the extracted keyword list, and the output is the analysis result with a risk score.

[1431] Step 5:

[1432] When the user reads the contract, the server uses an emotion analysis engine to analyze the user's emotions. Specifically, the server analyzes the user's facial expressions and voice using emotion recognition software (e.g., EmotionRecognition). The input is the user's facial expressions and voice data, and the output is the determined emotional information.

[1433] Step 6:

[1434] The server optimizes the notification content for the user based on the results of risk assessment and sentiment analysis. It uses a generative AI model to generate an appropriate message based on the sentiment. A specific example includes a message such as, "This contract has a high penalty fee. Please be careful." The input is the risk assessment result and sentiment information, and the output is an optimized message.

[1435] Step 7:

[1436] The server generates a risk report based on the evaluation results and sentiment information. The report highlights high-risk clauses and provides an evaluation score and advice. The input is the optimized message and the risk evaluation results, and the output is the generated risk report.

[1437] Step 8:

[1438] The server sends the generated risk report to the terminal, which displays it to the user, allowing the user to easily understand the contents of the contract and its potential risks. The input is the risk report, and the output is a detailed report displayed to the user.

[1439] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1440] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1441] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1442] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1443] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1444] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1445] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1446] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1447] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1448] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1449] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1450] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1451] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1452] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1453] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1454] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1455] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1456] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1457] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1458] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1459] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1460] The following is further disclosed regarding the above embodiment.

[1461] (Claim 1)

[1462] A means to upload an image of the contract or terms of use;

[1463] means for converting the uploaded image into text data using optical character recognition technology;

[1464] A means for extracting important keywords from the converted text data;

[1465] A means for evaluating the risk of causing harm to a user based on the extracted keywords;

[1466] means for notifying the user of the evaluation result;

[1467] A system including:

[1468] (Claim 2)

[1469] 10. The system of claim 1, further comprising: means for scraping the content of the contract or terms of use from the uploaded URL to obtain text data.

[1470] (Claim 3)

[1471] 10. The system of claim 1, further comprising: means for evaluating the extracted keywords using natural language processing techniques.

[1472] "Example 1"

[1473] (Claim 1)

[1474] A means to upload a digital image of the contract or terms of use;

[1475] means for converting the uploaded digital image into character string data using optical character recognition technology;

[1476] A means for extracting important keywords from character string data;

[1477] A means for analyzing the extracted keywords using natural language processing technology and assessing the risk of causing harm to users;

[1478] a means for notifying the user of the evaluation results and visually highlighting areas that require particular attention;

[1479] A system including:

[1480] (Claim 2)

[1481] 10. The system of claim 1, further comprising: means for extracting the content of the contract or terms of use from the identifier of the uploaded web resource to obtain the string data.

[1482] (Claim 3)

[1483] 10. The system of claim 1, further comprising means for performing a risk assessment using the generative AI model.

[1484] (Claim 4)

[1485] 10. The system of claim 1, further comprising: means for generating a warning message based on the evaluation result and displaying the warning message to the user.

[1486] "Application Example 1"

[1487] (Claim 1)

[1488] A means to upload an image of the contract or terms of use;

[1489] means for converting the uploaded image into text data using optical character recognition technology;

[1490] A means for extracting important keywords from the converted text data;

[1491] A means for evaluating the risk of causing harm to a user based on the extracted keywords;

[1492] means for notifying the user of the evaluation result;

[1493] A means of notifying evaluation results in real time;

[1494] a means of integrating adverse risk assessment into electronic payment environments;

[1495] A system including:

[1496] (Claim 2)

[1497] 10. The system of claim 1, further comprising: means for scraping the content of the contract or terms of use from the uploaded URL to obtain text data.

[1498] (Claim 3)

[1499] 10. The system of claim 1, further comprising: means for evaluating the extracted keywords using natural language processing techniques.

[1500] "Example 2: Combining Emotion Engines"

[1501] (Claim 1)

[1502] A means to upload an image of the contract or terms of use;

[1503] means for converting the uploaded image into text data using optical character recognition technology;

[1504] A means for extracting important keywords from the converted text data;

[1505] A means for evaluating the risk of causing harm to a user based on the extracted keywords;

[1506] A means of analyzing emotions from the user's voice, facial expressions, input text, etc.

[1507] A means for adjusting the evaluation results and notification content based on the analyzed emotional information;

[1508] means for notifying the user of the evaluation result;

[1509] A system including:

[1510] (Claim 2)

[1511] 10. The system of claim 1, further comprising: means for scraping the content of the contract or terms of use from the uploaded URL to obtain text data.

[1512] (Claim 3)

[1513] 10. The system of claim 1, further comprising: means for evaluating the extracted keywords using natural language processing techniques.

[1514] "Application example 2 when combining emotion engines"

[1515] (Claim 1)

[1516] A means to upload an image of the contract or terms of use;

[1517] means for converting the uploaded image into text data using optical character recognition technology;

[1518] A means for extracting important keywords from the converted text data;

[1519] A means for evaluating the risk of causing harm to a user based on the extracted keywords;

[1520] means for notifying the user of the evaluation result;

[1521] means for analyzing a user's emotion and optimizing notification content based on the determined emotion;

[1522] means for generating a risk report based on the evaluation result and the emotion information and transmitting the risk report to the terminal;

[1523] A system including:

[1524] (Claim 2)

[1525] 10. The system of claim 1, further comprising: means for scraping the content of the contract or terms of use from the uploaded URL to obtain text data.

[1526] (Claim 3)

[1527] 10. The system of claim 1, further comprising: means for evaluating the extracted keywords using natural language processing techniques. [Explanation of symbols]

[1528] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means to upload an image of the contract or terms of use; means for converting the uploaded image into text data using optical character recognition technology; A means for extracting important keywords from the converted text data; A means for evaluating the risk of causing harm to a user based on the extracted keywords; means for notifying the user of the evaluation result; A system including:

2. The system of claim 1 , further comprising: means for scraping the content of the contract or terms of use from the uploaded URL to obtain text data.

3. The system of claim 1 , further comprising means for evaluating the extracted keywords using natural language processing techniques.

Images