system

Abstract

Provide a system. 【Solution means】 In order to process data related to a user's electronic transactions, a user terminal acquires an image of a physical card, extracts character information from the acquired image and stores it as digital data, associates user location information and store information using the acquired electronic data, means for presenting information related to the user, analyzes the user's past purchase history and, based on the predicted purchase pattern, means for notifying campaign information and promotions, means for monitoring activities related to a website and warning of access to sites with potential risks, A system including.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern highly informationized society, efficient management of electronic money and point systems that consumers use daily is required. However, carrying multiple physical point cards and manually managing information of each service is a great burden, and preventing fraud sites and illegal acts is an extremely important issue in constantly using the Internet safely. An efficient and safe method for solving these problems is needed.

Means for Solving the Problems

[0005] This invention has a function that allows a user terminal to acquire an image of a physical card, extract text information from that image, and store it as digital data. Furthermore, it associates the user's location information with store information and automatically presents information relevant to the user. By analyzing past purchase history and notifying the user of campaign information that matches their purchase patterns, it optimizes purchasing behavior. In addition, it monitors internet activity and warns the user of access to potentially risky sites, thereby ensuring user safety. As a result, users can use electronic money efficiently and with peace of mind.

[0006] A "user terminal" refers to an electronic device used by a user, including mobile devices such as smartphones and tablets.

[0007] A "physical card" is a card that has a physical form, such as a loyalty card or credit card, made from plastic, paper, or other materials.

[0008] "Digital data" refers to information that is processed or stored electronically and is in binary format that a computer can understand.

[0009] "Location information" refers to information that indicates a user's current geographical location, determined using GPS or other technologies.

[0010] "Store information" refers to information about a specific store or retailer, including address, location, products offered, and promotional information.

[0011] "Purchase history" refers to a record of purchases made by a user in the past, including information such as the date and time of purchase, product details, and location of purchase.

[0012] "Campaign information" refers to promotional information related to specific products or services, including information on discounts, coupons, and special offers.

[0013] "Potential risks" refer to dangers and fraudulent activities that users may face, including fraudulent websites and phishing scams.

[0014] "Internet activity" refers to a series of online activities that users perform via the internet, including browsing websites, online shopping, and streaming digital content. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the language used in the following description will be described.

[0018] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] The system of the present invention is designed to efficiently and securely manage users' electronic transactions. This system is implemented primarily using user terminals, servers, and related devices, as follows:

[0037] The user terminal process begins with the user acquiring an image of their physical card. The image captured by the camera is analyzed within the terminal using OCR technology, and the card number and expiration date are extracted as digital data. This digitized card information is stored in a secure database, which the user can easily access and manage through the app.

[0038] The user's device also continuously acquires the user's location information and works with the server to search for nearby store information. Based on this location information, the system automatically notifies the user of available loyalty cards and related promotional information. For example, if the user is near a specific store, information about loyalty cards that can be used at that store will be sent via push notification.

[0039] Furthermore, the server analyzes the user's purchase history, collects and compares requested information based on the user's purchasing patterns, and provides the user with appropriate campaign information. For example, discount information on regularly purchased products and notifications of related campaigns are sent to the user's device. This allows users to efficiently take advantage of advantageous opportunities.

[0040] Furthermore, the user terminal connects with a dedicated wearable device to analyze the user's biometric information and behavioral data, providing personalized product recommendations based on their current condition. For example, the wearable device may measure the user's stress level and suggest products that can help reduce stress.

[0041] In terms of security, the server monitors the user's internet activity and immediately issues a warning if the user attempts to connect to a fraudulent site or a site potentially involving unauthorized access. This significantly reduces the risk of users becoming victims of fraud or other malicious activity.

[0042] By combining these functions, the system of the present invention makes it possible for users to simplify and securely use electronic money and point systems.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The user launches the app and opens a screen to retrieve an image of the physical card. The device uses its camera to take a picture of the card and saves the image.

[0046] Step 2:

[0047] The device uses OCR technology to analyze the image of the card that has been photographed, and extracts textual information such as the card number and expiration date. This information is stored as electronic data within the app.

[0048] Step 3:

[0049] The device periodically obtains the user's location information via GPS. When the user approaches a store, the device sends this information to the server.

[0050] Step 4:

[0051] The server searches for nearby store information based on the received location data. It then sends information about available loyalty cards and promotions to the device.

[0052] Step 5:

[0053] The device pushes promotional information to the user, providing them with purchasing opportunities at the optimal time.

[0054] Step 6:

[0055] The server analyzes the user's purchase history and extracts campaign information based on their purchase patterns. It then notifies the user's device of the relevant information.

[0056] Step 7:

[0057] The user's terminal acquires biometric information and behavioral data from wearable devices. Data analysis is used to determine the user's current condition and provide personalized purchase suggestions.

[0058] Step 8:

[0059] The server monitors internet activity and compares it against a list of known dangerous sites. If a potential risk is detected, it immediately alerts the user.

[0060] (Example 1)

[0061] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0062] In recent years, the proliferation of electronic transactions has increased the complexity of information management for users. Users conduct transactions using various digital media, and are required to provide location information, information related to internet activity, and even health management and promotional optimization using biometric data. However, there are limited systems that can centrally and securely manage this information and guide users to make appropriate choices. There is a need to provide effective solutions that can address this situation.

[0063] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0064] In this invention, the server includes means for a terminal device to acquire, digitize, and store images from a physical medium in order to process information related to the user's digital transactions; means for using the acquired information to associate the user's location information with facility information and provide information relevant to the user; and means for analyzing the user's past purchase history and notifying them of sales promotion information and guidance. This enables the user to securely manage a wide range of information and be automatically guided to make the best choices tailored to their needs.

[0065] A "user" refers to an individual or legal entity that uses the system to conduct digital transactions or manage information.

[0066] A "terminal device" refers to an electronic device that a user operates to acquire information from a physical medium, and includes smartphones and tablets.

[0067] "Physical media" refers to physical information media owned by users, such as credit cards and loyalty cards.

[0068] "Acquiring an image" refers to taking a photograph via a terminal device to obtain information from a physical medium.

[0069] "Digitization" refers to the process of analyzing textual information from images acquired from physical media and saving it as electronic data.

[0070] "Location information" refers to information obtained by a terminal device that indicates the user's current location.

[0071] "Facility information" refers to information about shops and public facilities located in the user's vicinity.

[0072] "Purchase history" refers to a record of purchases a user has made in the past, and purchasing patterns can be derived through analysis.

[0073] "Sales promotion information" refers to information about discounts and campaigns that are notified to users based on their purchase history and location information.

[0074] "Physiological information" refers to data that indicates the user's physical state, such as heart rate and stress level.

[0075] "Behavioral information" refers to data about a user's movement and activities, and is acquired from terminal devices and wearable devices.

[0076] This invention is a system for efficiently and securely managing information related to users' digital transactions and providing users with the information they need in a timely manner. This system mainly consists of terminal devices, servers, and related devices.

[0077] The terminal device uses a camera to capture images of physical media owned by the user, such as a credit card. This image acquisition process utilizes OCR technology running on the terminal device. Specifically, a general-purpose OCR engine (e.g., Tesseract or other image analysis engines) is used to analyze textual information from the image. The resulting textual information is stored as digital data within the terminal and transmitted to a secure database. During this process, the data is encrypted using SSL / TLS to ensure security.

[0078] Furthermore, the terminal device utilizes GPS functionality to continuously acquire the user's location information and transmit it to the server. Based on the received location information, the server searches its database for information on facilities around the user and provides the user with relevant information. This information provision includes, in particular, purchasing patterns and campaign information. The server also uses machine learning techniques to analyze the user's past purchase history and provides individually customized sales promotion information.

[0079] The terminal device works in conjunction with a dedicated wearable device to collect the user's physiological and behavioral information. Wireless communication technologies such as Bluetooth are used for this connection. By analyzing the collected information, personalized suggestions based on the user's current condition become possible.

[0080] This system incorporates security features that monitor users' internet activity and issue warnings when unauthorized access or connections to fraudulent websites are detected. This feature protects users from potential dangers.

[0081] Specific examples include situations where a user is walking near a cafe and wants to know if there are any perks or loyalty programs available at that cafe, or where they want recommendations for relaxation products when they are feeling stressed. This information is efficiently provided through prompt messages to support the user's decision-making.

[0082] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0083] Step 1:

[0084] The user uses a terminal device to acquire an image of a physical medium. The input is a physical medium such as a credit card. The terminal analyzes the image using OCR technology to obtain textual information such as the card number and expiration date. Through this data processing, the physical information is output as digital data. This output data is securely stored in a secure database.

[0085] Step 2:

[0086] The device uses GPS functionality to obtain the user's current location. The input is the user's real-time geographical location. The device sends this location information to the server, providing basic data for searching for facility information around the user. The server consults a facility database and searches for relevant information based on the location, outputting facility information and promotional information that should be provided to the user.

[0087] Step 3:

[0088] The server retrieves user purchase history data from a database. The input is the user's past transaction records. Based on this, the server uses machine learning algorithms to analyze purchasing patterns. This data analysis predicts products and campaign information that the user is likely to be interested in. As output, personalized sales promotion information is generated based on this analysis.

[0089] Step 4:

[0090] The terminal connects to the wearable device via Bluetooth and receives physiological information and activity data in real time. Inputs include physiological information such as heart rate and step count. The terminal analyzes this data to evaluate the user's current condition. Based on this data analysis, optimized product recommendations tailored to the user's state are output and notified to the user as prompt messages.

[0091] Step 5:

[0092] The server monitors the user's internet activity. The input is information about the websites the user accesses. Security software performs data calculations to detect potential unauthorized access. As a result of this calculation, it warns of access to potentially dangerous sites and outputs information to enhance security. The terminal notifies the user of this warning and prompts them to restrict or stop access.

[0093] (Application Example 1)

[0094] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0095] Modern consumers engage in purchasing activities in an environment overflowing with vast amounts of electronic transaction and marketing information. In this context, it is difficult for consumers to appropriately select the most beneficial information and conduct electronic transactions securely. Furthermore, systems that provide real-time product recommendations tailored to individual needs are scarce, and measures to protect personal information from fraudulent information and scams are needed. Solving these challenges and enabling consumers to conduct electronic transactions with peace of mind while providing information tailored to their individual needs is crucial.

[0096] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0097] In this invention, the server includes means for processing data related to the user's electronic transactions, such as acquiring images of physical identification information from the user's terminal and storing them as digital data; means for associating spatial information and commercial facility information using the acquired digital data and presenting relevant information; means for analyzing past purchase history and notifying sales promotion information based on predicted patterns; and means for monitoring activities related to online sites and warning of access to risky sites. This allows consumers to obtain the most relevant information and conduct electronic transactions with peace of mind.

[0098] A "user terminal" is an information processing device used by a user, and is a device that has the role of acquiring and processing electronic data.

[0099] "Physical identification information" refers to a means of identifying an entity that includes a user's personal information and payment information, and is typically embodied as a card or tag.

[0100] "Digital data" refers to data formats that are created by electronically converting physical information into a form that can be stored within a computer system.

[0101] "Spatial information" refers to information related to geographical location, and is data that identifies where users or facilities are located.

[0102] "Commercial facility information" refers to information about facilities that conduct commercial activities, including location, business hours, and information about the products offered.

[0103] "Past purchase history" refers to a record of a user's past purchasing activities, including information such as the products purchased and the date and time.

[0104] "Sales promotion information" refers to information provided to consumers to promote the sale of products, and includes details about discounts and campaigns.

[0105] An "online site" refers to a webpage or web service provided on the internet, where information is shared and electronic transactions take place.

[0106] "Risk" refers to factors or circumstances that could threaten user data or security, and typically involves the risk of fraud or data breaches.

[0107] The system for implementing this invention first acquires physical identification information using a camera mounted on the user terminal and converts it into digital data using OCR technology. The terminal then uses the Google® Cloud Vision API to perform text analysis on the image and securely stores the results in the Firebase Realtime Database.

[0108] Furthermore, the device uses the Google Maps API to obtain the user's current location and associates spatial information with commercial facility information based on that location data. Based on this, relevant sales promotion information is provided via push notifications.

[0109] In purchase history analysis, the server collects users' past purchase data and analyzes the patterns to provide personalized sales promotion information. This analysis utilizes machine learning algorithms to deliver customized information to users.

[0110] Furthermore, if a user is using a wearable device, biometric information is acquired from that device, and the user's physical and mental state is analyzed using the Fitbit API. This makes it possible to suggest products that are tailored to the user's current situation.

[0111] From a security standpoint, the server uses IBM Watson® Security Advisor to monitor users' online activity and immediately issues a warning if access to online sites with potential risks occurs.

[0112] Specific examples include providing real-time notifications on a user's smartphone about coffee discounts when they are near a particular cafe. Additionally, when a user's stress level is high, the system could provide information about products with relaxing effects.

[0113] An example of using a generative AI model to generate prompt text is to input the question, "How can an app notify users of promotional information for specific stores based on their current location?" into the model and obtain an appropriate answer.

[0114] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0115] Step 1:

[0116] The user's device uses its camera to acquire an image of physical identification information.

[0117] The input is an image of a physical card captured by a camera. The output is that image data, which is saved to local storage for further processing.

[0118] Step 2:

[0119] The device uses the Google Cloud Vision API to extract text information from the image.

[0120] The input is the image data acquired in Step 1. The image is analyzed using OCR technology to extract text data such as the card number and expiration date. The output is digital data in text format and is stored in the Firebase Realtime Database.

[0121] Step 3:

[0122] The device uses the Google Maps API to obtain the user's current location and collect spatial information.

[0123] The input is the user's current location. Location data is acquired using a GPS sensor and then cross-referenced with commercial facility information. The output is information about commercial facilities corresponding to the user's location.

[0124] Step 4:

[0125] The server analyzes past purchase history to predict the user's consumption patterns.

[0126] The input is purchase history data stored in the Firebase Realtime Database. A machine learning algorithm is used to analyze past data and identify predictable purchase patterns. The output is a list of promotional information and campaigns.

[0127] Step 5:

[0128] Wearable devices collect biometric information and evaluate health status via the Fitbit API.

[0129] The input consists of biometric data acquired by the device, such as heart rate and stress level. The Fitbit API is used to assess the user's current health status, and the output provides a report on the user's physical and mental condition.

[0130] Step 6:

[0131] The server monitors users' online activity and assesses risks using IBM Watson Security Advisor.

[0132] The input consists of URLs of online sites the user has accessed and activity logs. If a potential risk is detected, a warning is issued immediately. The output consists of the security assessment results and a warning message.

[0133] Step 7:

[0134] Promotional information will be sent to users based on their location.

[0135] The input consists of the commercial facility information obtained in step 3 and the sales promotion information from step 4. The terminal sends a push notification to the user informing them of available promotional information. The output is the notification message the user receives.

[0136] Step 8:

[0137] Generate product suggestions and notify users.

[0138] The input consists of the physical and mental condition information obtained in Step 5 and the sales promotion information from Step 4. A generative AI model is used to create optimal product suggestions for the user and notify the user via their device. The output is a list of recommended products presented to the user.

[0139] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0140] This invention is a system that incorporates an emotion engine to more highly optimize users' electronic transactions. This system is implemented using a user terminal, a server, and emotion recognition technology as follows:

[0141] The user's device first uses its camera and sensors to collect information such as the user's facial expressions and voice. An emotion engine built into the device analyzes this information to identify the user's emotional state. This emotional data is securely transmitted to a server with the user's consent.

[0142] The server records and analyzes the user's emotional state in real time based on the received emotional data. This analysis is combined with the user's purchase history and location information to generate optimal promotional and campaign information for the user. For example, if the server determines that the user is stressed, it will provide information on products suitable for relaxation.

[0143] Furthermore, the emotion engine continuously monitors the user's emotional changes and can flexibly adjust purchase suggestions according to their emotional state at any given time. For example, if the user is feeling happy, it will suggest active products and services.

[0144] Furthermore, this system maintains a security feature that warns users of access to fraudulent websites. By using emotion engine data, it can ensure a higher level of security by providing extra attention when users are feeling anxious.

[0145] This invention allows users to receive personalized services based on their emotional state, enabling them to enjoy a more fulfilling electronic transaction experience.

[0146] The following describes the processing flow.

[0147] Step 1:

[0148] The user's device activates an emotion engine and uses its camera and microphone to collect emotion data from the user's facial expressions and voice. This data is processed immediately on the device.

[0149] Step 2:

[0150] The device's built-in emotion engine analyzes collected information to determine the user's current emotional state in real time. For example, a smiling face indicates "joy," while a frown indicates "anxiety."

[0151] Step 3:

[0152] The device sends the determined emotion data to the server with the user's consent. The server then integrates this data with other user information (purchase history and location information).

[0153] Step 4:

[0154] The server analyzes the integrated data and generates promotional and campaign information optimized for the user's emotional state. For example, if stress is detected, discount information on relaxation products will be provided.

[0155] Step 5:

[0156] The server sends the generated promotional information to the device and sends a push notification to the user at the appropriate time. This notification allows the user to check the suggested products and services.

[0157] Step 6:

[0158] The user terminal continuously monitors emotional data, and if emotions change, it re-evaluates the emotional state and repeats the process from step 2 onward.

[0159] Step 7:

[0160] The server also monitors internet activity and, if it detects access to potentially risky websites, it issues additional warnings to the user, especially if the emotion engine detects anxiety.

[0161] (Example 2)

[0162] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0163] In modern e-commerce, information and promotions are often provided uniformly without considering the user's emotional state, making it difficult to deliver a truly valuable, personalized experience. Furthermore, the lack of flexible service delivery that reflects changes in user emotions in real time means missed opportunities to improve purchase satisfaction. Additionally, there are insufficient means to alleviate users' anxiety when accessing potentially risky websites.

[0164] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0165] In this invention, the server includes means for the user terminal to acquire the user's facial expressions and voice information using a camera and sensors, and for analyzing the acquired information to identify the user's emotional state; means for securely transmitting the identified emotional data to the server for real-time recording and analysis; and means for integrating the received emotional data with the user's past purchase history and location information to generate personalized suggestions for the user. This makes it possible to provide real-time and personalized information and purchase suggestions based on the user's emotions.

[0166] A "user terminal" is a device used by users to input information and collect emotional data, and is equipped with cameras and sensors.

[0167] "Cameras and sensors" are devices used to acquire user facial expressions and voice information, and have the function of collecting data in real time.

[0168] "Emotional state" refers to the psychological state analyzed from the user's facial expressions and voice, and includes emotions such as stress and relaxation.

[0169] A "server" is a computer device that receives emotional data transmitted from user terminals and records and analyzes it in real time.

[0170] "Personalized recommendations" refer to the provision of promotions and information optimized for individual users, generated based on their past purchase history and location information.

[0171] "Emotional analysis results" refer to the results of an analysis of the user's emotional state, presented as numerical values ​​or indicators.

[0172] "Potential risks" refer to the safety and security threats to websites and online services that users attempt to access.

[0173] This invention provides a system equipped with an emotion engine to highly optimize users' electronic transactions, and is implemented using a user terminal and a server. The user terminal is equipped with a camera and various sensors to collect the user's facial expressions and voice in real time. Specifically, in addition to a general computer device, the user terminal uses a camera for facial recognition and a microphone for capturing voice. This allows the terminal to analyze the user's emotional state in detail. For emotion analysis, software utilizing machine learning as the emotion engine is used to extract emotional data from the user's facial expressions and voice.

[0174] The device sends analyzed emotional data to the server with the user's consent. The server receives this data and records and analyzes it in real time. During this process, the server integrates past purchase history and location information using a database and utilizes the latest AI models to generate optimized information for the user. For example, if a user is experiencing stress, the server can suggest relaxation products and services, providing a more personalized service.

[0175] Furthermore, the server also monitors website access and warns users of potentially risky sites if they feel uneasy. Such security features enhance user confidence.

[0176] For example, if the terminal detects that the user is displaying a happy expression while shopping, the server immediately suggests products for summer camps and outdoor activities. At this point, prompts such as, "You are receiving personalized product suggestions based on your current emotional state. How does this system analyze the user's emotional data and generate purchase suggestions?" are used, enabling the generating AI model to provide optimal suggestions. In this way, the present invention provides personalized suggestions tailored to the user's emotions, offering a more valuable electronic transaction experience.

[0177] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0178] Step 1:

[0179] The user's device uses its camera and sensors to capture the user's facial expressions and voice.

[0180] The input is the user's real-time facial expressions and voice, and the output is digitized information of this data. The terminal prepares to analyze the user's emotional state based on this data. Specifically, the camera captures the user's face while the microphone records their voice.

[0181] Step 2:

[0182] An emotion engine built into the device analyzes digitized facial and voice data to identify the user's emotional state.

[0183] The input is the digitized data obtained in Step 1, and the output is an index or numerical value indicating the user's emotions. Based on this input data, the emotion engine uses a machine learning algorithm to analyze patterns in the emotional data and identify emotional states such as "joy" or "stress."

[0184] Step 3:

[0185] The device sends analyzed emotional data to the server with the user's consent.

[0186] The input is the emotional state identified in step 2, and the output is the emotional data sent to the server. Specifically, the terminal uses data encryption technology to securely send the data to the server.

[0187] Step 4:

[0188] The server receives emotional data and records and analyzes it in real time.

[0189] The input is sentiment data sent from the device, and the output is a user profile that integrates the analyzed sentiment data. The server uses a database to match sentiment data with past purchase history and location information, preparing to generate personalized information.

[0190] Step 5:

[0191] The server generates personalized suggestion information for the user and sends it to the terminal.

[0192] The input is an integrated user profile, and the output is personalized promotional and campaign information presented to the user. Using a generative AI model, it provides optimal suggestions that match the user's emotional state; for example, if the user needs relaxation, it recommends relaxation products.

[0193] Step 6:

[0194] The server monitors users' access to websites and warns them about accessing potentially risky sites if they feel uneasy.

[0195] The input is the user's emotional state and website access information, and the output is a warning message. Specifically, the server protects the user by displaying a message such as "This site may be unsafe" when the user attempts to access a site that may be fraudulent.

[0196] (Application Example 2)

[0197] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0198] There is a growing need to optimize electronic transactions based on users' emotional states and provide personalized services and security measures for individual users. However, current technology does not adequately reflect user emotions in product recommendations or improve security. It is necessary to address this challenge and provide a more fulfilling electronic transaction experience.

[0199] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0200] In this invention, the server includes means for the user terminal to acquire facial expression information and voice information using sensors to recognize the user's emotional state, analyze the acquired information and securely transmit emotional data to the server, analyze the acquired emotional data in combination with purchase history and generate optimized product suggestions based on the emotional state, and warn the user to access fraudulent sites to improve security when the user shows signs of anxiety. This enables personalized product suggestions that respond to the user's emotions and improved security.

[0201] A "user terminal" is a type of computer device operated by a user, and its role is to collect and process information.

[0202] A "sensor" is a device that collects physical information and outputs it as an electrical signal.

[0203] "Facial expression information" refers to data that represents the movements and characteristics of a user's face, and is used for emotion recognition.

[0204] "Audio information" refers to sound data, including the user's speech, which is used to analyze emotions and intentions.

[0205] "Emotional data" refers to data that quantifies or classifies a user's emotional state.

[0206] A "server" is a computer system used to process data and provide services over a network.

[0207] "Purchase history" refers to a record of products a user has purchased in the past, and serves as basic data for predicting future purchasing patterns.

[0208] "Optimized product recommendations" refer to personalized recommendations of products and services that take into account the user's needs and emotional state.

[0209] "Means of warning against accessing fraudulent sites to improve security" refers to technologies that have the function of issuing warnings to prevent users from accessing dangerous websites.

[0210] "Personalized product recommendations" refer to recommendations for products and services that are specially customized based on the user's individual attributes and circumstances.

[0211] This system consists of a user terminal, sensors, a server, and emotion recognition software.

[0212] The user terminal uses a camera and microphone to collect the user's facial expressions and voice information. This information is transmitted to the emotion recognition engine via sensors. The emotion recognition engine uses the Google Cloud Vision API and a natural language processing engine for voice analysis to analyze the user's emotional state. The analyzed emotion data is securely transmitted to the server with the user's consent.

[0213] The server integrates and analyzes received sentiment data with the user's past purchase history. Using Python, the server processes the data and generates optimal product recommendations based on the user's current emotional state. These recommendations are delivered to the user via email or in-app notifications. Furthermore, if the user expresses anxiety, the server immediately issues a warning about fraudulent websites to enhance security. This utilizes specialized software to strengthen security features.

[0214] For example, if the system analyzes that a user has a high need for relaxation while operating their device, the server may suggest relaxation-related products such as aroma diffusers or massage chairs. If the user is feeling stressed, the system may also encourage them to purchase yoga class vouchers or relaxation music.

[0215] An example of a prompt message when using a generative AI model is: "If the user is feeling stressed, create a list of recommended products. For example, list products with relaxation effects."

[0216] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0217] Step 1:

[0218] The user terminal uses a camera and microphone to acquire the user's facial expressions and voice information. The input is a facial image and voice, and the output is raw facial expression data and voice data. The terminal transmits this information to the emotion recognition engine via sensors.

[0219] Step 2:

[0220] The server uses an emotion recognition engine to analyze acquired facial and audio information. The input consists of facial image data and audio data, while the output is emotion data indicating the user's emotional state. The analysis utilizes the Google Cloud Vision API and an audio analysis engine, processing the data in real time.

[0221] Step 3:

[0222] The server integrates and analyzes emotional data and the user's past purchase history. The input is emotional data and past purchase history data, and the output is an optimized product suggestion list based on the emotional state. Python is used to combine this data and leverage a generative AI model to generate product suggestions.

[0223] Step 4:

[0224] The server provides the user with generated product suggestions. The input is a list of product suggestions, and the output is a notification to the user. The notification is delivered via email or in-app notification, allowing the user to receive product information that matches their emotional state.

[0225] Step 5:

[0226] If a user shows signs of anxiety while operating their device, the server will warn them about accessing a fraudulent website. The input is user sentiment data, and the output is a security warning. The server uses specialized security software to take action to ensure the user's safety.

[0227] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0228] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0229] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0230] [Second Embodiment]

[0231] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0232] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0233] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0234] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0235] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0236] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0237] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0238] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0239] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0240] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0241] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0242] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0243] The system of the present invention is designed to efficiently and securely manage users' electronic transactions. This system is implemented primarily using user terminals, servers, and related devices, as follows:

[0244] The user terminal process begins with the user acquiring an image of their physical card. The image captured by the camera is analyzed within the terminal using OCR technology, and the card number and expiration date are extracted as digital data. This digitized card information is stored in a secure database, which the user can easily access and manage through the app.

[0245] The user's device also continuously acquires the user's location information and works with the server to search for nearby store information. Based on this location information, the system automatically notifies the user of available loyalty cards and related promotional information. For example, if the user is near a specific store, information about loyalty cards that can be used at that store will be sent via push notification.

[0246] Furthermore, the server analyzes the user's purchase history, collects and compares requested information based on the user's purchasing patterns, and provides the user with appropriate campaign information. For example, discount information on regularly purchased products and notifications of related campaigns are sent to the user's device. This allows users to efficiently take advantage of advantageous opportunities.

[0247] Furthermore, the user terminal connects with a dedicated wearable device to analyze the user's biometric information and behavioral data, providing personalized product recommendations based on their current condition. For example, the wearable device may measure the user's stress level and suggest products that can help reduce stress.

[0248] In terms of security, the server monitors the user's internet activity and immediately issues a warning if the user attempts to connect to a fraudulent site or a site potentially involving unauthorized access. This significantly reduces the risk of users becoming victims of fraud or other malicious activity.

[0249] By combining these functions, the system of the present invention makes it possible for users to simplify and securely use electronic money and point systems.

[0250] The following describes the processing flow.

[0251] Step 1:

[0252] The user launches the app and opens a screen to retrieve an image of the physical card. The device uses its camera to take a picture of the card and saves the image.

[0253] Step 2:

[0254] The device uses OCR technology to analyze the image of the card that has been photographed, and extracts textual information such as the card number and expiration date. This information is stored as electronic data within the app.

[0255] Step 3:

[0256] The device periodically obtains the user's location information via GPS. When the user approaches a store, the device sends this information to the server.

[0257] Step 4:

[0258] The server searches for nearby store information based on the received location data. It then sends information about available loyalty cards and promotions to the device.

[0259] Step 5:

[0260] The device pushes promotional information to the user, providing them with purchasing opportunities at the optimal time.

[0261] Step 6:

[0262] The server analyzes the user's purchase history and extracts campaign information based on their purchase patterns. It then notifies the user's device of the relevant information.

[0263] Step 7:

[0264] The user's terminal acquires biometric information and behavioral data from wearable devices. Data analysis is used to determine the user's current condition and provide personalized purchase suggestions.

[0265] Step 8:

[0266] The server monitors internet activity and compares it against a list of known dangerous sites. If a potential risk is detected, it immediately alerts the user.

[0267] (Example 1)

[0268] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0269] In recent years, the proliferation of electronic transactions has increased the complexity of information management for users. Users conduct transactions using various digital media, and are required to provide location information, information related to internet activity, and even health management and promotional optimization using biometric data. However, there are limited systems that can centrally and securely manage this information and guide users to make appropriate choices. There is a need to provide effective solutions that can address this situation.

[0270] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0271] In this invention, the server includes means for a terminal device to acquire, digitize, and store images from a physical medium in order to process information related to the user's digital transactions; means for using the acquired information to associate the user's location information with facility information and provide information relevant to the user; and means for analyzing the user's past purchase history and notifying them of sales promotion information and guidance. This enables the user to securely manage a wide range of information and be automatically guided to make the best choices tailored to their needs.

[0272] A "user" refers to an individual or legal entity that uses the system to conduct digital transactions or manage information.

[0273] A "terminal device" refers to an electronic device that a user operates to acquire information from a physical medium, and includes smartphones and tablets.

[0274] "Physical media" refers to physical information media owned by users, such as credit cards and loyalty cards.

[0275] "Acquiring an image" refers to taking a photograph via a terminal device to obtain information from a physical medium.

[0276] "Digitization" refers to the process of analyzing textual information from images acquired from physical media and saving it as electronic data.

[0277] "Location information" refers to information obtained by a terminal device that indicates the user's current location.

[0278] "Facility information" refers to information about shops and public facilities located in the user's vicinity.

[0279] "Purchase history" refers to a record of purchases a user has made in the past, and purchasing patterns can be derived through analysis.

[0280] "Sales promotion information" refers to information about discounts and campaigns that are notified to users based on their purchase history and location information.

[0281] "Physiological information" refers to data that indicates the user's physical state, such as heart rate and stress level.

[0282] "Behavioral information" refers to data about a user's movement and activities, and is acquired from terminal devices and wearable devices.

[0283] This invention is a system for efficiently and securely managing information related to users' digital transactions and providing users with the information they need in a timely manner. This system mainly consists of terminal devices, servers, and related devices.

[0284] The terminal device uses a camera to obtain an image of a physical medium owned by the user, such as a credit card. In this image acquisition process, OCR technology operating on the terminal device is used. Specifically, a general OCR engine (e.g., Tesseract or other image analysis engines) is used to analyze character information from the image. The character information obtained from this analysis is stored in the terminal as digital data and transmitted to a secure database. During this process, the data is encrypted by SSL / TLS to ensure security.

[0285] Furthermore, the terminal device utilizes the GPS function to continuously obtain the user's location information and transmit it to the server. Based on the received location information, the server searches the database for facility information around the user and provides relevant information to the user. This information provision particularly includes purchase patterns and campaign information. The server also uses machine learning technology to analyze the user's past purchase history and provides individually customized sales promotion information.

[0286] The terminal device cooperates with a dedicated wearable device to collect the user's physiological information and behavioral information. Wireless communication technologies such as Bluetooth are used for this cooperation. By analyzing the obtained information, individual proposals based on the user's current condition become possible.

[0287] This system incorporates a security function that monitors the user's Internet activities and issues a warning when detecting unauthorized access or connection to a fraud site. With this function, the user is protected from potential risks.

[0288] As a specific example, when the user is walking near a café with a mobile phone and wants to know if there are any privileges or point cards available at that café, or when the user has accumulated stress and relaxation products are recommended. This information is efficiently provided through prompt messages to support the user's decision-making.

[0289] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0290] Step 1:

[0291] The user uses a terminal device to acquire an image of a physical medium. The input is a physical medium such as a credit card. The terminal analyzes the image using OCR technology to obtain textual information such as the card number and expiration date. Through this data processing, the physical information is output as digital data. This output data is securely stored in a secure database.

[0292] Step 2:

[0293] The device uses GPS functionality to obtain the user's current location. The input is the user's real-time geographical location. The device sends this location information to the server, providing basic data for searching for facility information around the user. The server consults a facility database and searches for relevant information based on the location, outputting facility information and promotional information that should be provided to the user.

[0294] Step 3:

[0295] The server retrieves user purchase history data from a database. The input is the user's past transaction records. Based on this, the server uses machine learning algorithms to analyze purchasing patterns. This data analysis predicts products and campaign information that the user is likely to be interested in. As output, personalized sales promotion information is generated based on this analysis.

[0296] Step 4:

[0297] The terminal connects to the wearable device via Bluetooth and receives physiological information and activity data in real time. Inputs include physiological information such as heart rate and step count. The terminal analyzes this data to evaluate the user's current condition. Based on this data analysis, optimized product recommendations tailored to the user's state are output and notified to the user as prompt messages.

[0298] Step 5:

[0299] The server monitors the user's internet activity. The input is information about the websites the user accesses. Security software performs data calculations to detect potential unauthorized access. As a result of this calculation, it warns of access to potentially dangerous sites and outputs information to enhance security. The terminal notifies the user of this warning and prompts them to restrict or stop access.

[0300] (Application Example 1)

[0301] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0302] Modern consumers engage in purchasing activities in an environment overflowing with vast amounts of electronic transaction and marketing information. In this context, it is difficult for consumers to appropriately select the most beneficial information and conduct electronic transactions securely. Furthermore, systems that provide real-time product recommendations tailored to individual needs are scarce, and measures to protect personal information from fraudulent information and scams are needed. Solving these challenges and enabling consumers to conduct electronic transactions with peace of mind while providing information tailored to their individual needs is crucial.

[0303] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0304] In this invention, the server includes means for the user terminal to acquire an image of physical identification information and store it as digital data in order to process data related to the user's electronic transactions; means for associating spatial information and commercial facility information using the acquired electronic data and presenting the related information; means for analyzing the past purchase history and notifying sales promotion information based on the predicted patterns; and means for monitoring activities related to the online site and warning against access to risky sites. As a result, consumers can obtain the most relevant information for themselves and can conduct electronic transactions with confidence.

[0305] The "user terminal" refers to an information processing device used by the user and is a device that plays a role in acquiring and processing electronic data.

[0306] The "physical identification information" is an identification means for entities including the user's personal information and payment information, and is usually embodied as a card or tag.

[0307] The "digital data" is a data format obtained by electronically converting physical information and is converted into a form that can be stored in a computer system.

[0308] The "spatial information" is information related to geographical locations and is data for identifying where the user and facilities are located.

[0309] The "commercial facility information" is information related to facilities conducting commercial activities and includes information such as location, business hours, and offered products.

[0310] The "past purchase history" is a record of the user's past purchase activities and includes information such as purchased products and dates.

[0311] The "sales promotion information" is information provided to consumers to promote the sales of products and includes details regarding discounts and campaigns.

[0312] An "online site" refers to a webpage or web service provided on the internet, where information is shared and electronic transactions take place.

[0313] "Risk" refers to factors or circumstances that could threaten user data or security, and typically involves the risk of fraud or data breaches.

[0314] The system for implementing this invention first acquires physical identification information using a camera mounted on the user terminal and converts it into digital data using OCR technology. The terminal then uses the Google Cloud Vision API to perform text analysis on the image and securely stores the results in the Firebase Realtime Database.

[0315] Furthermore, the device uses the Google Maps API to obtain the user's current location and associates spatial information with commercial facility information based on that location data. Based on this, relevant sales promotion information is provided via push notifications.

[0316] In purchase history analysis, the server collects users' past purchase data and analyzes the patterns to provide personalized sales promotion information. This analysis utilizes machine learning algorithms to deliver customized information to users.

[0317] Furthermore, if a user is using a wearable device, biometric information is acquired from that device, and the user's physical and mental state is analyzed using the Fitbit API. This makes it possible to suggest products that are tailored to the user's current situation.

[0318] From a security perspective, the server uses IBM Watson Security Advisor to monitor users' online activity and immediately issues a warning if access to online sites with potential risks occurs.

[0319] Specific examples include providing real-time notifications on a user's smartphone about coffee discounts when they are near a particular cafe. Additionally, when a user's stress level is high, the system could provide information about products with relaxing effects.

[0320] An example of using a generative AI model to generate prompt text is to input the question, "How can an app notify users of promotional information for specific stores based on their current location?" into the model and obtain an appropriate answer.

[0321] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0322] Step 1:

[0323] The user's device uses its camera to acquire an image of physical identification information.

[0324] The input is an image of a physical card captured by a camera. The output is that image data, which is saved to local storage for further processing.

[0325] Step 2:

[0326] The device uses the Google Cloud Vision API to extract text information from the image.

[0327] The input is the image data acquired in Step 1. The image is analyzed using OCR technology to extract text data such as the card number and expiration date. The output is digital data in text format and is stored in the Firebase Realtime Database.

[0328] Step 3:

[0329] The device uses the Google Maps API to obtain the user's current location and collect spatial information.

[0330] The input is the user's current location. Location data is acquired using a GPS sensor and then cross-referenced with commercial facility information. The output is information about commercial facilities corresponding to the user's location.

[0331] Step 4:

[0332] The server analyzes past purchase history to predict the user's consumption patterns.

[0333] The input is purchase history data stored in the Firebase Realtime Database. A machine learning algorithm is used to analyze past data and identify predictable purchase patterns. The output is a list of promotional information and campaigns.

[0334] Step 5:

[0335] Wearable devices collect biometric information and evaluate health status via the Fitbit API.

[0336] The input consists of biometric data acquired by the device, such as heart rate and stress level. The Fitbit API is used to assess the user's current health status, and the output provides a report on the user's physical and mental condition.

[0337] Step 6:

[0338] The server monitors users' online activity and assesses risks using IBM Watson Security Advisor.

[0339] The input consists of URLs of online sites the user has accessed and activity logs. If a potential risk is detected, a warning is issued immediately. The output consists of the security assessment results and a warning message.

[0340] Step 7:

[0341] Promotional information will be sent to users based on their location.

[0342] The input consists of the commercial facility information obtained in step 3 and the sales promotion information from step 4. The terminal sends a push notification to the user informing them of available promotional information. The output is the notification message the user receives.

[0343] Step 8:

[0344] Generate product suggestions and notify users.

[0345] The input consists of the physical and mental condition information obtained in Step 5 and the sales promotion information from Step 4. A generative AI model is used to create optimal product suggestions for the user and notify the user via their device. The output is a list of recommended products presented to the user.

[0346] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0347] This invention is a system that incorporates an emotion engine to more highly optimize users' electronic transactions. This system is implemented using a user terminal, a server, and emotion recognition technology as follows:

[0348] The user's device first uses its camera and sensors to collect information such as the user's facial expressions and voice. An emotion engine built into the device analyzes this information to identify the user's emotional state. This emotional data is securely transmitted to a server with the user's consent.

[0349] The server records and analyzes the user's emotional state in real time based on the received emotional data. This analysis is combined with the user's purchase history and location information to generate optimal promotional and campaign information for the user. For example, if the server determines that the user is stressed, it will provide information on products suitable for relaxation.

[0350] Furthermore, the emotion engine continuously monitors the user's emotional changes and can flexibly adjust purchase suggestions according to their emotional state at any given time. For example, if the user is feeling happy, it will suggest active products and services.

[0351] Furthermore, this system maintains a security feature that warns users of access to fraudulent websites. By using emotion engine data, it can ensure a higher level of security by providing extra attention when users are feeling anxious.

[0352] This invention allows users to receive personalized services based on their emotional state, enabling them to enjoy a more fulfilling electronic transaction experience.

[0353] The following describes the processing flow.

[0354] Step 1:

[0355] The user's device activates an emotion engine and uses its camera and microphone to collect emotion data from the user's facial expressions and voice. This data is processed immediately on the device.

[0356] Step 2:

[0357] The device's built-in emotion engine analyzes collected information to determine the user's current emotional state in real time. For example, a smiling face indicates "joy," while a frown indicates "anxiety."

[0358] Step 3:

[0359] The device sends the determined emotion data to the server with the user's consent. The server then integrates this data with other user information (purchase history and location information).

[0360] Step 4:

[0361] The server analyzes the integrated data and generates promotional and campaign information optimized for the user's emotional state. For example, if stress is detected, discount information on relaxation products will be provided.

[0362] Step 5:

[0363] The server sends the generated promotional information to the device and sends a push notification to the user at the appropriate time. This notification allows the user to check the suggested products and services.

[0364] Step 6:

[0365] The user terminal continuously monitors emotional data, and if emotions change, it re-evaluates the emotional state and repeats the process from step 2 onward.

[0366] Step 7:

[0367] The server also monitors internet activity and, if it detects access to potentially risky websites, it issues additional warnings to the user, especially if the emotion engine detects anxiety.

[0368] (Example 2)

[0369] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0370] In modern e-commerce, information and promotions are often provided uniformly without considering the user's emotional state, making it difficult to deliver a truly valuable, personalized experience. Furthermore, the lack of flexible service delivery that reflects changes in user emotions in real time means missed opportunities to improve purchase satisfaction. Additionally, there are insufficient means to alleviate users' anxiety when accessing potentially risky websites.

[0371] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0372] In this invention, the server includes means for the user terminal to acquire the user's facial expressions and voice information using a camera and sensors, and for analyzing the acquired information to identify the user's emotional state; means for securely transmitting the identified emotional data to the server for real-time recording and analysis; and means for integrating the received emotional data with the user's past purchase history and location information to generate personalized suggestions for the user. This makes it possible to provide real-time and personalized information and purchase suggestions based on the user's emotions.

[0373] A "user terminal" is a device used by users to input information and collect emotional data, and is equipped with cameras and sensors.

[0374] "Cameras and sensors" are devices used to acquire user facial expressions and voice information, and have the function of collecting data in real time.

[0375] "Emotional state" refers to the psychological state analyzed from the user's facial expressions and voice, and includes emotions such as stress and relaxation.

[0376] A "server" is a computer device that receives emotional data transmitted from user terminals and records and analyzes it in real time.

[0377] "Personalized recommendations" refer to the provision of promotions and information optimized for individual users, generated based on their past purchase history and location information.

[0378] "Emotional analysis results" refer to the results of an analysis of the user's emotional state, presented as numerical values ​​or indicators.

[0379] "Potential risks" refer to the safety and security threats to websites and online services that users attempt to access.

[0380] This invention provides a system equipped with an emotion engine to highly optimize users' electronic transactions, and is implemented using a user terminal and a server. The user terminal is equipped with a camera and various sensors to collect the user's facial expressions and voice in real time. Specifically, in addition to a general computer device, the user terminal uses a camera for facial recognition and a microphone for capturing voice. This allows the terminal to analyze the user's emotional state in detail. For emotion analysis, software utilizing machine learning as the emotion engine is used to extract emotional data from the user's facial expressions and voice.

[0381] The device sends analyzed emotional data to the server with the user's consent. The server receives this data and records and analyzes it in real time. During this process, the server integrates past purchase history and location information using a database and utilizes the latest AI models to generate optimized information for the user. For example, if a user is experiencing stress, the server can suggest relaxation products and services, providing a more personalized service.

[0382] Furthermore, the server also monitors website access and warns users of potentially risky sites if they feel uneasy. Such security features enhance user confidence.

[0383] For example, if the terminal detects that the user is displaying a happy expression while shopping, the server immediately suggests products for summer camps and outdoor activities. At this point, prompts such as, "You are receiving personalized product suggestions based on your current emotional state. How does this system analyze the user's emotional data and generate purchase suggestions?" are used, enabling the generating AI model to provide optimal suggestions. In this way, the present invention provides personalized suggestions tailored to the user's emotions, offering a more valuable electronic transaction experience.

[0384] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0385] Step 1:

[0386] The user's device uses its camera and sensors to capture the user's facial expressions and voice.

[0387] The input is the user's real-time facial expressions and voice, and the output is digitized information of this data. The terminal prepares to analyze the user's emotional state based on this data. Specifically, the camera captures the user's face while the microphone records their voice.

[0388] Step 2:

[0389] An emotion engine built into the device analyzes digitized facial and voice data to identify the user's emotional state.

[0390] The input is the digitized data obtained in Step 1, and the output is an index or numerical value indicating the user's emotions. Based on this input data, the emotion engine uses a machine learning algorithm to analyze patterns in the emotional data and identify emotional states such as "joy" or "stress."

[0391] Step 3:

[0392] The device sends analyzed emotional data to the server with the user's consent.

[0393] The input is the emotional state identified in step 2, and the output is the emotional data sent to the server. Specifically, the terminal uses data encryption technology to securely send the data to the server.

[0394] Step 4:

[0395] The server receives emotional data and records and analyzes it in real time.

[0396] The input is sentiment data sent from the device, and the output is a user profile that integrates the analyzed sentiment data. The server uses a database to match sentiment data with past purchase history and location information, preparing to generate personalized information.

[0397] Step 5:

[0398] The server generates personalized suggestion information for the user and sends it to the terminal.

[0399] The input is an integrated user profile, and the output is personalized promotional and campaign information presented to the user. Using a generative AI model, it provides optimal suggestions that match the user's emotional state; for example, if the user needs relaxation, it recommends relaxation products.

[0400] Step 6:

[0401] The server monitors users' access to websites and warns them about accessing potentially risky sites if they feel uneasy.

[0402] The input is the user's emotional state and website access information, and the output is a warning message. Specifically, the server protects the user by displaying a message such as "This site may be unsafe" when the user attempts to access a site that may be fraudulent.

[0403] (Application Example 2)

[0404] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0405] There is a growing need to optimize electronic transactions based on users' emotional states and provide personalized services and security measures for individual users. However, current technology does not adequately reflect user emotions in product recommendations or improve security. It is necessary to address this challenge and provide a more fulfilling electronic transaction experience.

[0406] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0407] In this invention, the server includes means for the user terminal to acquire facial expression information and voice information using sensors to recognize the user's emotional state, analyze the acquired information and securely transmit emotional data to the server, analyze the acquired emotional data in combination with purchase history and generate optimized product suggestions based on the emotional state, and warn the user to access fraudulent sites to improve security when the user shows signs of anxiety. This enables personalized product suggestions that respond to the user's emotions and improved security.

[0408] A "user terminal" is a type of computer device operated by a user, and its role is to collect and process information.

[0409] A "sensor" is a device that collects physical information and outputs it as an electrical signal.

[0410] "Facial expression information" refers to data that represents the movements and characteristics of a user's face, and is used for emotion recognition.

[0411] "Audio information" refers to sound data, including the user's speech, which is used to analyze emotions and intentions.

[0412] "Emotional data" refers to data that quantifies or classifies a user's emotional state.

[0413] A "server" is a computer system used to process data and provide services over a network.

[0414] "Purchase history" refers to a record of products a user has purchased in the past, and serves as basic data for predicting future purchasing patterns.

[0415] "Optimized product recommendations" refer to personalized recommendations of products and services that take into account the user's needs and emotional state.

[0416] "Means of warning against accessing fraudulent sites to improve security" refers to technologies that have the function of issuing warnings to prevent users from accessing dangerous websites.

[0417] "Personalized product recommendations" refer to recommendations for products and services that are specially customized based on the user's individual attributes and circumstances.

[0418] This system consists of a user terminal, sensors, a server, and emotion recognition software.

[0419] The user terminal uses a camera and microphone to collect the user's facial expressions and voice information. This information is transmitted to the emotion recognition engine via sensors. The emotion recognition engine uses the Google Cloud Vision API and a natural language processing engine for voice analysis to analyze the user's emotional state. The analyzed emotion data is securely transmitted to the server with the user's consent.

[0420] The server integrates and analyzes received sentiment data with the user's past purchase history. Using Python, the server processes the data and generates optimal product recommendations based on the user's current emotional state. These recommendations are delivered to the user via email or in-app notifications. Furthermore, if the user expresses anxiety, the server immediately issues a warning about fraudulent websites to enhance security. This utilizes specialized software to strengthen security features.

[0421] For example, if the system analyzes that a user has a high need for relaxation while operating their device, the server may suggest relaxation-related products such as aroma diffusers or massage chairs. If the user is feeling stressed, the system may also encourage them to purchase yoga class vouchers or relaxation music.

[0422] An example of a prompt message when using a generative AI model is: "If the user is feeling stressed, create a list of recommended products. For example, list products with relaxation effects."

[0423] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0424] Step 1:

[0425] The user terminal uses a camera and microphone to acquire the user's facial expressions and voice information. The input is a facial image and voice, and the output is raw facial expression data and voice data. The terminal transmits this information to the emotion recognition engine via sensors.

[0426] Step 2:

[0427] The server uses an emotion recognition engine to analyze acquired facial and audio information. The input consists of facial image data and audio data, while the output is emotion data indicating the user's emotional state. The analysis utilizes the Google Cloud Vision API and an audio analysis engine, processing the data in real time.

[0428] Step 3:

[0429] The server integrates and analyzes emotional data and the user's past purchase history. The input is emotional data and past purchase history data, and the output is an optimized product suggestion list based on the emotional state. Python is used to combine this data and leverage a generative AI model to generate product suggestions.

[0430] Step 4:

[0431] The server provides the user with generated product suggestions. The input is a list of product suggestions, and the output is a notification to the user. The notification is delivered via email or in-app notification, allowing the user to receive product information that matches their emotional state.

[0432] Step 5:

[0433] If a user shows signs of anxiety while operating their device, the server will warn them about accessing a fraudulent website. The input is user sentiment data, and the output is a security warning. The server uses specialized security software to take action to ensure the user's safety.

[0434] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0435] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0436] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0437] [Third Embodiment]

[0438] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0439] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0440] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0441] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0442] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0443] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0444] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0445] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0446] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0447] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0448] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0449] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0450] The system of the present invention is designed to efficiently and securely manage users' electronic transactions. This system is implemented primarily using user terminals, servers, and related devices, as follows:

[0451] The user terminal process begins with the user acquiring an image of their physical card. The image captured by the camera is analyzed within the terminal using OCR technology, and the card number and expiration date are extracted as digital data. This digitized card information is stored in a secure database, which the user can easily access and manage through the app.

[0452] The user's device also continuously acquires the user's location information and works with the server to search for nearby store information. Based on this location information, the system automatically notifies the user of available loyalty cards and related promotional information. For example, if the user is near a specific store, information about loyalty cards that can be used at that store will be sent via push notification.

[0453] Furthermore, the server analyzes the user's purchase history, collects and compares requested information based on the user's purchasing patterns, and provides the user with appropriate campaign information. For example, discount information on regularly purchased products and notifications of related campaigns are sent to the user's device. This allows users to efficiently take advantage of advantageous opportunities.

[0454] Furthermore, the user terminal connects with a dedicated wearable device to analyze the user's biometric information and behavioral data, providing personalized product recommendations based on their current condition. For example, the wearable device may measure the user's stress level and suggest products that can help reduce stress.

[0455] In terms of security, the server monitors the user's internet activity and immediately issues a warning if the user attempts to connect to a fraudulent site or a site potentially involving unauthorized access. This significantly reduces the risk of users becoming victims of fraud or other malicious activity.

[0456] By combining these functions, the system of the present invention makes it possible for users to simplify and securely use electronic money and point systems.

[0457] The following describes the processing flow.

[0458] Step 1:

[0459] The user launches the app and opens a screen to retrieve an image of the physical card. The device uses its camera to take a picture of the card and saves the image.

[0460] Step 2:

[0461] The device uses OCR technology to analyze the image of the card that has been photographed, and extracts textual information such as the card number and expiration date. This information is stored as electronic data within the app.

[0462] Step 3:

[0463] The device periodically obtains the user's location information via GPS. When the user approaches a store, the device sends this information to the server.

[0464] Step 4:

[0465] The server searches for nearby store information based on the received location data. It then sends information about available loyalty cards and promotions to the device.

[0466] Step 5:

[0467] The device pushes promotional information to the user, providing them with purchasing opportunities at the optimal time.

[0468] Step 6:

[0469] The server analyzes the user's purchase history and extracts campaign information based on their purchase patterns. It then notifies the user's device of the relevant information.

[0470] Step 7:

[0471] The user's terminal acquires biometric information and behavioral data from wearable devices. Data analysis is used to determine the user's current condition and provide personalized purchase suggestions.

[0472] Step 8:

[0473] The server monitors internet activity and compares it against a list of known dangerous sites. If a potential risk is detected, it immediately alerts the user.

[0474] (Example 1)

[0475] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0476] In recent years, the proliferation of electronic transactions has increased the complexity of information management for users. Users conduct transactions using various digital media, and are required to provide location information, information related to internet activity, and even health management and promotional optimization using biometric data. However, there are limited systems that can centrally and securely manage this information and guide users to make appropriate choices. There is a need to provide effective solutions that can address this situation.

[0477] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0478] In this invention, the server includes means for a terminal device to acquire, digitize, and store images from a physical medium in order to process information related to the user's digital transactions; means for using the acquired information to associate the user's location information with facility information and provide information relevant to the user; and means for analyzing the user's past purchase history and notifying them of sales promotion information and guidance. This enables the user to securely manage a wide range of information and be automatically guided to make the best choices tailored to their needs.

[0479] A "user" refers to an individual or legal entity that uses the system to conduct digital transactions or manage information.

[0480] A "terminal device" refers to an electronic device that a user operates to acquire information from a physical medium, and includes smartphones and tablets.

[0481] "Physical media" refers to physical information media owned by users, such as credit cards and loyalty cards.

[0482] "Acquiring an image" refers to taking a photograph via a terminal device to obtain information from a physical medium.

[0483] "Digitization" refers to the process of analyzing textual information from images acquired from physical media and saving it as electronic data.

[0484] "Location information" refers to information obtained by a terminal device that indicates the user's current location.

[0485] "Facility information" refers to information about shops and public facilities located in the user's vicinity.

[0486] "Purchase history" refers to a record of purchases a user has made in the past, and purchasing patterns can be derived through analysis.

[0487] "Sales promotion information" refers to information about discounts and campaigns that are notified to users based on their purchase history and location information.

[0488] "Physiological information" refers to data that indicates the user's physical state, such as heart rate and stress level.

[0489] "Behavioral information" refers to data about a user's movement and activities, and is acquired from terminal devices and wearable devices.

[0490] This invention is a system for efficiently and securely managing information related to users' digital transactions and providing users with the information they need in a timely manner. This system mainly consists of terminal devices, servers, and related devices.

[0491] The terminal device uses a camera to capture images of physical media owned by the user, such as a credit card. This image acquisition process utilizes OCR technology running on the terminal device. Specifically, a general-purpose OCR engine (e.g., Tesseract or other image analysis engines) is used to analyze textual information from the image. The resulting textual information is stored as digital data within the terminal and transmitted to a secure database. During this process, the data is encrypted using SSL / TLS to ensure security.

[0492] Furthermore, the terminal device utilizes GPS functionality to continuously acquire the user's location information and transmit it to the server. Based on the received location information, the server searches its database for information on facilities around the user and provides the user with relevant information. This information provision includes, in particular, purchasing patterns and campaign information. The server also uses machine learning techniques to analyze the user's past purchase history and provides individually customized sales promotion information.

[0493] The terminal device works in conjunction with a dedicated wearable device to collect the user's physiological and behavioral information. Wireless communication technologies such as Bluetooth are used for this connection. By analyzing the collected information, personalized suggestions based on the user's current condition become possible.

[0494] This system incorporates security features that monitor users' internet activity and issue warnings when unauthorized access or connections to fraudulent websites are detected. This feature protects users from potential dangers.

[0495] Specific examples include situations where a user is walking near a cafe and wants to know if there are any perks or loyalty programs available at that cafe, or where they want recommendations for relaxation products when they are feeling stressed. This information is efficiently provided through prompt messages to support the user's decision-making.

[0496] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0497] Step 1:

[0498] The user uses a terminal device to acquire an image of a physical medium. The input is a physical medium such as a credit card. The terminal analyzes the image using OCR technology to obtain textual information such as the card number and expiration date. Through this data processing, the physical information is output as digital data. This output data is securely stored in a secure database.

[0499] Step 2:

[0500] The device uses GPS functionality to obtain the user's current location. The input is the user's real-time geographical location. The device sends this location information to the server, providing basic data for searching for facility information around the user. The server consults a facility database and searches for relevant information based on the location, outputting facility information and promotional information that should be provided to the user.

[0501] Step 3:

[0502] The server retrieves user purchase history data from a database. The input is the user's past transaction records. Based on this, the server uses machine learning algorithms to analyze purchasing patterns. This data analysis predicts products and campaign information that the user is likely to be interested in. As output, personalized sales promotion information is generated based on this analysis.

[0503] Step 4:

[0504] The terminal connects to the wearable device via Bluetooth and receives physiological information and activity data in real time. Inputs include physiological information such as heart rate and step count. The terminal analyzes this data to evaluate the user's current condition. Based on this data analysis, optimized product recommendations tailored to the user's state are output and notified to the user as prompt messages.

[0505] Step 5:

[0506] The server monitors the user's internet activity. The input is information about the websites the user accesses. Security software performs data calculations to detect potential unauthorized access. As a result of this calculation, it warns of access to potentially dangerous sites and outputs information to enhance security. The terminal notifies the user of this warning and prompts them to restrict or stop access.

[0507] (Application Example 1)

[0508] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0509] Modern consumers engage in purchasing activities in an environment overflowing with vast amounts of electronic transaction and marketing information. In this context, it is difficult for consumers to appropriately select the most beneficial information and conduct electronic transactions securely. Furthermore, systems that provide real-time product recommendations tailored to individual needs are scarce, and measures to protect personal information from fraudulent information and scams are needed. Solving these challenges and enabling consumers to conduct electronic transactions with peace of mind while providing information tailored to their individual needs is crucial.

[0510] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0511] In this invention, the server includes means for processing data related to the user's electronic transactions, such as acquiring images of physical identification information from the user's terminal and storing them as digital data; means for associating spatial information and commercial facility information using the acquired digital data and presenting relevant information; means for analyzing past purchase history and notifying sales promotion information based on predicted patterns; and means for monitoring activities related to online sites and warning of access to risky sites. This allows consumers to obtain the most relevant information and conduct electronic transactions with peace of mind.

[0512] A "user terminal" is an information processing device used by a user, and is a device that has the role of acquiring and processing electronic data.

[0513] "Physical identification information" refers to a means of identifying an entity that includes a user's personal information and payment information, and is typically embodied as a card or tag.

[0514] "Digital data" refers to data formats that are created by electronically converting physical information into a form that can be stored within a computer system.

[0515] "Spatial information" refers to information related to geographical location, and is data that identifies where users or facilities are located.

[0516] "Commercial facility information" refers to information about facilities that conduct commercial activities, including location, business hours, and information about the products offered.

[0517] "Past purchase history" refers to a record of a user's past purchasing activities, including information such as the products purchased and the date and time.

[0518] "Sales promotion information" refers to information provided to consumers to promote the sale of products, and includes details about discounts and campaigns.

[0519] An "online site" refers to a webpage or web service provided on the internet, where information is shared and electronic transactions take place.

[0520] "Risk" refers to factors or circumstances that could threaten user data or security, and typically involves the risk of fraud or data breaches.

[0521] The system for implementing this invention first acquires physical identification information using a camera mounted on the user terminal and converts it into digital data using OCR technology. The terminal then uses the Google Cloud Vision API to perform text analysis on the image and securely stores the results in the Firebase Realtime Database.

[0522] Furthermore, the device uses the Google Maps API to obtain the user's current location and associates spatial information with commercial facility information based on that location data. Based on this, relevant sales promotion information is provided via push notifications.

[0523] In purchase history analysis, the server collects users' past purchase data and analyzes the patterns to provide personalized sales promotion information. This analysis utilizes machine learning algorithms to deliver customized information to users.

[0524] Furthermore, if a user is using a wearable device, biometric information is acquired from that device, and the user's physical and mental state is analyzed using the Fitbit API. This makes it possible to suggest products that are tailored to the user's current situation.

[0525] From a security perspective, the server uses IBM Watson Security Advisor to monitor users' online activity and immediately issues a warning if access to online sites with potential risks occurs.

[0526] Specific examples include providing real-time notifications on a user's smartphone about coffee discounts when they are near a particular cafe. Additionally, when a user's stress level is high, the system could provide information about products with relaxing effects.

[0527] An example of using a generative AI model to generate prompt text is to input the question, "How can an app notify users of promotional information for specific stores based on their current location?" into the model and obtain an appropriate answer.

[0528] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0529] Step 1:

[0530] The user's device uses its camera to acquire an image of physical identification information.

[0531] The input is an image of a physical card captured by a camera. The output is that image data, which is saved to local storage for further processing.

[0532] Step 2:

[0533] The device uses the Google Cloud Vision API to extract text information from the image.

[0534] The input is the image data acquired in Step 1. The image is analyzed using OCR technology to extract text data such as the card number and expiration date. The output is digital data in text format and is stored in the Firebase Realtime Database.

[0535] Step 3:

[0536] The device uses the Google Maps API to obtain the user's current location and collect spatial information.

[0537] The input is the user's current location. Location data is acquired using a GPS sensor and then cross-referenced with commercial facility information. The output is information about commercial facilities corresponding to the user's location.

[0538] Step 4:

[0539] The server analyzes past purchase history to predict the user's consumption patterns.

[0540] The input is purchase history data stored in the Firebase Realtime Database. A machine learning algorithm is used to analyze past data and identify predictable purchase patterns. The output is a list of promotional information and campaigns.

[0541] Step 5:

[0542] Wearable devices collect biometric information and evaluate health status via the Fitbit API.

[0543] The input consists of biometric data acquired by the device, such as heart rate and stress level. The Fitbit API is used to assess the user's current health status, and the output provides a report on the user's physical and mental condition.

[0544] Step 6:

[0545] The server monitors users' online activity and assesses risks using IBM Watson Security Advisor.

[0546] The input consists of URLs of online sites the user has accessed and activity logs. If a potential risk is detected, a warning is issued immediately. The output consists of the security assessment results and a warning message.

[0547] Step 7:

[0548] Promotional information will be sent to users based on their location.

[0549] The input consists of the commercial facility information obtained in step 3 and the sales promotion information from step 4. The terminal sends a push notification to the user informing them of available promotional information. The output is the notification message the user receives.

[0550] Step 8:

[0551] Generate product suggestions and notify users.

[0552] The input consists of the physical and mental condition information obtained in Step 5 and the sales promotion information from Step 4. A generative AI model is used to create optimal product suggestions for the user and notify the user via their device. The output is a list of recommended products presented to the user.

[0553] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0554] This invention is a system that incorporates an emotion engine to more highly optimize users' electronic transactions. This system is implemented using a user terminal, a server, and emotion recognition technology as follows:

[0555] The user's device first uses its camera and sensors to collect information such as the user's facial expressions and voice. An emotion engine built into the device analyzes this information to identify the user's emotional state. This emotional data is securely transmitted to a server with the user's consent.

[0556] The server records and analyzes the user's emotional state in real time based on the received emotional data. This analysis is combined with the user's purchase history and location information to generate optimal promotional and campaign information for the user. For example, if the server determines that the user is stressed, it will provide information on products suitable for relaxation.

[0557] Furthermore, the emotion engine continuously monitors the user's emotional changes and can flexibly adjust purchase suggestions according to their emotional state at any given time. For example, if the user is feeling happy, it will suggest active products and services.

[0558] Furthermore, this system maintains a security feature that warns users of access to fraudulent websites. By using emotion engine data, it can ensure a higher level of security by providing extra attention when users are feeling anxious.

[0559] This invention allows users to receive personalized services based on their emotional state, enabling them to enjoy a more fulfilling electronic transaction experience.

[0560] The following describes the processing flow.

[0561] Step 1:

[0562] The user's device activates an emotion engine and uses its camera and microphone to collect emotion data from the user's facial expressions and voice. This data is processed immediately on the device.

[0563] Step 2:

[0564] The device's built-in emotion engine analyzes collected information to determine the user's current emotional state in real time. For example, a smiling face indicates "joy," while a frown indicates "anxiety."

[0565] Step 3:

[0566] The device sends the determined emotion data to the server with the user's consent. The server then integrates this data with other user information (purchase history and location information).

[0567] Step 4:

[0568] The server analyzes the integrated data and generates promotional and campaign information optimized for the user's emotional state. For example, if stress is detected, discount information on relaxation products will be provided.

[0569] Step 5:

[0570] The server sends the generated promotional information to the device and sends a push notification to the user at the appropriate time. This notification allows the user to check the suggested products and services.

[0571] Step 6:

[0572] The user terminal continuously monitors emotional data, and if emotions change, it re-evaluates the emotional state and repeats the process from step 2 onward.

[0573] Step 7:

[0574] The server also monitors internet activity and, if it detects access to potentially risky websites, it issues additional warnings to the user, especially if the emotion engine detects anxiety.

[0575] (Example 2)

[0576] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0577] In modern e-commerce, information and promotions are often provided uniformly without considering the user's emotional state, making it difficult to deliver a truly valuable, personalized experience. Furthermore, the lack of flexible service delivery that reflects changes in user emotions in real time means missed opportunities to improve purchase satisfaction. Additionally, there are insufficient means to alleviate users' anxiety when accessing potentially risky websites.

[0578] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0579] In this invention, the server includes means for the user terminal to acquire the user's facial expressions and voice information using a camera and sensors, and for analyzing the acquired information to identify the user's emotional state; means for securely transmitting the identified emotional data to the server for real-time recording and analysis; and means for integrating the received emotional data with the user's past purchase history and location information to generate personalized suggestions for the user. This makes it possible to provide real-time and personalized information and purchase suggestions based on the user's emotions.

[0580] A "user terminal" is a device used by users to input information and collect emotional data, and is equipped with cameras and sensors.

[0581] "Cameras and sensors" are devices used to acquire user facial expressions and voice information, and have the function of collecting data in real time.

[0582] "Emotional state" refers to the psychological state analyzed from the user's facial expressions and voice, and includes emotions such as stress and relaxation.

[0583] A "server" is a computer device that receives emotional data transmitted from user terminals and records and analyzes it in real time.

[0584] "Personalized recommendations" refer to the provision of promotions and information optimized for individual users, generated based on their past purchase history and location information.

[0585] "Emotional analysis results" refer to the results of an analysis of the user's emotional state, presented as numerical values ​​or indicators.

[0586] "Potential risks" refer to the safety and security threats to websites and online services that users attempt to access.

[0587] This invention provides a system equipped with an emotion engine to highly optimize users' electronic transactions, and is implemented using a user terminal and a server. The user terminal is equipped with a camera and various sensors to collect the user's facial expressions and voice in real time. Specifically, in addition to a general computer device, the user terminal uses a camera for facial recognition and a microphone for capturing voice. This allows the terminal to analyze the user's emotional state in detail. For emotion analysis, software utilizing machine learning as the emotion engine is used to extract emotional data from the user's facial expressions and voice.

[0588] The device sends analyzed emotional data to the server with the user's consent. The server receives this data and records and analyzes it in real time. During this process, the server integrates past purchase history and location information using a database and utilizes the latest AI models to generate optimized information for the user. For example, if a user is experiencing stress, the server can suggest relaxation products and services, providing a more personalized service.

[0589] Furthermore, the server also monitors website access and warns users of potentially risky sites if they feel uneasy. Such security features enhance user confidence.

[0590] For example, if the terminal detects that the user is displaying a happy expression while shopping, the server immediately suggests products for summer camps and outdoor activities. At this point, prompts such as, "You are receiving personalized product suggestions based on your current emotional state. How does this system analyze the user's emotional data and generate purchase suggestions?" are used, enabling the generating AI model to provide optimal suggestions. In this way, the present invention provides personalized suggestions tailored to the user's emotions, offering a more valuable electronic transaction experience.

[0591] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0592] Step 1:

[0593] The user's device uses its camera and sensors to capture the user's facial expressions and voice.

[0594] The input is the user's real-time facial expressions and voice, and the output is digitized information of this data. The terminal prepares to analyze the user's emotional state based on this data. Specifically, the camera captures the user's face while the microphone records their voice.

[0595] Step 2:

[0596] An emotion engine built into the device analyzes digitized facial and voice data to identify the user's emotional state.

[0597] The input is the digitized data obtained in Step 1, and the output is an index or numerical value indicating the user's emotions. Based on this input data, the emotion engine uses a machine learning algorithm to analyze patterns in the emotional data and identify emotional states such as "joy" or "stress."

[0598] Step 3:

[0599] The device sends analyzed emotional data to the server with the user's consent.

[0600] The input is the emotional state identified in step 2, and the output is the emotional data sent to the server. Specifically, the terminal uses data encryption technology to securely send the data to the server.

[0601] Step 4:

[0602] The server receives emotional data and records and analyzes it in real time.

[0603] The input is sentiment data sent from the device, and the output is a user profile that integrates the analyzed sentiment data. The server uses a database to match sentiment data with past purchase history and location information, preparing to generate personalized information.

[0604] Step 5:

[0605] The server generates personalized suggestion information for the user and sends it to the terminal.

[0606] The input is an integrated user profile, and the output is personalized promotional and campaign information presented to the user. Using a generative AI model, it provides optimal suggestions that match the user's emotional state; for example, if the user needs relaxation, it recommends relaxation products.

[0607] Step 6:

[0608] The server monitors users' access to websites and warns them about accessing potentially risky sites if they feel uneasy.

[0609] The input is the user's emotional state and website access information, and the output is a warning message. Specifically, the server protects the user by displaying a message such as "This site may be unsafe" when the user attempts to access a site that may be fraudulent.

[0610] (Application Example 2)

[0611] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0612] There is a growing need to optimize electronic transactions based on users' emotional states and provide personalized services and security measures for individual users. However, current technology does not adequately reflect user emotions in product recommendations or improve security. It is necessary to address this challenge and provide a more fulfilling electronic transaction experience.

[0613] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0614] In this invention, the server includes means for the user terminal to acquire facial expression information and voice information using sensors to recognize the user's emotional state, analyze the acquired information and securely transmit emotional data to the server, analyze the acquired emotional data in combination with purchase history and generate optimized product suggestions based on the emotional state, and warn the user to access fraudulent sites to improve security when the user shows signs of anxiety. This enables personalized product suggestions that respond to the user's emotions and improved security.

[0615] A "user terminal" is a type of computer device operated by a user, and its role is to collect and process information.

[0616] A "sensor" is a device that collects physical information and outputs it as an electrical signal.

[0617] "Facial expression information" refers to data that represents the movements and characteristics of a user's face, and is used for emotion recognition.

[0618] "Audio information" refers to sound data, including the user's speech, which is used to analyze emotions and intentions.

[0619] "Emotional data" refers to data that quantifies or classifies a user's emotional state.

[0620] A "server" is a computer system used to process data and provide services over a network.

[0621] "Purchase history" refers to a record of products a user has purchased in the past, and serves as basic data for predicting future purchasing patterns.

[0622] "Optimized product recommendations" refer to personalized recommendations of products and services that take into account the user's needs and emotional state.

[0623] "Means of warning against accessing fraudulent sites to improve security" refers to technologies that have the function of issuing warnings to prevent users from accessing dangerous websites.

[0624] "Personalized product recommendations" refer to recommendations for products and services that are specially customized based on the user's individual attributes and circumstances.

[0625] This system consists of a user terminal, sensors, a server, and emotion recognition software.

[0626] The user terminal uses a camera and microphone to collect the user's facial expressions and voice information. This information is transmitted to the emotion recognition engine via sensors. The emotion recognition engine uses the Google Cloud Vision API and a natural language processing engine for voice analysis to analyze the user's emotional state. The analyzed emotion data is securely transmitted to the server with the user's consent.

[0627] The server integrates and analyzes received sentiment data with the user's past purchase history. Using Python, the server processes the data and generates optimal product recommendations based on the user's current emotional state. These recommendations are delivered to the user via email or in-app notifications. Furthermore, if the user expresses anxiety, the server immediately issues a warning about fraudulent websites to enhance security. This utilizes specialized software to strengthen security features.

[0628] For example, if the system analyzes that a user has a high need for relaxation while operating their device, the server may suggest relaxation-related products such as aroma diffusers or massage chairs. If the user is feeling stressed, the system may also encourage them to purchase yoga class vouchers or relaxation music.

[0629] An example of a prompt message when using a generative AI model is: "If the user is feeling stressed, create a list of recommended products. For example, list products with relaxation effects."

[0630] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0631] Step 1:

[0632] The user terminal uses a camera and microphone to acquire the user's facial expressions and voice information. The input is a facial image and voice, and the output is raw facial expression data and voice data. The terminal transmits this information to the emotion recognition engine via sensors.

[0633] Step 2:

[0634] The server uses an emotion recognition engine to analyze acquired facial and audio information. The input consists of facial image data and audio data, while the output is emotion data indicating the user's emotional state. The analysis utilizes the Google Cloud Vision API and an audio analysis engine, processing the data in real time.

[0635] Step 3:

[0636] The server integrates and analyzes emotional data and the user's past purchase history. The input is emotional data and past purchase history data, and the output is an optimized product suggestion list based on the emotional state. Python is used to combine this data and leverage a generative AI model to generate product suggestions.

[0637] Step 4:

[0638] The server provides the user with generated product suggestions. The input is a list of product suggestions, and the output is a notification to the user. The notification is delivered via email or in-app notification, allowing the user to receive product information that matches their emotional state.

[0639] Step 5:

[0640] If a user shows signs of anxiety while operating their device, the server will warn them about accessing a fraudulent website. The input is user sentiment data, and the output is a security warning. The server uses specialized security software to take action to ensure the user's safety.

[0641] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0642] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0643] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0644] [Fourth Embodiment]

[0645] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0646] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0647] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0648] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0649] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0650] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0651] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0652] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0653] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0654] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0655] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0656] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0657] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0658] The system of the present invention is designed to efficiently and securely manage users' electronic transactions. This system is implemented primarily using user terminals, servers, and related devices, as follows:

[0659] The user terminal process begins with the user acquiring an image of their physical card. The image captured by the camera is analyzed within the terminal using OCR technology, and the card number and expiration date are extracted as digital data. This digitized card information is stored in a secure database, which the user can easily access and manage through the app.

[0660] The user's device also continuously acquires the user's location information and works with the server to search for nearby store information. Based on this location information, the system automatically notifies the user of available loyalty cards and related promotional information. For example, if the user is near a specific store, information about loyalty cards that can be used at that store will be sent via push notification.

[0661] Furthermore, the server analyzes the user's purchase history, collects and compares requested information based on the user's purchasing patterns, and provides the user with appropriate campaign information. For example, discount information on regularly purchased products and notifications of related campaigns are sent to the user's device. This allows users to efficiently take advantage of advantageous opportunities.

[0662] Furthermore, the user terminal connects with a dedicated wearable device to analyze the user's biometric information and behavioral data, providing personalized product recommendations based on their current condition. For example, the wearable device may measure the user's stress level and suggest products that can help reduce stress.

[0663] In terms of security, the server monitors the user's internet activity and immediately issues a warning if the user attempts to connect to a fraudulent site or a site potentially involving unauthorized access. This significantly reduces the risk of users becoming victims of fraud or other malicious activity.

[0664] By combining these functions, the system of the present invention makes it possible for users to simplify and securely use electronic money and point systems.

[0665] The following describes the processing flow.

[0666] Step 1:

[0667] The user launches the app and opens a screen to retrieve an image of the physical card. The device uses its camera to take a picture of the card and saves the image.

[0668] Step 2:

[0669] The device uses OCR technology to analyze the image of the card that has been photographed, and extracts textual information such as the card number and expiration date. This information is stored as electronic data within the app.

[0670] Step 3:

[0671] The device periodically obtains the user's location information via GPS. When the user approaches a store, the device sends this information to the server.

[0672] Step 4:

[0673] The server searches for nearby store information based on the received location data. It then sends information about available loyalty cards and promotions to the device.

[0674] Step 5:

[0675] The device pushes promotional information to the user, providing them with purchasing opportunities at the optimal time.

[0676] Step 6:

[0677] The server analyzes the user's purchase history and extracts campaign information based on their purchase patterns. It then notifies the user's device of the relevant information.

[0678] Step 7:

[0679] The user's terminal acquires biometric information and behavioral data from wearable devices. Data analysis is used to determine the user's current condition and provide personalized purchase suggestions.

[0680] Step 8:

[0681] The server monitors internet activity and compares it against a list of known dangerous sites. If a potential risk is detected, it immediately alerts the user.

[0682] (Example 1)

[0683] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0684] In recent years, the proliferation of electronic transactions has increased the complexity of information management for users. Users conduct transactions using various digital media, and are required to provide location information, information related to internet activity, and even health management and promotional optimization using biometric data. However, there are limited systems that can centrally and securely manage this information and guide users to make appropriate choices. There is a need to provide effective solutions that can address this situation.

[0685] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0686] In this invention, the server includes means for a terminal device to acquire, digitize, and store images from a physical medium in order to process information related to the user's digital transactions; means for using the acquired information to associate the user's location information with facility information and provide information relevant to the user; and means for analyzing the user's past purchase history and notifying them of sales promotion information and guidance. This enables the user to securely manage a wide range of information and be automatically guided to make the best choices tailored to their needs.

[0687] A "user" refers to an individual or legal entity that uses the system to conduct digital transactions or manage information.

[0688] A "terminal device" refers to an electronic device that a user operates to acquire information from a physical medium, and includes smartphones and tablets.

[0689] "Physical media" refers to physical information media owned by users, such as credit cards and loyalty cards.

[0690] "Acquiring an image" refers to taking a photograph via a terminal device to obtain information from a physical medium.

[0691] "Digitization" refers to the process of analyzing textual information from images acquired from physical media and saving it as electronic data.

[0692] "Location information" refers to information obtained by a terminal device that indicates the user's current location.

[0693] "Facility information" refers to information about shops and public facilities located in the user's vicinity.

[0694] "Purchase history" refers to a record of purchases a user has made in the past, and purchasing patterns can be derived through analysis.

[0695] "Sales promotion information" refers to information about discounts and campaigns that are notified to users based on their purchase history and location information.

[0696] "Physiological information" refers to data that indicates the user's physical state, such as heart rate and stress level.

[0697] "Behavioral information" refers to data about a user's movement and activities, and is acquired from terminal devices and wearable devices.

[0698] This invention is a system for efficiently and securely managing information related to users' digital transactions and providing users with the information they need in a timely manner. This system mainly consists of terminal devices, servers, and related devices.

[0699] The terminal device uses a camera to capture images of physical media owned by the user, such as a credit card. This image acquisition process utilizes OCR technology running on the terminal device. Specifically, a general-purpose OCR engine (e.g., Tesseract or other image analysis engines) is used to analyze textual information from the image. The resulting textual information is stored as digital data within the terminal and transmitted to a secure database. During this process, the data is encrypted using SSL / TLS to ensure security.

[0700] Furthermore, the terminal device utilizes GPS functionality to continuously acquire the user's location information and transmit it to the server. Based on the received location information, the server searches its database for information on facilities around the user and provides the user with relevant information. This information provision includes, in particular, purchasing patterns and campaign information. The server also uses machine learning techniques to analyze the user's past purchase history and provides individually customized sales promotion information.

[0701] The terminal device works in conjunction with a dedicated wearable device to collect the user's physiological and behavioral information. Wireless communication technologies such as Bluetooth are used for this connection. By analyzing the collected information, personalized suggestions based on the user's current condition become possible.

[0702] This system incorporates security features that monitor users' internet activity and issue warnings when unauthorized access or connections to fraudulent websites are detected. This feature protects users from potential dangers.

[0703] Specific examples include situations where a user is walking near a cafe and wants to know if there are any perks or loyalty programs available at that cafe, or where they want recommendations for relaxation products when they are feeling stressed. This information is efficiently provided through prompt messages to support the user's decision-making.

[0704] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0705] Step 1:

[0706] The user uses a terminal device to acquire an image of a physical medium. The input is a physical medium such as a credit card. The terminal analyzes the image using OCR technology to obtain textual information such as the card number and expiration date. Through this data processing, the physical information is output as digital data. This output data is securely stored in a secure database.

[0707] Step 2:

[0708] The device uses GPS functionality to obtain the user's current location. The input is the user's real-time geographical location. The device sends this location information to the server, providing basic data for searching for facility information around the user. The server consults a facility database and searches for relevant information based on the location, outputting facility information and promotional information that should be provided to the user.

[0709] Step 3:

[0710] The server retrieves user purchase history data from a database. The input is the user's past transaction records. Based on this, the server uses machine learning algorithms to analyze purchasing patterns. This data analysis predicts products and campaign information that the user is likely to be interested in. As output, personalized sales promotion information is generated based on this analysis.

[0711] Step 4:

[0712] The terminal connects to the wearable device via Bluetooth and receives physiological information and activity data in real time. Inputs include physiological information such as heart rate and step count. The terminal analyzes this data to evaluate the user's current condition. Based on this data analysis, optimized product recommendations tailored to the user's state are output and notified to the user as prompt messages.

[0713] Step 5:

[0714] The server monitors the user's internet activity. The input is information about the websites the user accesses. Security software performs data calculations to detect potential unauthorized access. As a result of this calculation, it warns of access to potentially dangerous sites and outputs information to enhance security. The terminal notifies the user of this warning and prompts them to restrict or stop access.

[0715] (Application Example 1)

[0716] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0717] Modern consumers engage in purchasing activities in an environment overflowing with vast amounts of electronic transaction and marketing information. In this context, it is difficult for consumers to appropriately select the most beneficial information and conduct electronic transactions securely. Furthermore, systems that provide real-time product recommendations tailored to individual needs are scarce, and measures to protect personal information from fraudulent information and scams are needed. Solving these challenges and enabling consumers to conduct electronic transactions with peace of mind while providing information tailored to their individual needs is crucial.

[0718] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0719] In this invention, the server includes means for processing data related to the user's electronic transactions, such as acquiring images of physical identification information from the user's terminal and storing them as digital data; means for associating spatial information and commercial facility information using the acquired digital data and presenting relevant information; means for analyzing past purchase history and notifying sales promotion information based on predicted patterns; and means for monitoring activities related to online sites and warning of access to risky sites. This allows consumers to obtain the most relevant information and conduct electronic transactions with peace of mind.

[0720] A "user terminal" is an information processing device used by a user, and is a device that has the role of acquiring and processing electronic data.

[0721] "Physical identification information" refers to a means of identifying an entity that includes a user's personal information and payment information, and is typically embodied as a card or tag.

[0722] "Digital data" refers to data formats that are created by electronically converting physical information into a form that can be stored within a computer system.

[0723] "Spatial information" refers to information related to geographical location, and is data that identifies where users or facilities are located.

[0724] "Commercial facility information" refers to information about facilities that conduct commercial activities, including location, business hours, and information about the products offered.

[0725] "Past purchase history" refers to a record of a user's past purchasing activities, including information such as the products purchased and the date and time.

[0726] "Sales promotion information" refers to information provided to consumers to promote the sale of products, and includes details about discounts and campaigns.

[0727] An "online site" refers to a webpage or web service provided on the internet, where information is shared and electronic transactions take place.

[0728] "Risk" refers to factors or circumstances that could threaten user data or security, and typically involves the risk of fraud or data breaches.

[0729] The system for implementing this invention first acquires physical identification information using a camera mounted on the user terminal and converts it into digital data using OCR technology. The terminal then uses the Google Cloud Vision API to perform text analysis on the image and securely stores the results in the Firebase Realtime Database.

[0730] Furthermore, the device uses the Google Maps API to obtain the user's current location and associates spatial information with commercial facility information based on that location data. Based on this, relevant sales promotion information is provided via push notifications.

[0731] In purchase history analysis, the server collects users' past purchase data and analyzes the patterns to provide personalized sales promotion information. This analysis utilizes machine learning algorithms to deliver customized information to users.

[0732] Furthermore, if a user is using a wearable device, biometric information is acquired from that device, and the user's physical and mental state is analyzed using the Fitbit API. This makes it possible to suggest products that are tailored to the user's current situation.

[0733] From a security perspective, the server uses IBM Watson Security Advisor to monitor users' online activity and immediately issues a warning if access to online sites with potential risks occurs.

[0734] Specific examples include providing real-time notifications on a user's smartphone about coffee discounts when they are near a particular cafe. Additionally, when a user's stress level is high, the system could provide information about products with relaxing effects.

[0735] An example of using a generative AI model to generate prompt text is to input the question, "How can an app notify users of promotional information for specific stores based on their current location?" into the model and obtain an appropriate answer.

[0736] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0737] Step 1:

[0738] The user's device uses its camera to acquire an image of physical identification information.

[0739] The input is an image of a physical card captured by a camera. The output is that image data, which is saved to local storage for further processing.

[0740] Step 2:

[0741] The device uses the Google Cloud Vision API to extract text information from the image.

[0742] The input is the image data acquired in Step 1. The image is analyzed using OCR technology to extract text data such as the card number and expiration date. The output is digital data in text format and is stored in the Firebase Realtime Database.

[0743] Step 3:

[0744] The device uses the Google Maps API to obtain the user's current location and collect spatial information.

[0745] The input is the user's current location. Location data is acquired using a GPS sensor and then cross-referenced with commercial facility information. The output is information about commercial facilities corresponding to the user's location.

[0746] Step 4:

[0747] The server analyzes past purchase history to predict the user's consumption patterns.

[0748] The input is purchase history data stored in the Firebase Realtime Database. A machine learning algorithm is used to analyze past data and identify predictable purchase patterns. The output is a list of promotional information and campaigns.

[0749] Step 5:

[0750] Wearable devices collect biometric information and evaluate health status via the Fitbit API.

[0751] The input consists of biometric data acquired by the device, such as heart rate and stress level. The Fitbit API is used to assess the user's current health status, and the output provides a report on the user's physical and mental condition.

[0752] Step 6:

[0753] The server monitors users' online activity and assesses risks using IBM Watson Security Advisor.

[0754] The input consists of URLs of online sites the user has accessed and activity logs. If a potential risk is detected, a warning is issued immediately. The output consists of the security assessment results and a warning message.

[0755] Step 7:

[0756] Promotional information will be sent to users based on their location.

[0757] The input consists of the commercial facility information obtained in step 3 and the sales promotion information from step 4. The terminal sends a push notification to the user informing them of available promotional information. The output is the notification message the user receives.

[0758] Step 8:

[0759] Generate product suggestions and notify users.

[0760] The input consists of the physical and mental condition information obtained in Step 5 and the sales promotion information from Step 4. A generative AI model is used to create optimal product suggestions for the user and notify the user via their device. The output is a list of recommended products presented to the user.

[0761] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0762] This invention is a system that incorporates an emotion engine to more highly optimize users' electronic transactions. This system is implemented using a user terminal, a server, and emotion recognition technology as follows:

[0763] The user's device first uses its camera and sensors to collect information such as the user's facial expressions and voice. An emotion engine built into the device analyzes this information to identify the user's emotional state. This emotional data is securely transmitted to a server with the user's consent.

[0764] The server records and analyzes the user's emotional state in real time based on the received emotional data. This analysis is combined with the user's purchase history and location information to generate optimal promotional and campaign information for the user. For example, if the server determines that the user is stressed, it will provide information on products suitable for relaxation.

[0765] Furthermore, the emotion engine continuously monitors the user's emotional changes and can flexibly adjust purchase suggestions according to their emotional state at any given time. For example, if the user is feeling happy, it will suggest active products and services.

[0766] Furthermore, this system maintains a security feature that warns users of access to fraudulent websites. By using emotion engine data, it can ensure a higher level of security by providing extra attention when users are feeling anxious.

[0767] This invention allows users to receive personalized services based on their emotional state, enabling them to enjoy a more fulfilling electronic transaction experience.

[0768] The following describes the processing flow.

[0769] Step 1:

[0770] The user's device activates an emotion engine and uses its camera and microphone to collect emotion data from the user's facial expressions and voice. This data is processed immediately on the device.

[0771] Step 2:

[0772] The device's built-in emotion engine analyzes collected information to determine the user's current emotional state in real time. For example, a smiling face indicates "joy," while a frown indicates "anxiety."

[0773] Step 3:

[0774] The device sends the determined emotion data to the server with the user's consent. The server then integrates this data with other user information (purchase history and location information).

[0775] Step 4:

[0776] The server analyzes the integrated data and generates promotional and campaign information optimized for the user's emotional state. For example, if stress is detected, discount information on relaxation products will be provided.

[0777] Step 5:

[0778] The server sends the generated promotional information to the device and sends a push notification to the user at the appropriate time. This notification allows the user to check the suggested products and services.

[0779] Step 6:

[0780] The user terminal continuously monitors emotional data, and if emotions change, it re-evaluates the emotional state and repeats the process from step 2 onward.

[0781] Step 7:

[0782] The server also monitors internet activity and, if it detects access to potentially risky websites, it issues additional warnings to the user, especially if the emotion engine detects anxiety.

[0783] (Example 2)

[0784] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0785] In modern e-commerce, information and promotions are often provided uniformly without considering the user's emotional state, making it difficult to deliver a truly valuable, personalized experience. Furthermore, the lack of flexible service delivery that reflects changes in user emotions in real time means missed opportunities to improve purchase satisfaction. Additionally, there are insufficient means to alleviate users' anxiety when accessing potentially risky websites.

[0786] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0787] In this invention, the server includes means for the user terminal to acquire the user's facial expressions and voice information using a camera and sensors, and for analyzing the acquired information to identify the user's emotional state; means for securely transmitting the identified emotional data to the server for real-time recording and analysis; and means for integrating the received emotional data with the user's past purchase history and location information to generate personalized suggestions for the user. This makes it possible to provide real-time and personalized information and purchase suggestions based on the user's emotions.

[0788] A "user terminal" is a device used by users to input information and collect emotional data, and is equipped with cameras and sensors.

[0789] "Cameras and sensors" are devices used to acquire user facial expressions and voice information, and have the function of collecting data in real time.

[0790] "Emotional state" refers to the psychological state analyzed from the user's facial expressions and voice, and includes emotions such as stress and relaxation.

[0791] A "server" is a computer device that receives emotional data transmitted from user terminals and records and analyzes it in real time.

[0792] "Personalized recommendations" refer to the provision of promotions and information optimized for individual users, generated based on their past purchase history and location information.

[0793] "Emotional analysis results" refer to the results of an analysis of the user's emotional state, presented as numerical values ​​or indicators.

[0794] "Potential risks" refer to the safety and security threats to websites and online services that users attempt to access.

[0795] This invention provides a system equipped with an emotion engine to highly optimize users' electronic transactions, and is implemented using a user terminal and a server. The user terminal is equipped with a camera and various sensors to collect the user's facial expressions and voice in real time. Specifically, in addition to a general computer device, the user terminal uses a camera for facial recognition and a microphone for capturing voice. This allows the terminal to analyze the user's emotional state in detail. For emotion analysis, software utilizing machine learning as the emotion engine is used to extract emotional data from the user's facial expressions and voice.

[0796] The device sends analyzed emotional data to the server with the user's consent. The server receives this data and records and analyzes it in real time. During this process, the server integrates past purchase history and location information using a database and utilizes the latest AI models to generate optimized information for the user. For example, if a user is experiencing stress, the server can suggest relaxation products and services, providing a more personalized service.

[0797] Furthermore, the server also monitors website access and warns users of potentially risky sites if they feel uneasy. Such security features enhance user confidence.

[0798] For example, if the terminal detects that the user is displaying a happy expression while shopping, the server immediately suggests products for summer camps and outdoor activities. At this point, prompts such as, "You are receiving personalized product suggestions based on your current emotional state. How does this system analyze the user's emotional data and generate purchase suggestions?" are used, enabling the generating AI model to provide optimal suggestions. In this way, the present invention provides personalized suggestions tailored to the user's emotions, offering a more valuable electronic transaction experience.

[0799] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0800] Step 1:

[0801] The user's device uses its camera and sensors to capture the user's facial expressions and voice.

[0802] The input is the user's real-time facial expressions and voice, and the output is digitized information of this data. The terminal prepares to analyze the user's emotional state based on this data. Specifically, the camera captures the user's face while the microphone records their voice.

[0803] Step 2:

[0804] An emotion engine built into the device analyzes digitized facial and voice data to identify the user's emotional state.

[0805] The input is the digitized data obtained in Step 1, and the output is an index or numerical value indicating the user's emotions. Based on this input data, the emotion engine uses a machine learning algorithm to analyze patterns in the emotional data and identify emotional states such as "joy" or "stress."

[0806] Step 3:

[0807] The device sends analyzed emotional data to the server with the user's consent.

[0808] The input is the emotional state identified in step 2, and the output is the emotional data sent to the server. Specifically, the terminal uses data encryption technology to securely send the data to the server.

[0809] Step 4:

[0810] The server receives emotional data and records and analyzes it in real time.

[0811] The input is sentiment data sent from the device, and the output is a user profile that integrates the analyzed sentiment data. The server uses a database to match sentiment data with past purchase history and location information, preparing to generate personalized information.

[0812] Step 5:

[0813] The server generates personalized suggestion information for the user and sends it to the terminal.

[0814] The input is an integrated user profile, and the output is personalized promotional and campaign information presented to the user. Using a generative AI model, it provides optimal suggestions that match the user's emotional state; for example, if the user needs relaxation, it recommends relaxation products.

[0815] Step 6:

[0816] The server monitors users' access to websites and warns them about accessing potentially risky sites if they feel uneasy.

[0817] The input is the user's emotional state and website access information, and the output is a warning message. Specifically, the server protects the user by displaying a message such as "This site may be unsafe" when the user attempts to access a site that may be fraudulent.

[0818] (Application Example 2)

[0819] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0820] There is a growing need to optimize electronic transactions based on users' emotional states and provide personalized services and security measures for individual users. However, current technology does not adequately reflect user emotions in product recommendations or improve security. It is necessary to address this challenge and provide a more fulfilling electronic transaction experience.

[0821] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0822] In this invention, the server includes means for the user terminal to acquire facial expression information and voice information using sensors to recognize the user's emotional state, analyze the acquired information and securely transmit emotional data to the server, analyze the acquired emotional data in combination with purchase history and generate optimized product suggestions based on the emotional state, and warn the user to access fraudulent sites to improve security when the user shows signs of anxiety. This enables personalized product suggestions that respond to the user's emotions and improved security.

[0823] A "user terminal" is a type of computer device operated by a user, and its role is to collect and process information.

[0824] A "sensor" is a device that collects physical information and outputs it as an electrical signal.

[0825] "Facial expression information" refers to data that represents the movements and characteristics of a user's face, and is used for emotion recognition.

[0826] "Audio information" refers to sound data, including the user's speech, which is used to analyze emotions and intentions.

[0827] "Emotional data" refers to data that quantifies or classifies a user's emotional state.

[0828] A "server" is a computer system used to process data and provide services over a network.

[0829] "Purchase history" refers to a record of products a user has purchased in the past, and serves as basic data for predicting future purchasing patterns.

[0830] "Optimized product recommendations" refer to personalized recommendations of products and services that take into account the user's needs and emotional state.

[0831] "Means of warning against accessing fraudulent sites to improve security" refers to technologies that have the function of issuing warnings to prevent users from accessing dangerous websites.

[0832] "Personalized product recommendations" refer to recommendations for products and services that are specially customized based on the user's individual attributes and circumstances.

[0833] This system consists of a user terminal, sensors, a server, and emotion recognition software.

[0834] The user terminal uses a camera and microphone to collect the user's facial expressions and voice information. This information is transmitted to the emotion recognition engine via sensors. The emotion recognition engine uses the Google Cloud Vision API and a natural language processing engine for voice analysis to analyze the user's emotional state. The analyzed emotion data is securely transmitted to the server with the user's consent.

[0835] The server integrates and analyzes received sentiment data with the user's past purchase history. Using Python, the server processes the data and generates optimal product recommendations based on the user's current emotional state. These recommendations are delivered to the user via email or in-app notifications. Furthermore, if the user expresses anxiety, the server immediately issues a warning about fraudulent websites to enhance security. This utilizes specialized software to strengthen security features.

[0836] For example, if the system analyzes that a user has a high need for relaxation while operating their device, the server may suggest relaxation-related products such as aroma diffusers or massage chairs. If the user is feeling stressed, the system may also encourage them to purchase yoga class vouchers or relaxation music.

[0837] An example of a prompt message when using a generative AI model is: "If the user is feeling stressed, create a list of recommended products. For example, list products with relaxation effects."

[0838] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0839] Step 1:

[0840] The user terminal uses a camera and microphone to acquire the user's facial expressions and voice information. The input is a facial image and voice, and the output is raw facial expression data and voice data. The terminal transmits this information to the emotion recognition engine via sensors.

[0841] Step 2:

[0842] The server uses an emotion recognition engine to analyze acquired facial and audio information. The input consists of facial image data and audio data, while the output is emotion data indicating the user's emotional state. The analysis utilizes the Google Cloud Vision API and an audio analysis engine, processing the data in real time.

[0843] Step 3:

[0844] The server integrates and analyzes emotional data and the user's past purchase history. The input is emotional data and past purchase history data, and the output is an optimized product suggestion list based on the emotional state. Python is used to combine this data and leverage a generative AI model to generate product suggestions.

[0845] Step 4:

[0846] The server provides the user with generated product suggestions. The input is a list of product suggestions, and the output is a notification to the user. The notification is delivered via email or in-app notification, allowing the user to receive product information that matches their emotional state.

[0847] Step 5:

[0848] If a user shows signs of anxiety while operating their device, the server will warn them about accessing a fraudulent website. The input is user sentiment data, and the output is a security warning. The server uses specialized security software to take action to ensure the user's safety.

[0849] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0850] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0851] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0852] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0853] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0854] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0855] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0856] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0857] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0858] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0859] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0860] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0861] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0862] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0863] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0864] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0865] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0866] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0867] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0868] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0869] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0870] The following is further disclosed regarding the embodiments described above.

[0871] (Claim 1)

[0872] To process data related to the user's electronic transactions, the user terminal acquires an image of the physical card.

[0873] A method for extracting text information from acquired images and saving it as digital data,

[0874] The acquired electronic data is used to associate user location information with store information.

[0875] Means of presenting information relevant to the user,

[0876] By analyzing the user's past purchase history and predicting purchase patterns,

[0877] A means of notifying campaign information and promotions,

[0878] A means of monitoring website-related activity and warning of access to potentially risky sites,

[0879] A system that includes this.

[0880] (Claim 2)

[0881] It works in conjunction with a dedicated device that acquires the user's biometric information or behavioral data.

[0882] We analyze the acquired data and provide personalized suggestions based on the user's condition.

[0883] The system according to claim 1.

[0884] (Claim 3)

[0885] It has a means of recommending purchasing information necessary for specific events, in conjunction with the user's calendar information that manages their schedule.

[0886] The system according to claim 1.

[0887] "Example 1"

[0888] (Claim 1)

[0889] To process information related to the user's digital transactions, the terminal device acquires images from physical media,

[0890] A method for extracting text information from acquired images, digitizing it, and saving it,

[0891] The acquired digital information is used to associate the user's location information with facility information.

[0892] Means of providing information related to the user,

[0893] By analyzing the user's past purchase history and predicting purchase patterns,

[0894] A means of notifying sales promotion information and announcements,

[0895] A means of monitoring computer network-related activity and warning of access to potentially dangerous destinations,

[0896] A system that includes this.

[0897] (Claim 2)

[0898] It works in conjunction with a dedicated terminal that acquires the user's physiological or behavioral information.

[0899] The acquired information is analyzed to provide personalized suggestions based on the user's condition.

[0900] The system according to claim 1.

[0901] (Claim 3)

[0902] It has a means of linking with timetable information that manages the user's plans and recommending purchase information necessary for specific events.

[0903] The system according to claim 1.

[0904] "Application Example 1"

[0905] (Claim 1)

[0906] To process data related to the user's electronic transactions, the user terminal acquires an image of physical identification information.

[0907] A method for extracting text information from acquired images and saving it as digital data,

[0908] Using the acquired electronic data, we associate spatial information with commercial facility information.

[0909] Means of presenting information relevant to the user,

[0910] By analyzing the user's past purchase history and predicting purchase patterns,

[0911] Means of notifying sales promotion information,

[0912] A means of monitoring online site-related activity and warning of access to potentially risky sites,

[0913] A means for managing the user's digitized identification information and notifying them of available promotional information according to their location,

[0914] A means of presenting recommended products based on one's physical and mental state,

[0915] A system that includes this.

[0916] (Claim 2)

[0917] It works in conjunction with a dedicated device that acquires the user's biometric information or behavioral data.

[0918] The acquired data is analyzed to provide personalized product recommendations based on the user's condition.

[0919] The system according to claim 1.

[0920] (Claim 3)

[0921] It has a means of recommending purchasing information necessary for specific events, in conjunction with scheduling information that manages the user's schedule.

[0922] The system according to claim 1.

[0923] "Example 2 of combining an emotion engine"

[0924] (Claim 1)

[0925] To process data related to the user's electronic transactions, the user terminal uses a camera and sensors to acquire the user's facial expressions and voice information.

[0926] A means of analyzing acquired information to identify the user's emotional state,

[0927] A means of securely transmitting identified emotional data to a server for real-time recording and analysis,

[0928] A means for integrating received emotional data with the user's past purchase history and location information to generate personalized suggestions for the user,

[0929] A means to continuously monitor changes in user emotions and flexibly adjust purchase suggestions,

[0930] A means of monitoring website-related activity and warning users about accessing potentially risky sites when they feel uneasy,

[0931] A system that includes this.

[0932] (Claim 2)

[0933] The system according to claim 1, comprising an analysis engine for analyzing the user's emotional state and providing personalized suggestions based on the acquired emotional data.

[0934] (Claim 3)

[0935] The system according to claim 1, comprising means for providing purchasing information related to a specific event, taking into account the user's schedule information and emotional state.

[0936] "Application example 2 when combining with an emotional engine"

[0937] (Claim 1)

[0938] To recognize the user's emotional state, the user terminal uses sensors to acquire facial expression and voice information.

[0939] A means of analyzing the acquired information and securely transmitting emotional data to a server,

[0940] A means for analyzing acquired emotional data in combination with purchase history to generate optimized product recommendations based on emotional state,

[0941] When a user expresses anxiety, a means of warning them about accessing fraudulent sites to improve security,

[0942] A system that includes this.

[0943] (Claim 2)

[0944] The system according to claim 1, which analyzes biometric or behavioral data, including user emotional data, to provide personalized suggestions based on changes in the user's emotions.

[0945] (Claim 3)

[0946] The system according to claim 1, comprising means for recommending products tailored to specific events or situations based on purchase predictions that take into account emotional states. [Explanation of Symbols]

[0947] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. To process data related to the user's electronic transactions, the user terminal acquires an image of the physical card. A method for extracting text information from acquired images and saving it as digital data, The acquired electronic data is used to associate user location information with store information. Means of presenting information relevant to the user, By analyzing the user's past purchase history and predicting purchase patterns, A means of notifying campaign information and promotions, A means of monitoring website-related activity and warning of access to potentially risky sites, A system that includes this.

2. It works in conjunction with a dedicated device that acquires the user's biometric information or behavioral data. We analyze the acquired data and provide personalized suggestions based on the user's condition. The system according to claim 1.

3. It has a means of recommending purchasing information necessary for specific events, in conjunction with the user's calendar information that manages their schedule. The system according to claim 1.

Images