system

Abstract

A system is provided.SOLUTION: This system includes a voice input means for inputting the attribute information of a friend by voice, a voice recognition means for converting the voice-inputted attribute information into text data, a means for registering the friend by using a QR code or ID retrieval, and a database management means for analyzing the text data, and for identifying a corresponding group, and for allocating the friend to the group.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, the use of the LINE Messenger app for business purposes is increasing, but it can be inconvenient as it takes time to identify who a friend is, due to the fact that their handle names are abbreviated and the synchronization information for phone numbers is difficult to understand. Furthermore, the current grouping function is limited to "favorites," and as the number of friends increases, searching becomes time-consuming. A new system that can efficiently solve these problems is needed. [Means for solving the problem]

[0005] The present invention solves the aforementioned problems by providing the following means. Specifically, it provides a system that includes a voice input means for inputting friend attribute information by voice, a voice recognition means for converting the voice-input attribute information into text data, a means for registering friends using QR codes or ID search, and a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to those groups. The system also includes a means for scanning existing phone book data and comparing it with the LINE Messenger database, and a data linking means for acquiring and adding matching friend information from the phone book. The system also includes a means for classifying friend attribute information into specific groups using the database management means, and a means for allowing users to edit, add, or delete attribute information. In this way, a system is realized that achieves efficient management of friend information and simplified search.

[0006] The "voice input means" is an interface that allows the user to input attribute information of friends using voice.

[0007] "Speech recognition means" refers to technology or equipment for converting input speech into text data.

[0008] "Methods for registering friends" is a function that allows you to register new friends by scanning a QR code or searching by ID.

[0009] The "database management means" is a system management function for analyzing the text data, identifying the corresponding groups, and assigning friends appropriately.

[0010] "Data linking method" is a function that compares existing phone book data with the database within LINE Messenger and automatically retrieves and adds matching friend information.

[0011] A "group" is a category that users create to classify and manage their friends. Examples include "business partners" and "customers."

[0012] "Attribute information" is additional information necessary for the user to manage friends, such as their occupation, affiliation, and position.

[0013] The "voice input mode" is an operating state that allows the voice input means to be used.

[0014] "Text data" is character information converted by a voice recognition means.

[0015] "Matching" is the process of comparing existing phone book data with information in the LINE Messenger database to confirm matches. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention relates to a system for efficiently managing and classifying friend information in the LINE Messenger application. Specifically, it provides a function that allows users to register friends using QR codes or ID searches, and then classify the friends into appropriate groups using voice input. It also includes a function that links existing phone book data with LINE Messenger data and automatically updates friend attribute information.

[0038] Program processing

[0039] 1. Add friends

[0040] When a user opens the LINE Messenger application and scans a QR code or searches for an ID on the Add Friends screen, the device uses this information to store the friend's basic information (such as name, ID, and profile picture) in the LINE Messenger database.

[0041] 2. Audio grouping

[0042] After registering a friend, the user taps the microphone icon on the contact details screen to switch to voice input mode. When the user enters the friend's attribute information by voice, the device converts the voice into text data. The device then analyzes the converted text data to check whether a corresponding group exists. If no existing group exists, the device creates a new group, assigns the friend to it, and updates the database.

[0043] 3. Phonebook data integration

[0044] The device periodically scans the smartphone's contacts and compares them with the existing LINE Messenger database. If a match is found, additional information such as occupation and company name is retrieved from the contacts and added to the LINE Messenger contact information.

[0045] 4. Attribute Information Management

[0046] Users select the "Group Management" option from the friend list menu in the LINE Messenger app. The device displays all currently registered groups and friend information. Users can edit the displayed groups and attribute information, add new categories, or delete unnecessary groups. The device reflects the user's actions and updates the database accordingly.

[0047] Example: Classifying your friends' voices

[0048] For example, suppose a user registers a new friend using a QR code and then speaks "business partner." The device converts the speech into text data and assigns the friend to a group called "business partners." The device then scans the phone book to obtain the friend's detailed information (e.g., company name and occupation) and adds it to the LINE Messenger database.

[0049] This invention allows users to efficiently manage and classify information about their friends using only voice input, making it possible to quickly search and refer to the information they need, even when they have a large number of friends. Furthermore, the automatic linking function with phone book data significantly reduces the effort required for manually updating information.

[0050] The processing flow will be explained below.

[0051] Step 1:

[0052] The user opens the LINE Messenger application and scans the QR code on the Add Friends screen or searches for the ID. The device adds the friend information obtained from the QR code or ID to the LINE Messenger database.

[0053] Step 2:

[0054] After the registered friend information is displayed, the user taps the microphone icon on the contact details screen to switch to voice input mode, and the device enters a state waiting for the user's voice input.

[0055] Step 3:

[0056] The user inputs the group attribute by voice (e.g., "business partner"). The terminal receives the voice input and converts the voice into text data using a voice recognition means.

[0057] Step 4:

[0058] The device analyzes the converted text data and compares it with the LINE Messenger database. If there is no existing group, the device creates a new group. If there is an existing group, the device assigns friends to it.

[0059] Step 5:

[0060] The device scans the existing contacts data stored in the user's smartphone. The scan is performed periodically, for example, once a day.

[0061] Step 6:

[0062] The device compares the friend information registered in the LINE Messenger database with the data in the phone book. The server verifies the match and obtains additional information such as occupation and company name from the phone book.

[0063] Step 7:

[0064] The server integrates the additional information it has acquired into LINE Messenger's friend information, and the device updates the database to complete the friend's attribute information.

[0065] Step 8:

[0066] When a user selects the "Group Management" option from the friend list menu in the LINE Messenger application, all groups and friend information currently registered on the device is displayed.

[0067] Step 9:

[0068] Users can tap on a displayed group to edit their friends' attribute information. Users can add new attribute categories or delete unnecessary groups.

[0069] Step 10:

[0070] The device saves the user's edits to the database and reflects the latest status of group and attribute information, allowing the user to always manage the latest friend information.

[0071] Example 1

[0072] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0073] In conventional friend management systems, friend attribute information and group management are done manually, making management cumbersome, especially for users with many friends. Furthermore, since there is no function to update information in conjunction with phone book data, friend information is often out of date. For this reason, there is a demand for a system that can efficiently manage and classify friend information and automatically update the latest information.

[0074] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0075] In this invention, the server includes a voice input means for inputting friend attribute information by voice, a voice recognition means for converting the voice-input attribute information into text data, a means for registering friends using identification data, a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to those groups, and a data linking means for periodically scanning existing contact data, collating the information, acquiring additional information, and updating the database. This allows for efficient friend information management through voice input, and automatic updating of the latest information through linking with phone book data.

[0076] "Friend attribute information" is information relating to specific characteristics or categories of registered friends.

[0077] "Voice input means" refers to a device or function that allows a user to input information into a system using voice.

[0078] "Speech recognition means" refers to technology or devices for converting input speech into text data.

[0079] "Identification data" is data that uniquely identifies a friend, such as a QR code or ID.

[0080] "Database management means" refers to a system or device for efficiently managing, classifying, and storing friend information and related data.

[0081] "Data integration means" refers to the technology or function that compares existing contact data or other data sources with data within the system to obtain and supplement the necessary information.

[0082] A "group" is a category for managing friends who share common attribute information.

[0083] "Contact data" refers to data stored in a telephone directory or the like that includes information such as names, numbers, and company names.

[0084] This invention is a system for efficiently managing and classifying attribute information of friends. This system utilizes the LINE Messenger application, uses voice input to appropriately classify friends, and automatically links with phone book data to keep friend information up to date.

[0085] Hardware and software used

[0086] 1. LINE Messenger application: Used to register and manage friend information.

[0087] 2. Smartphone: The device on which the user runs the LINE Messenger app.

[0088] 3. Voice input engine (e.g. Google Speech-to-Text API): Used to convert voice into text data.

[0089] 4. Database system (e.g. SQLite): Used to store and update friend and group information.

[0090] Specific operation of the system

[0091] Add a friend

[0092] When a user opens the LINE Messenger application and scans a QR code or searches for an ID on the Add Friends screen, the device uses this information to store the friend's basic information (such as name, ID, and profile picture) in the LINE Messenger database.

[0093] Audio grouping

[0094] After registering a friend, the user taps the microphone icon on the contact details screen to switch to voice input mode. When the user enters attribute information such as "business partner" by voice, the device converts the voice into text data. The device analyzes the converted text data, and if there is no existing group, it creates a new group and assigns the friend to that group. The database management means processes this and updates the database.

[0095] Phone book data integration

[0096] The device periodically scans the smartphone's contacts and compares them with the existing LINE Messenger database. If a match is found, additional information such as occupation and company name is retrieved from the contacts and added to the LINE Messenger contacts. This keeps the contacts up to date.

[0097] Attribute information management

[0098] The user selects the "Group Management" option from the friend list menu in the LINE Messenger app. The device displays all registered groups and friend information. The user can edit the displayed groups and attribute information, add new categories, or delete unnecessary groups. The device reflects these operations and updates the database.

[0099] Examples of specific examples and prompts

[0100] For example, if a user registers a new friend using a QR code and speaks "business partner," the device converts the voice to text data and assigns the friend to a group called "business partners." The device then scans the phone book to obtain the friend's detailed information (e.g., company name and occupation) and adds it to the LINE Messenger database. This invention allows users to efficiently manage and categorize friend information using only voice input.

[0101] Example prompt sentence:

[0102] "How do I add new friends to the "Business Partners" group in the LINE Messenger app?"

[0103] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0104] Step 1:

[0105] Start friend registration

[0106] The user opens the LINE Messenger application and scans the QR code on the Add Friends screen or searches for the ID.

[0107] The device will retrieve the QR code information or ID search results based on this input and save the friend's basic information (name, ID, profile picture, etc.) in the LINE Messenger database.

[0108] Step 2:

[0109] Activating voice input mode

[0110] After users register friends using QR codes or ID searches, they can tap the microphone icon on the contact details screen to switch to voice input mode.

[0111] The device launches a voice input engine (e.g., Google Speech-to-Text API) and waits for the user's voice input.

[0112] Step 3:

[0113] Voice input of attribute information

[0114] The user inputs attribute information such as "business partner" by voice.

[0115] The device sends this voice data to the Google Speech-to-Text API and obtains the text data "Customer."

[0116] Input: Audio data

[0117] Output: Text data ("Customer")

[0118] Step 4:

[0119] Attribute information analysis and grouping

[0120] The text data "Customer" acquired by the device is analyzed using a natural language processing engine (e.g., NLTK) to check whether a corresponding group exists.

[0121] If the terminal does not have a group called "Customer", it will create a new group called "Customer".

[0122] The device assigns friends to the "Business Partners" group and updates the LINE Messenger database.

[0123] Input: Text data ("Customer")

[0124] Output: Updated database

[0125] Step 5:

[0126] Phone book data integration

[0127] The device periodically scans the phone book data on your smartphone.

[0128] The device compares the LINE Messenger database with the phone book data, and if a matching friend is found, it retrieves additional information from the phone book, such as occupation and company name.

[0129] The device will add the additional information it has acquired to the LINE Messenger friend information and update the database.

[0130] Input: Phonebook data

[0131] Output: Updated friend information

[0132] Step 6:

[0133] Attribute information management

[0134] The user selects the "Group Management" option from the friend list menu in the LINE Messenger app.

[0135] Displays all groups and friend information to which the device is registered.

[0136] Users can edit the displayed groups and attribute information, add new categories, or delete unnecessary groups.

[0137] The terminal will update the database with these changes.

[0138] Input: User operation (group edit, add, delete)

[0139] Output: Updated database

[0140] This system allows users to efficiently manage and categorize their friends' information using only voice input. In addition, the automatic linking function with phone book data significantly reduces the effort required to manually update information.

[0141] (Application example 1)

[0142] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0143] Traditionally, customer management in brick-and-mortar stores has been largely manual, making the classification and management of customer information particularly cumbersome in stores with large customer databases. Furthermore, classification based on voice input of customer information has not been realized, making efficient operation difficult. Furthermore, there was a need for a system that would reduce the workload of store employees in managing customer information and quickly and accurately reflect customer information.

[0144] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0145] In this invention, the server includes a voice input means for inputting friend attribute information by voice, a voice recognition means for converting the voice-input attribute information into text data, a means for registering friends using QR codes or ID search, a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to those groups, and a customer management means for store employees to manage customer information. This allows store employees to efficiently manage and classify customer information using QR codes or voice input.

[0146] "Voice input means" refers to a device or system that allows a user to input data through voice.

[0147] "Speech recognition means" refers to software or hardware for converting input voice data into text data.

[0148] "Means for registering friends using QR codes or ID search" refers to the functions or processes for registering friend information using QR code scanning or ID search.

[0149] The "database management means for analyzing text data, identifying corresponding groups, and assigning friends to those groups" is a system or program that analyzes text data, identifies appropriate groups, and assigns friend information to those groups.

[0150] "Customer management means" refers to the tools and systems that store employees use to efficiently manage customer information.

[0151] The "means for scanning communication data and comparing it with the communication application's database" is a function that reads existing contact data and compares it with the communication application's database to search for matching information.

[0152] "Data linkage means" refers to methods and technologies for exchanging information between different databases and integrating matching information.

[0153] The "means for editing, adding, and deleting attribute information" refers to an interface and method that allows a user to freely edit, add, or delete attribute information in a database.

[0154] A "means for categorizing customer information via voice input" is a system or function that uses voice input to automatically categorize customer information into specific categories.

[0155] The present invention relates to a system for improving the efficiency of customer management in a store and for classifying customer information through voice input.

[0156] System Overview

[0157] The system includes the following means:

[0158] 1. Voice input means: The server uses a voice input device that allows users to input customer information through voice, including the microphone of a smartphone.

[0159] 2. Speech recognition means: The server uses the speech_recognition library to convert the input voice data into text data.

[0160] 3. Registering friends by QR code or ID search: The server registers customer information by scanning QR code or ID search. This method includes qrcode library.

[0161] 4. Database management means: The server has a database for analyzing the converted text data, identifying the corresponding groups, and assigning customers to those groups. This database uses SQLite.

[0162] 5. Customer Management Tool: A management tool used by store employees to manage customer information. This tool supports editing, adding, and grouping customer information.

[0163] Program processing explanation

[0164] Processing of voice input means

[0165] When a user enters customer information by voice, the terminal captures the voice data through the microphone. This data is converted into text data by a voice recognition means. The specific hardware used is the microphone of a smartphone.

[0166] Processing of speech recognition means

[0167] The device uses the speech_recognition library to convert the recorded voice data into text data. This data conversion is done in real time, so the user can move on to the next operation without any delay.

[0168] Registration via QR code or ID search

[0169] When a user registers a new customer, they scan a QR code using their smartphone camera or enter their ID, which is sent to the server and stored in a SQLite database.

[0170] Database Management

[0171] The information obtained from the text data is analyzed on the server and classified into corresponding groups. Natural language processing technology is likely to be used for this process. SQLite is used for database management, allowing for efficient data retrieval and updating.

[0172] Examples of customer management

[0173] For example, imagine an employee registering a new customer with a QR code and typing "Regular Customer" by voice. This speech is converted to text by the speech_recognition library, and the customer's information is assigned to a group called "Regular Customer." This can all be done on a smartphone.

[0174] Prompt Sentence Examples

[0175] When using a generative AI model, you can use the following prompts:

[0176] "Create an application that categorizes customers into 'loyal customers' groups based on their voice."

[0177] This system allows store employees to efficiently manage and classify customer information, and makes it possible to quickly search and refer to necessary information, especially when there are a large number of customers. In addition, the combined use of voice input and QR codes makes customer information management even more efficient.

[0178] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0179] Step 1:

[0180] Users scan a QR code using their smartphone camera to register a new customer.

[0181] Input: QR code image

[0182] Data processing: Using the QR code library, the QR code is decoded, and basic customer information (name, contact details, etc.) is obtained.

[0183] Output: Customer basic information (name, contact details)

[0184] Step 2:

[0185] The basic customer information acquired by the terminal is sent to the server and stored in an SQLite database.

[0186] Input: Customer basic information

[0187] Data processing: Use SQL queries to register customer information in the database.

[0188] Output: New customer information registered in the database

[0189] Step 3:

[0190] Users enter customer attribute information through voice input, and the voice is recorded using the smartphone's microphone.

[0191] Input: Audio data

[0192] Data processing: Capture audio data through the microphone.

[0193] Output: Recorded audio data

[0194] Step 4:

[0195] The server converts the audio data into text data using the speech_recognition library.

[0196] Input: Audio data

[0197] Data processing: Voice data is converted into text data using a voice recognition algorithm.

[0198] Output: Text data (e.g., "Regular Customer")

[0199] Step 5:

[0200] The device analyzes the converted text data and classifies it into appropriate customer groups.

[0201] Input: Text data

[0202] Data processing: Using natural language processing technology, the text data is analyzed and group names are identified.

[0203] Output: Group information (e.g., "Regular Customers")

[0204] Step 6:

[0205] The server updates the customer and group information into a database and assigns customers to specific groups.

[0206] Input: Customer information, group information

[0207] Data manipulation: Using SQL queries to add new group information and update customer information in the database.

[0208] Output: Updated database

[0209] Step 7:

[0210] The user checks the group information reflected in the database and edits, adds, or deletes as necessary.

[0211] Input: User operation (edit, add, delete)

[0212] Data processing: The operations performed by the user are reflected in the database through the user interface.

[0213] Output: Updated customer and group information

[0214] This series of processes enables store employees to efficiently manage and classify customer information using QR codes and voice input.

[0215] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0216] This invention realizes more advanced friend management by combining a system that manages and classifies friend information using voice input means and registers friends using QR codes and ID searches with an emotion engine that recognizes the user's emotions. This system provides functions that allow users to register friends, group them using voice input, and acquire emotion information and reflect it in a database.

[0217] Program processing

[0218] Add friends and manage their basic information

[0219] 1. The user opens the LINE Messenger application and scans the QR code on the Add Friends screen or searches for an ID. The device adds the friend information obtained from the QR code or ID to the LINE Messenger database.

[0220] Audio grouping and emotion recognition

[0221] 2. The user taps the microphone icon on the contact details screen to switch to voice input mode, and the device enters a state where it is waiting for the user's voice input.

[0222] 3. The user inputs the group attribute by voice (e.g., "business partner"). The terminal receives the voice input and converts the voice into text data using a voice recognition means.

[0223] 4. The device analyzes the converted text data and compares it with the LINE Messenger database. If the group does not exist, the device creates a new group. If it does, the device assigns friends to the group.

[0224] 5. The device analyzes the voice content from the voice input means using an emotion engine to obtain the user's emotional information. For example, if a user speaks emotionally about an "important customer" when assigning a friend to the "business partner" group, that emotional information is obtained.

[0225] 6. The device records the acquired emotional information in a database and stores it in association with the friend's attribute information. This adds detailed information such as "business partner" and "important."

[0226] Integration with existing phone book data

[0227] 7. The device periodically scans the phone book data on the smartphone, for example, once a day.

[0228] 8. The device compares the friend information registered in the LINE Messenger database with the data in the phone book. The server verifies the match and obtains additional information such as occupation and company name from the phone book.

[0229] 9. The server integrates the additional information it has acquired into the friend information on LINE Messenger. The device updates the database and completes the friend's attribute information.

[0230] Managing and editing attribute information

[0231] 10. The user selects the "Group Management" option from the friend list menu in the LINE Messenger application. This displays all groups and friend information that the device is currently registered in.

[0232] 11. Users can tap on the displayed group to edit their friends' attribute information. Users can add new attribute categories or delete unnecessary groups.

[0233] 12. The device saves the user's edits to the database and reflects the latest status of group and attribute information, allowing the user to always manage the latest friend information.

[0234] Specific examples

[0235] Friend voice classification and emotion recognition

[0236] For example, after registering a new friend using a QR code, a user can enter "business partner" by voice. If the user speaks with emotion, the emotion engine analyzes the emotion. The device converts the voice into text data and assigns the friend to a group called "business partners." Furthermore, emotional information such as "important customer" is also acquired and recorded in the database. Next, the device scans the phone book to obtain the friend's detailed information (e.g., company name and occupation) and adds it to the friend information.

[0237] This invention allows users to efficiently manage and classify information about their friends through voice input and emotion recognition. This makes it possible to quickly search and refer to the information needed, even when there are a large number of friends. Furthermore, the automatic linking function of the emotion engine significantly reduces the effort required for manually updating information.

[0238] The processing flow will be explained below.

[0239] Step 1:

[0240] The user opens the LINE Messenger app and scans the QR code on the Add Friends screen or searches for the ID. The device adds the friend information obtained from the QR code or ID to the LINE Messenger database.

[0241] Step 2:

[0242] After the registered friend information is displayed, the user taps the microphone icon on the contact details screen to switch to voice input mode, and the device enters a state waiting for the user's voice input.

[0243] Step 3:

[0244] The user inputs the group attribute by voice (e.g., "business partner"). The terminal receives the voice input and converts the voice into text data using a voice recognition means.

[0245] Step 4:

[0246] The device inputs the text data obtained by voice recognition into an emotion engine, which analyzes the user's emotional information (e.g., the emphasis on "important business partner").

[0247] Step 5:

[0248] The device analyzes the converted text data and the analyzed emotional information and compares it with the LINE Messenger database. If no existing group exists, the device creates a new group. If an existing group exists, the device assigns friends to that group.

[0249] Step 6:

[0250] The device scans the existing contacts data in the user's smartphone. This scan operation is performed periodically (for example, once a day).

[0251] Step 7:

[0252] The device compares the friend information registered in the LINE Messenger database with the data in the phone book. The server verifies the match and obtains additional information such as occupation and company name from the phone book.

[0253] Step 8:

[0254] The server integrates the additional information it has acquired into LINE Messenger's friend information, and the device updates the database to complete the friend's attribute information.

[0255] Step 9:

[0256] When a user selects the "Group Management" option from the friend list menu in the LINE Messenger app, all groups and friend information currently registered on the device will be displayed.

[0257] Step 10:

[0258] Users can tap on a displayed group to edit their friends' attribute information. Users can add new attribute categories or delete unnecessary groups.

[0259] Step 11:

[0260] The device saves the user's edits to the database, reflecting the latest status of group and attribute information, allowing the user to always manage the latest friend information.

[0261] Specific examples

[0262] For example, a user registers a new friend using a QR code and then speaks to that friend, saying "business partner." If the user emphasizes "important business partner," the emotion engine analyzes that emphasis. The device converts the speech into text data, and based on the "important" information obtained from the emotion engine, creates a new group called "important business partner" and assigns the friend to it. The device then scans the phone book to obtain additional information about the friend (for example, the company name "XYZ Corporation" and the occupation "Sales Representative"), which is then integrated into the LINE Messenger database. This allows users to manage their friend information in detail.

[0263] Example 2

[0264] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0265] In today's world, many users need to manage and categorize their friend information. However, this management process is time-consuming, and adding emotional information is particularly difficult. Furthermore, there is a lack of efficient ways to integrate existing contact data with data from messaging applications. Therefore, there is a need for efficient and detailed management of friend information.

[0266] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0267] In this invention, the server includes a voice input means for acquiring friend attribute information input by voice, a voice recognition means for converting the voice-input attribute information into text data, a means for registering friends using identification information, a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to those groups, and an emotion engine means for analyzing the user's emotions and adding the acquired emotion information to the attribute information. This allows the user to efficiently register and classify friend information using voice, and further enables more detailed and accurate friend management by integrating the emotion information.

[0268] "Voice input means" refers to a device or system for receiving voice and analyzing the voice data.

[0269] "Speech recognition means" refers to the technology or algorithms used to convert voice data into text data.

[0270] "Identification information" refers to information that uniquely identifies a friend, such as a QR code or ID.

[0271] "Database management measures" refers to the systems and processes used to analyze text data and classify and assign friend information to appropriate groups.

[0272] The "emotion engine means" refers to an engine or software for analyzing the user's emotions from voice input and acquiring the emotion information.

[0273] "Data linkage means" refers to the means for collating and integrating data between different databases.

[0274] The present invention is a system that manages and classifies friend information by voice, registers friends using identification information, and analyzes the user's emotions and adds that information. This system includes a voice input means, a voice recognition means, a database management means, an emotion engine means, and a data linking means.

[0275] First, the user opens the appropriate application to register a friend. The user scans or enters identification information (e.g., QR code or ID) on the friend addition screen, and the device receives it and adds the friend information to the database.

[0276] The user taps the microphone icon on the contact details screen to switch to voice input mode. This puts the device into a voice input standby state, and the user voice-inputs the group attribute. For example, if the user voice-inputs "business partner," the device receives the voice and converts the voice into text data using the voice recognition means. The device then analyzes the converted text data, identifies the corresponding group using the database management means, and assigns friends to the group.

[0277] Furthermore, the terminal analyzes the voice content using the emotion engine means to acquire the user's emotion information. For example, if the user assigns a friend to a group called "business partners" and speaks with emotion such as "important customer," that emotion information is acquired. The acquired emotion information is added to the friend's attribute information by the database management means and recorded in the database.

[0278] Periodically, the device scans the smartphone's contact data and matches the existing contact data with the data in the messaging application. The server checks for matching friend information, extracts additional information from the contact data, such as occupation or company name, and integrates it into the friend information in the messaging application. The device then updates the database with this information.

[0279] Users can also edit their friends' attribute information by selecting the "Group Management" option from the friend list menu in the messaging application and tapping on a displayed group. Users can add new attribute categories or delete unnecessary groups. The device saves the user's edits to the database, ensuring that the latest friend information is always available.

[0280] Specific examples

[0281] For example, after registering a new friend using a QR code, a user can enter "business partner" by voice. If the user speaks with emotion, the emotion engine analyzes the emotion. The device converts the voice into text data and assigns the friend to a group called "business partners." Furthermore, the emotional information "important customer" is also acquired and recorded in the database. Next, the device scans the contact data to obtain the friend's detailed information (e.g., company name and occupation) and can add the friend to the friend information.

[0282] Prompt Sentence Examples

[0283] "Please register new friends using QR codes and group them using voice input. Also, please identify emotions and add appropriate emotional information when registering."

[0284] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0285] Step 1:

[0286] The user launches a messaging application.

[0287] (Input) User action (application launch)

[0288] (Output) The application starts and the home screen is displayed.

[0289] Specifically, a user taps an icon on their smartphone to launch the application, and the messaging application home screen appears on the screen.

[0290] Step 2:

[0291] The user scans the QR code on the Add Friend screen and enters their identification information.

[0292] (Input) Scan QR code or enter ID

[0293] (Output) Identification information is obtained.

[0294] Specifically, the user taps the camera icon on the Add Friend screen, scans the QR code, or enters the ID into the input field. The device then acquires this identification information.

[0295] Step 3:

[0296] The terminal adds the acquired identification information to a database.

[0297] (Input) Identification information (QR code or ID)

[0298] (Output) The friend information is added to the database.

[0299] Specifically, the device registers the LINE ID, name, and profile information obtained from the QR code in a database.

[0300] Step 4:

[0301] The user taps the microphone icon on the contact details screen to switch to voice input mode.

[0302] (Input) User action (tapping the microphone icon)

[0303] (Output) Enters voice input standby mode.

[0304] Specifically, the user taps the microphone icon on the contact details screen, causing the device to enter a voice input standby state.

[0305] Step 5:

[0306] The user speaks a group attribute (for example, "account").

[0307] (Input) User's voice (group attributes)

[0308] (Output) Audio data is obtained.

[0309] As a specific operation, the user speaks "business partner" and the terminal receives the voice data.

[0310] Step 6:

[0311] The terminal receives voice input and converts the voice into text data using a voice recognition means.

[0312] (Input) Audio data

[0313] (Output) Text data (e.g., "Business Partner")

[0314] Specifically, the device uses a voice recognition engine to analyze the voice and convert it into text data.

[0315] Step 7:

[0316] The terminal analyzes the converted text data, identifies the corresponding group by means of a database management means, and assigns friends to the group.

[0317] (Input) Text data (e.g., "Business Partner")

[0318] (Output) Friend information assigned to the group

[0319] Specifically, the device matches the text "business partner" with existing groups in the database, and creates new groups as needed, assigning friend information.

[0320] Step 8:

[0321] The device analyzes the voice content using an emotion engine and obtains the user's emotional information.

[0322] (Input) Audio data

[0323] (Output) Emotional information

[0324] Specifically, the device passes the voice recognition results to the emotion engine, which analyzes the emotions and obtains the emotion information.

[0325] Step 9:

[0326] The terminal records the acquired emotion information in a database.

[0327] (Input) Emotion information

[0328] (Output) Emotion information recorded in the database

[0329] Specifically, the device associates the acquired emotion information with friend information and stores it in a database.

[0330] Step 10:

[0331] The device periodically scans your smartphone for contact data.

[0332] (Input) Contact Data

[0333] (Output) Scan results

[0334] Specifically, the device will automatically scan your contact data every night.

[0335] Step 11:

[0336] The device checks the contact data against the messaging application's database.

[0337] (Input) Contact data, messaging application data

[0338] (Output) Matching result

[0339] Specifically, the device compares the contact data with the data in the messaging application to find matching information.

[0340] Step 12:

[0341] The server checks for a matching friend and retrieves additional information from the contact data, such as occupation and company name.

[0342] (Input) Matching results, contact information

[0343] (Output) Additional information (occupation, company name)

[0344] Specifically, the server checks for matching friend information and retrieves additional information.

[0345] Step 13:

[0346] The server integrates the acquired additional information into the friend information of the messaging application.

[0347] (Input) Additional information (occupation, company name)

[0348] (Output) Integrated friend information

[0349] Specifically, the server combines the acquired information with friend information in the messaging application.

[0350] Step 14:

[0351] The device updates the database.

[0352] (Input) Integrated friend information

[0353] (Output) Updated database

[0354] Specifically, the terminal reflects information from the server and keeps the database up to date.

[0355] Step 15:

[0356] A user selects the "Manage Groups" option from the friends list menu of a messaging application.

[0357] (Input) User action (selection of "Group Management")

[0358] (Output) Group management screen

[0359] Specifically, the user opens the friend list menu and selects "Group Management."

[0360] Step 16:

[0361] View all groups and friend information that your device is currently registered to.

[0362] (Input) User action

[0363] (Output) Displayed group and friend information

[0364] Specifically, the device retrieves all group and friend information from the database and displays it on the screen.

[0365] Step 17:

[0366] The user taps on the displayed group and edits the attribute information of the friend.

[0367] (Input) User action (tap on group, edit)

[0368] (Output) Edited attribute information

[0369] Specifically, the user taps the group he or she wants to edit and edits the attribute information.

[0370] Step 18:

[0371] Users can add new attribute categories and delete unnecessary groups.

[0372] (Input) User action (add, delete)

[0373] (Output) Updated attribute categories

[0374] As a specific operation, the user adds new attributes and deletes unnecessary groups.

[0375] Step 19:

[0376] The device saves the user's edits in a database.

[0377] (Input) Edited attribute information

[0378] (Output) Saved database

[0379] Specifically, the terminal records the edited content of the user in a database and updates it to the latest version.

[0380] (Application example 2)

[0381] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0382] Modern industrial and production sites require smooth communication between workers and robots. In particular, it is important to improve production efficiency and worker satisfaction by providing appropriate support based on the worker's emotions and state. However, existing systems lack robots that can properly understand the worker's voice instructions and respond flexibly based on emotion recognition, so new technologies are needed to solve this problem.

[0383] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice input means for inputting friend attribute information by voice, a voice recognition means for converting the voice-input attribute information into text data, a means for registering friends using QR codes or ID search, a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to those groups, an emotion recognition means for analyzing the acquired voice content and acquiring user emotion information, and a database storage means for storing the acquired emotion information in association with the friend attribute information. This makes it possible to realize a factory robot that appropriately analyzes a worker's voice instructions and provides flexible and optimal support based on the emotion information.

[0384] "Friend attribute information" is information set by the user to classify friends, and indicates categories such as "business partner" and "customer."

[0385] The "voice input means" is a device or system that allows a user to input attribute information of a friend through voice.

[0386] "Speech recognition means" refers to a technology or system that converts speech input by a speech input means into text data.

[0387] "Methods for registering friends using QR codes or ID searches" refers to methods in which users scan QR codes or enter IDs to identify and register friends.

[0388] The "database management means" is a system that has the function of analyzing text data, identifying corresponding groups, and assigning friends to those groups.

[0389] "Emotion recognition means" refers to a technique or device for analyzing the content of a voice and acquiring information about the user's emotions.

[0390] The "database storage means" is a system that has the function of storing the obtained emotional information in association with the attribute information of the friends.

[0391] "Means for scanning telephone directory data and comparing it with a database" refers to a technique for periodically checking existing telephone directory data and comparing it with information in a database.

[0392] The "data linking means" is a system that has the function of obtaining matching friend information from the phone book and adding it to the friend information in the database.

[0393] The "classification means" is a technology that uses a database management means to sort the attribute information of friends into new groups.

[0394] The "means for setting the support level" is a system that has the function of analyzing the voice instructions of the worker and acquiring emotional information to provide appropriate support according to the worker's condition.

[0395] The present invention provides a system for efficiently managing and classifying friend information and work support information using voice input and emotion recognition of a worker. Specific embodiments for carrying out the present invention will be described below.

[0396] Hardware and software used

[0397] This system uses the following hardware and software:

[0398] Audio input microphone (e.g., a general audio input device)

[0399] Industrial robots (e.g. Universal Robots)

[0400] Speech recognition API (e.g., Google Cloud Speech-to-Text)

[0401] Emotion recognition engine (e.g., Affectiva's Emotion Recognition SDK)

[0402] Data management software (e.g., SQLite database)

[0403] System configuration

[0404] The system consists of the following main components:

[0405] 1. Voice input means: A microphone device that allows workers to input their friends' attribute information and work instructions through voice.

[0406] 2. Speech recognition means: A speech recognition API is used to convert voice-input information into text data.

[0407] 3. How to register friends: Register friends using QR codes or ID searches.

[0408] 4. Database management means: Analyze the text data, identify the corresponding groups, and assign friends to those groups.

[0409] 5. Emotion recognition means: Analyzes the user's emotions from the voice content and obtains emotional information.

[0410] 6. Database storage means: The acquired emotional information is stored in association with the friend's attribute information.

[0411] 7. Data integration method: Scan existing phone book data and add it to the database.

[0412] 8. Support level setting means: Set an appropriate support level based on voice instructions and emotional information.

[0413] System Operation

[0414] First, the worker inputs the friend's attribute information and work instructions using the voice input means. This voice data is converted into text data by the voice recognition means. This text data is analyzed by the database management means and classified into appropriate groups.

[0415] Next, the emotion recognition means analyzes the voice data to obtain the user's emotion information, which is then stored in the database by the database storage means in association with the friend's attribute information.

[0416] Furthermore, the data linking means periodically scans the telephone book data, and the matching friend information is acquired from the telephone book and added to the database.

[0417] Finally, the support level setting means analyzes the voice instructions and emotional information of the worker and sets an appropriate support level, allowing the industrial robot to provide appropriate support according to the worker's workload.

[0418] Specific examples

[0419] For example, a worker may give a voice command such as "I'm tired today, so please increase my support." This voice is first converted into text data by a voice recognition API, and then emotional information indicating "fatigue" is obtained by an emotion recognition engine. Based on this information, the database management means increases the level of support for the worker, and the industrial robot automatically strengthens its support.

[0420] Prompt Sentence Examples

[0421] "Please create a program that analyzes the worker's voice instructions, obtains emotional information using an emotion recognition engine, and sets an appropriate assistance level. Please also provide specific details about the voice recognition API and emotion recognition engine you will use, as well as how you will save the data."

[0422] As described above, according to the embodiment of the invention, it is possible to efficiently manage and automate friend information and task support information based on the user's voice input and emotion recognition.

[0423] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0424] Step 1:

[0425] Audio Input:

[0426] The user inputs voice into the microphone device. The voice input includes attribute information of friends and work instructions. This input data is captured by the microphone device as an analog voice signal.

[0427] Step 2:

[0428] Voice Recognition:

[0429] The device sends voice input data to a speech recognition API (e.g., Google Cloud Speech-to-Text), which converts the analog voice signal into text data. In this case, the input is voice data and the output is text data.

[0430] Step 3:

[0431] Text data analysis:

[0432] The device analyzes the text data received from the voice recognition API using a database management means. The input is text data, and the data content is analyzed to identify related groups and assign the corresponding friend information to those groups. The output is the friend information categorized into groups.

[0433] Step 4:

[0434] Add as friend:

[0435] When a user registers a friend using a QR code or ID search, the device accepts the QR code scan or ID input and retrieves the corresponding friend information. The input is the QR code information or ID information, and the output is the friend information.

[0436] Step 5:

[0437] Emotion recognition:

[0438] The device uses an emotion recognition engine (e.g., Affectiva SDK) to recognize the user's emotions from text data. The emotion recognition engine receives text data as input and outputs emotional information. In this case, the input is text data and the output is emotional information.

[0439] Step 6:

[0440] Emotional information storage:

[0441] The emotion information acquired by the device from the emotion recognition engine is stored in a database by a database storage means in association with the friend's attribute information. The input is the emotion information and the friend's attribute information, and the output is the updated database.

[0442] Step 7:

[0443] Scan and match phone book data:

[0444] The device periodically scans its existing contact list and matches it with the friend information in its database. The input is the contact list data, and the output is the matching friend information.

[0445] Step 8:

[0446] Data integration:

[0447] The server adds the matching friend information retrieved from the phone book to the friend information in the database. The input is the phone book data and the database information, and the output is the updated friend information.

[0448] Step 9:

[0449] Support Level Settings:

[0450] The server uses the results of analysis by an emotion recognition engine based on the voice instructions and emotional information to set an appropriate support level. The input is the voice instructions and emotional information, and the output is the set support level.

[0451] Step 10:

[0452] Industrial robot control:

[0453] The terminal issues instructions to the industrial robot to assist in the work according to the set assistance level. This operation is performed using a robot motion control program. The input is assistance level information, and the output is robot motion control.

[0454] The system of this embodiment enables efficient management and automation of friend information and work support information based on the user's voice input and emotion recognition.

[0455] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0456] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0457] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0458] [Second embodiment]

[0459] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0460] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0461] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0462] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0463] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0464] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0465] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0466] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0467] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0468] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0469] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0470] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0471] This invention relates to a system for efficiently managing and classifying friend information in the LINE Messenger application. Specifically, it provides a function that allows users to register friends using QR codes or ID searches, and then classify the friends into appropriate groups using voice input. It also includes a function that links existing phone book data with LINE Messenger data and automatically updates friend attribute information.

[0472] Program processing

[0473] 1. Add friends

[0474] When a user opens the LINE Messenger application and scans a QR code or searches for an ID on the Add Friends screen, the device uses this information to store the friend's basic information (such as name, ID, and profile picture) in the LINE Messenger database.

[0475] 2. Audio grouping

[0476] After registering a friend, the user taps the microphone icon on the contact details screen to switch to voice input mode. When the user enters the friend's attribute information by voice, the device converts the voice into text data. The device then analyzes the converted text data to check whether a corresponding group exists. If no existing group exists, the device creates a new group, assigns the friend to it, and updates the database.

[0477] 3. Phonebook data integration

[0478] The device periodically scans the smartphone's contacts and compares them with the existing LINE Messenger database. If a match is found, additional information such as occupation and company name is retrieved from the contacts and added to the LINE Messenger contact information.

[0479] 4. Attribute Information Management

[0480] Users select the "Group Management" option from the friend list menu in the LINE Messenger app. The device displays all currently registered groups and friend information. Users can edit the displayed groups and attribute information, add new categories, or delete unnecessary groups. The device reflects the user's actions and updates the database accordingly.

[0481] Example: Classifying your friends' voices

[0482] For example, suppose a user registers a new friend using a QR code and then speaks "business partner." The device converts the speech into text data and assigns the friend to a group called "business partners." The device then scans the phone book to obtain the friend's detailed information (e.g., company name and occupation) and adds it to the LINE Messenger database.

[0483] This invention allows users to efficiently manage and classify information about their friends using only voice input, making it possible to quickly search and refer to the information they need, even when they have a large number of friends. Furthermore, the automatic linking function with phone book data significantly reduces the effort required for manually updating information.

[0484] The processing flow will be explained below.

[0485] Step 1:

[0486] The user opens the LINE Messenger application and scans the QR code on the Add Friends screen or searches for the ID. The device adds the friend information obtained from the QR code or ID to the LINE Messenger database.

[0487] Step 2:

[0488] After the registered friend information is displayed, the user taps the microphone icon on the contact details screen to switch to voice input mode, and the device enters a state waiting for the user's voice input.

[0489] Step 3:

[0490] The user inputs the group attribute by voice (e.g., "business partner"). The terminal receives the voice input and converts the voice into text data using a voice recognition means.

[0491] Step 4:

[0492] The device analyzes the converted text data and compares it with the LINE Messenger database. If there is no existing group, the device creates a new group. If there is an existing group, the device assigns friends to it.

[0493] Step 5:

[0494] The device scans the existing contacts data stored in the user's smartphone. The scan is performed periodically, for example, once a day.

[0495] Step 6:

[0496] The device compares the friend information registered in the LINE Messenger database with the data in the phone book. The server verifies the match and obtains additional information such as occupation and company name from the phone book.

[0497] Step 7:

[0498] The server integrates the additional information it has acquired into LINE Messenger's friend information, and the device updates the database to complete the friend's attribute information.

[0499] Step 8:

[0500] When a user selects the "Group Management" option from the friend list menu in the LINE Messenger application, all groups and friend information currently registered on the device is displayed.

[0501] Step 9:

[0502] Users can tap on a displayed group to edit their friends' attribute information. Users can add new attribute categories or delete unnecessary groups.

[0503] Step 10:

[0504] The device saves the user's edits to the database and reflects the latest status of group and attribute information, allowing the user to always manage the latest friend information.

[0505] Example 1

[0506] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0507] In conventional friend management systems, friend attribute information and group management are done manually, making management cumbersome, especially for users with many friends. Furthermore, since there is no function to update information in conjunction with phone book data, friend information is often out of date. For this reason, there is a demand for a system that can efficiently manage and classify friend information and automatically update the latest information.

[0508] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0509] In this invention, the server includes a voice input means for inputting friend attribute information by voice, a voice recognition means for converting the voice-input attribute information into text data, a means for registering friends using identification data, a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to those groups, and a data linking means for periodically scanning existing contact data, collating the information, acquiring additional information, and updating the database. This allows for efficient friend information management through voice input, and automatic updating of the latest information through linking with phone book data.

[0510] "Friend attribute information" is information relating to specific characteristics or categories of registered friends.

[0511] "Voice input means" refers to a device or function that allows a user to input information into a system using voice.

[0512] "Speech recognition means" refers to technology or devices for converting input speech into text data.

[0513] "Identification data" is data that uniquely identifies a friend, such as a QR code or ID.

[0514] "Database management means" refers to a system or device for efficiently managing, classifying, and storing friend information and related data.

[0515] "Data integration means" refers to the technology or function that compares existing contact data or other data sources with data within the system to obtain and supplement the necessary information.

[0516] A "group" is a category for managing friends who share common attribute information.

[0517] "Contact data" refers to data stored in a telephone directory or the like that includes information such as names, numbers, and company names.

[0518] This invention is a system for efficiently managing and classifying attribute information of friends. This system utilizes the LINE Messenger application, uses voice input to appropriately classify friends, and automatically links with phone book data to keep friend information up to date.

[0519] Hardware and software used

[0520] 1. LINE Messenger application: Used to register and manage friend information.

[0521] 2. Smartphone: The device on which the user runs the LINE Messenger app.

[0522] 3. Voice input engine (e.g. Google Speech-to-Text API): Used to convert voice into text data.

[0523] 4. Database system (e.g. SQLite): Used to store and update friend and group information.

[0524] Specific operation of the system

[0525] Add a friend

[0526] When a user opens the LINE Messenger application and scans a QR code or searches for an ID on the Add Friends screen, the device uses this information to store the friend's basic information (such as name, ID, and profile picture) in the LINE Messenger database.

[0527] Audio grouping

[0528] After registering a friend, the user taps the microphone icon on the contact details screen to switch to voice input mode. When the user enters attribute information such as "business partner" by voice, the device converts the voice into text data. The device analyzes the converted text data, and if there is no existing group, it creates a new group and assigns the friend to that group. The database management means processes this and updates the database.

[0529] Phone book data integration

[0530] The device periodically scans the smartphone's contacts and compares them with the existing LINE Messenger database. If a match is found, additional information such as occupation and company name is retrieved from the contacts and added to the LINE Messenger contacts. This keeps the contacts up to date.

[0531] Attribute information management

[0532] The user selects the "Group Management" option from the friend list menu in the LINE Messenger app. The device displays all registered groups and friend information. The user can edit the displayed groups and attribute information, add new categories, or delete unnecessary groups. The device reflects these operations and updates the database.

[0533] Examples of specific examples and prompts

[0534] For example, if a user registers a new friend using a QR code and speaks "business partner," the device converts the voice to text data and assigns the friend to a group called "business partners." The device then scans the phone book to obtain the friend's detailed information (e.g., company name and occupation) and adds it to the LINE Messenger database. This invention allows users to efficiently manage and categorize friend information using only voice input.

[0535] Example prompt sentence:

[0536] "How do I add new friends to the "Business Partners" group in the LINE Messenger app?"

[0537] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0538] Step 1:

[0539] Start friend registration

[0540] The user opens the LINE Messenger application and scans the QR code on the Add Friends screen or searches for the ID.

[0541] The device will retrieve the QR code information or ID search results based on this input and save the friend's basic information (name, ID, profile picture, etc.) in the LINE Messenger database.

[0542] Step 2:

[0543] Activating voice input mode

[0544] After users register friends using QR codes or ID searches, they can tap the microphone icon on the contact details screen to switch to voice input mode.

[0545] The device launches a voice input engine (e.g., Google Speech-to-Text API) and waits for the user's voice input.

[0546] Step 3:

[0547] Voice input of attribute information

[0548] The user inputs attribute information such as "business partner" by voice.

[0549] The device sends this voice data to the Google Speech-to-Text API and obtains the text data "Customer."

[0550] Input: Audio data

[0551] Output: Text data ("Customer")

[0552] Step 4:

[0553] Attribute information analysis and grouping

[0554] The text data "Customer" acquired by the device is analyzed using a natural language processing engine (e.g., NLTK) to check whether a corresponding group exists.

[0555] If the terminal does not have a group called "Customer", it will create a new group called "Customer".

[0556] The device assigns friends to the "Business Partners" group and updates the LINE Messenger database.

[0557] Input: Text data ("Customer")

[0558] Output: Updated database

[0559] Step 5:

[0560] Phone book data integration

[0561] The device periodically scans the phone book data on your smartphone.

[0562] The device compares the LINE Messenger database with the phone book data, and if a matching friend is found, it retrieves additional information from the phone book, such as occupation and company name.

[0563] The device will add the additional information it has acquired to the LINE Messenger friend information and update the database.

[0564] Input: Phonebook data

[0565] Output: Updated friend information

[0566] Step 6:

[0567] Attribute information management

[0568] The user selects the "Group Management" option from the friend list menu in the LINE Messenger app.

[0569] Displays all groups and friend information to which the device is registered.

[0570] Users can edit the displayed groups and attribute information, add new categories, or delete unnecessary groups.

[0571] The terminal will update the database with these changes.

[0572] Input: User operation (group edit, add, delete)

[0573] Output: Updated database

[0574] This system allows users to efficiently manage and categorize their friends' information using only voice input. In addition, the automatic linking function with phone book data significantly reduces the effort required to manually update information.

[0575] (Application example 1)

[0576] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0577] Traditionally, customer management in brick-and-mortar stores has been largely manual, making the classification and management of customer information particularly cumbersome in stores with large customer databases. Furthermore, classification based on voice input of customer information has not been realized, making efficient operation difficult. Furthermore, there was a need for a system that would reduce the workload of store employees in managing customer information and quickly and accurately reflect customer information.

[0578] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0579] In this invention, the server includes a voice input means for inputting friend attribute information by voice, a voice recognition means for converting the voice-input attribute information into text data, a means for registering friends using QR codes or ID search, a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to those groups, and a customer management means for store employees to manage customer information. This allows store employees to efficiently manage and classify customer information using QR codes or voice input.

[0580] "Voice input means" refers to a device or system that allows a user to input data through voice.

[0581] "Speech recognition means" refers to software or hardware for converting input voice data into text data.

[0582] "Means for registering friends using QR codes or ID search" refers to the functions or processes for registering friend information using QR code scanning or ID search.

[0583] The "database management means for analyzing text data, identifying corresponding groups, and assigning friends to those groups" is a system or program that analyzes text data, identifies appropriate groups, and assigns friend information to those groups.

[0584] "Customer management means" refers to the tools and systems that store employees use to efficiently manage customer information.

[0585] The "means for scanning communication data and comparing it with the communication application's database" is a function that reads existing contact data and compares it with the communication application's database to search for matching information.

[0586] "Data linkage means" refers to methods and technologies for exchanging information between different databases and integrating matching information.

[0587] The "means for editing, adding, and deleting attribute information" refers to an interface and method that allows a user to freely edit, add, or delete attribute information in a database.

[0588] A "means for categorizing customer information via voice input" is a system or function that uses voice input to automatically categorize customer information into specific categories.

[0589] The present invention relates to a system for improving the efficiency of customer management in a store and for classifying customer information through voice input.

[0590] System Overview

[0591] The system includes the following means:

[0592] 1. Voice input means: The server uses a voice input device that allows users to input customer information through voice, including the microphone of a smartphone.

[0593] 2. Speech recognition means: The server uses the speech_recognition library to convert the input voice data into text data.

[0594] 3. Registering friends by QR code or ID search: The server registers customer information by scanning QR code or ID search. This method includes qrcode library.

[0595] 4. Database management means: The server has a database for analyzing the converted text data, identifying the corresponding groups, and assigning customers to those groups. This database uses SQLite.

[0596] 5. Customer Management Tool: A management tool used by store employees to manage customer information. This tool supports editing, adding, and grouping customer information.

[0597] Program processing explanation

[0598] Processing of voice input means

[0599] When a user enters customer information by voice, the terminal captures the voice data through the microphone. This data is converted into text data by a voice recognition means. The specific hardware used is the microphone of a smartphone.

[0600] Processing of speech recognition means

[0601] The device uses the speech_recognition library to convert the recorded voice data into text data. This data conversion is done in real time, so the user can move on to the next operation without any delay.

[0602] Registration via QR code or ID search

[0603] When a user registers a new customer, they scan a QR code using their smartphone camera or enter their ID, which is sent to the server and stored in a SQLite database.

[0604] Database Management

[0605] The information obtained from the text data is analyzed on the server and classified into corresponding groups. Natural language processing technology is likely to be used for this process. SQLite is used for database management, allowing for efficient data retrieval and updating.

[0606] Examples of customer management

[0607] For example, imagine an employee registering a new customer with a QR code and typing "Regular Customer" by voice. This speech is converted to text by the speech_recognition library, and the customer's information is assigned to a group called "Regular Customer." This can all be done on a smartphone.

[0608] Prompt Sentence Examples

[0609] When using a generative AI model, you can use the following prompts:

[0610] "Create an application that categorizes customers into 'loyal customers' groups based on their voice."

[0611] This system allows store employees to efficiently manage and classify customer information, and makes it possible to quickly search and refer to necessary information, especially when there are a large number of customers. In addition, the combined use of voice input and QR codes makes customer information management even more efficient.

[0612] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0613] Step 1:

[0614] Users scan a QR code using their smartphone camera to register a new customer.

[0615] Input: QR code image

[0616] Data processing: Using the QR code library, the QR code is decoded, and basic customer information (name, contact details, etc.) is obtained.

[0617] Output: Customer basic information (name, contact details)

[0618] Step 2:

[0619] The basic customer information acquired by the terminal is sent to the server and stored in an SQLite database.

[0620] Input: Customer basic information

[0621] Data processing: Use SQL queries to register customer information in the database.

[0622] Output: New customer information registered in the database

[0623] Step 3:

[0624] Users enter customer attribute information through voice input, and the voice is recorded using the smartphone's microphone.

[0625] Input: Audio data

[0626] Data processing: Capture audio data through the microphone.

[0627] Output: Recorded audio data

[0628] Step 4:

[0629] The server converts the audio data into text data using the speech_recognition library.

[0630] Input: Audio data

[0631] Data processing: Voice data is converted into text data using a voice recognition algorithm.

[0632] Output: Text data (e.g., "Regular Customer")

[0633] Step 5:

[0634] The device analyzes the converted text data and classifies it into appropriate customer groups.

[0635] Input: Text data

[0636] Data processing: Using natural language processing technology, the text data is analyzed and group names are identified.

[0637] Output: Group information (e.g., "Regular Customers")

[0638] Step 6:

[0639] The server updates the customer and group information into a database and assigns customers to specific groups.

[0640] Input: Customer information, group information

[0641] Data manipulation: Using SQL queries to add new group information and update customer information in the database.

[0642] Output: Updated database

[0643] Step 7:

[0644] The user checks the group information reflected in the database and edits, adds, or deletes as necessary.

[0645] Input: User operation (edit, add, delete)

[0646] Data processing: The operations performed by the user are reflected in the database through the user interface.

[0647] Output: Updated customer and group information

[0648] This series of processes enables store employees to efficiently manage and classify customer information using QR codes and voice input.

[0649] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0650] This invention realizes more advanced friend management by combining a system that manages and classifies friend information using voice input means and registers friends using QR codes and ID searches with an emotion engine that recognizes the user's emotions. This system provides functions that allow users to register friends, group them using voice input, and acquire emotion information and reflect it in a database.

[0651] Program processing

[0652] Add friends and manage their basic information

[0653] 1. The user opens the LINE Messenger application and scans the QR code on the Add Friends screen or searches for an ID. The device adds the friend information obtained from the QR code or ID to the LINE Messenger database.

[0654] Audio grouping and emotion recognition

[0655] 2. The user taps the microphone icon on the contact details screen to switch to voice input mode, and the device enters a state where it is waiting for the user's voice input.

[0656] 3. The user inputs the group attribute by voice (e.g., "business partner"). The terminal receives the voice input and converts the voice into text data using a voice recognition means.

[0657] 4. The device analyzes the converted text data and compares it with the LINE Messenger database. If the group does not exist, the device creates a new group. If it does, the device assigns friends to the group.

[0658] 5. The device analyzes the voice content from the voice input means using an emotion engine to obtain the user's emotional information. For example, if a user speaks emotionally about an "important customer" when assigning a friend to the "business partner" group, that emotional information is obtained.

[0659] 6. The device records the acquired emotional information in a database and stores it in association with the friend's attribute information. This adds detailed information such as "business partner" and "important."

[0660] Integration with existing phone book data

[0661] 7. The device periodically scans the phone book data on the smartphone, for example, once a day.

[0662] 8. The device compares the friend information registered in the LINE Messenger database with the data in the phone book. The server verifies the match and obtains additional information such as occupation and company name from the phone book.

[0663] 9. The server integrates the additional information it has acquired into the friend information on LINE Messenger. The device updates the database and completes the friend's attribute information.

[0664] Managing and editing attribute information

[0665] 10. The user selects the "Group Management" option from the friend list menu in the LINE Messenger application. This displays all groups and friend information that the device is currently registered in.

[0666] 11. Users can tap on the displayed group to edit their friends' attribute information. Users can add new attribute categories or delete unnecessary groups.

[0667] 12. The device saves the user's edits to the database and reflects the latest status of group and attribute information, allowing the user to always manage the latest friend information.

[0668] Specific examples

[0669] Friend voice classification and emotion recognition

[0670] For example, after registering a new friend using a QR code, a user can enter "business partner" by voice. If the user speaks with emotion, the emotion engine analyzes the emotion. The device converts the voice into text data and assigns the friend to a group called "business partners." Furthermore, emotional information such as "important customer" is also acquired and recorded in the database. Next, the device scans the phone book to obtain the friend's detailed information (e.g., company name and occupation) and adds it to the friend information.

[0671] This invention allows users to efficiently manage and classify information about their friends through voice input and emotion recognition. This makes it possible to quickly search and refer to the information needed, even when there are a large number of friends. Furthermore, the automatic linking function of the emotion engine significantly reduces the effort required for manually updating information.

[0672] The processing flow will be explained below.

[0673] Step 1:

[0674] The user opens the LINE Messenger app and scans the QR code on the Add Friends screen or searches for the ID. The device adds the friend information obtained from the QR code or ID to the LINE Messenger database.

[0675] Step 2:

[0676] After the registered friend information is displayed, the user taps the microphone icon on the contact details screen to switch to voice input mode, and the device enters a state waiting for the user's voice input.

[0677] Step 3:

[0678] The user inputs the group attribute by voice (e.g., "business partner"). The terminal receives the voice input and converts the voice into text data using a voice recognition means.

[0679] Step 4:

[0680] The device inputs the text data obtained by voice recognition into an emotion engine, which analyzes the user's emotional information (e.g., the emphasis on "important business partner").

[0681] Step 5:

[0682] The device analyzes the converted text data and the analyzed emotional information and compares it with the LINE Messenger database. If no existing group exists, the device creates a new group. If an existing group exists, the device assigns friends to that group.

[0683] Step 6:

[0684] The device scans the existing contacts data in the user's smartphone. This scan operation is performed periodically (for example, once a day).

[0685] Step 7:

[0686] The device compares the friend information registered in the LINE Messenger database with the data in the phone book. The server verifies the match and obtains additional information such as occupation and company name from the phone book.

[0687] Step 8:

[0688] The server integrates the additional information it has acquired into LINE Messenger's friend information, and the device updates the database to complete the friend's attribute information.

[0689] Step 9:

[0690] When a user selects the "Group Management" option from the friend list menu in the LINE Messenger app, all groups and friend information currently registered on the device will be displayed.

[0691] Step 10:

[0692] Users can tap on a displayed group to edit their friends' attribute information. Users can add new attribute categories or delete unnecessary groups.

[0693] Step 11:

[0694] The device saves the user's edits to the database, reflecting the latest status of group and attribute information, allowing the user to always manage the latest friend information.

[0695] Specific examples

[0696] For example, a user registers a new friend using a QR code and then speaks to that friend, saying "business partner." If the user emphasizes "important business partner," the emotion engine analyzes that emphasis. The device converts the speech into text data, and based on the "important" information obtained from the emotion engine, creates a new group called "important business partner" and assigns the friend to it. The device then scans the phone book to obtain additional information about the friend (for example, the company name "XYZ Corporation" and the occupation "Sales Representative"), which is then integrated into the LINE Messenger database. This allows users to manage their friend information in detail.

[0697] Example 2

[0698] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0699] In today's world, many users need to manage and categorize their friend information. However, this management process is time-consuming, and adding emotional information is particularly difficult. Furthermore, there is a lack of efficient ways to integrate existing contact data with data from messaging applications. Therefore, there is a need for efficient and detailed management of friend information.

[0700] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0701] In this invention, the server includes a voice input means for acquiring friend attribute information input by voice, a voice recognition means for converting the voice-input attribute information into text data, a means for registering friends using identification information, a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to those groups, and an emotion engine means for analyzing the user's emotions and adding the acquired emotion information to the attribute information. This allows the user to efficiently register and classify friend information using voice, and further enables more detailed and accurate friend management by integrating the emotion information.

[0702] "Voice input means" refers to a device or system for receiving voice and analyzing the voice data.

[0703] "Speech recognition means" refers to the technology or algorithms used to convert voice data into text data.

[0704] "Identification information" refers to information that uniquely identifies a friend, such as a QR code or ID.

[0705] "Database management measures" refers to the systems and processes used to analyze text data and classify and assign friend information to appropriate groups.

[0706] The "emotion engine means" refers to an engine or software for analyzing the user's emotions from voice input and acquiring the emotion information.

[0707] "Data linkage means" refers to the means for collating and integrating data between different databases.

[0708] The present invention is a system that manages and classifies friend information by voice, registers friends using identification information, and analyzes the user's emotions and adds that information. This system includes a voice input means, a voice recognition means, a database management means, an emotion engine means, and a data linking means.

[0709] First, the user opens the appropriate application to register a friend. The user scans or enters identification information (e.g., QR code or ID) on the friend addition screen, and the device receives it and adds the friend information to the database.

[0710] The user taps the microphone icon on the contact details screen to switch to voice input mode. This puts the device into a voice input standby state, and the user voice-inputs the group attribute. For example, if the user voice-inputs "business partner," the device receives the voice and converts the voice into text data using the voice recognition means. The device then analyzes the converted text data, identifies the corresponding group using the database management means, and assigns friends to the group.

[0711] Furthermore, the terminal analyzes the voice content using the emotion engine means to acquire the user's emotion information. For example, if the user assigns a friend to a group called "business partners" and speaks with emotion such as "important customer," that emotion information is acquired. The acquired emotion information is added to the friend's attribute information by the database management means and recorded in the database.

[0712] Periodically, the device scans the smartphone's contact data and matches the existing contact data with the data in the messaging application. The server checks for matching friend information, extracts additional information from the contact data, such as occupation or company name, and integrates it into the friend information in the messaging application. The device then updates the database with this information.

[0713] Users can also edit their friends' attribute information by selecting the "Group Management" option from the friend list menu in the messaging application and tapping on a displayed group. Users can add new attribute categories or delete unnecessary groups. The device saves the user's edits to the database, ensuring that the latest friend information is always available.

[0714] Specific examples

[0715] For example, after registering a new friend using a QR code, a user can enter "business partner" by voice. If the user speaks with emotion, the emotion engine analyzes the emotion. The device converts the voice into text data and assigns the friend to a group called "business partners." Furthermore, the emotional information "important customer" is also acquired and recorded in the database. Next, the device scans the contact data to obtain the friend's detailed information (e.g., company name and occupation) and can add the friend to the friend information.

[0716] Prompt Sentence Examples

[0717] "Please register new friends using QR codes and group them using voice input. Also, please identify emotions and add appropriate emotional information when registering."

[0718] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0719] Step 1:

[0720] The user launches a messaging application.

[0721] (Input) User action (application launch)

[0722] (Output) The application starts and the home screen is displayed.

[0723] Specifically, a user taps an icon on their smartphone to launch the application, and the messaging application home screen appears on the screen.

[0724] Step 2:

[0725] The user scans the QR code on the Add Friend screen and enters their identification information.

[0726] (Input) Scan QR code or enter ID

[0727] (Output) Identification information is obtained.

[0728] Specifically, the user taps the camera icon on the Add Friend screen, scans the QR code, or enters the ID into the input field. The device then acquires this identification information.

[0729] Step 3:

[0730] The terminal adds the acquired identification information to a database.

[0731] (Input) Identification information (QR code or ID)

[0732] (Output) The friend information is added to the database.

[0733] Specifically, the device registers the LINE ID, name, and profile information obtained from the QR code in a database.

[0734] Step 4:

[0735] The user taps the microphone icon on the contact details screen to switch to voice input mode.

[0736] (Input) User action (tapping the microphone icon)

[0737] (Output) Enters voice input standby mode.

[0738] Specifically, the user taps the microphone icon on the contact details screen, causing the device to enter a voice input standby state.

[0739] Step 5:

[0740] The user speaks a group attribute (for example, "account").

[0741] (Input) User's voice (group attributes)

[0742] (Output) Audio data is obtained.

[0743] As a specific operation, the user speaks "business partner" and the terminal receives the voice data.

[0744] Step 6:

[0745] The terminal receives voice input and converts the voice into text data using a voice recognition means.

[0746] (Input) Audio data

[0747] (Output) Text data (e.g., "Business Partner")

[0748] Specifically, the device uses a voice recognition engine to analyze the voice and convert it into text data.

[0749] Step 7:

[0750] The terminal analyzes the converted text data, identifies the corresponding group by means of a database management means, and assigns friends to the group.

[0751] (Input) Text data (e.g., "Business Partner")

[0752] (Output) Friend information assigned to the group

[0753] Specifically, the device matches the text "business partner" with existing groups in the database, and creates new groups as needed, assigning friend information.

[0754] Step 8:

[0755] The device analyzes the voice content using an emotion engine and obtains the user's emotional information.

[0756] (Input) Audio data

[0757] (Output) Emotional information

[0758] Specifically, the device passes the voice recognition results to the emotion engine, which analyzes the emotions and obtains the emotion information.

[0759] Step 9:

[0760] The terminal records the acquired emotion information in a database.

[0761] (Input) Emotion information

[0762] (Output) Emotion information recorded in the database

[0763] Specifically, the device associates the acquired emotion information with friend information and stores it in a database.

[0764] Step 10:

[0765] The device periodically scans your smartphone for contact data.

[0766] (Input) Contact Data

[0767] (Output) Scan results

[0768] Specifically, the device will automatically scan your contact data every night.

[0769] Step 11:

[0770] The device checks the contact data against the messaging application's database.

[0771] (Input) Contact data, messaging application data

[0772] (Output) Matching result

[0773] Specifically, the device compares the contact data with the data in the messaging application to find matching information.

[0774] Step 12:

[0775] The server checks for a matching friend and retrieves additional information from the contact data, such as occupation and company name.

[0776] (Input) Matching results, contact information

[0777] (Output) Additional information (occupation, company name)

[0778] Specifically, the server checks for matching friend information and retrieves additional information.

[0779] Step 13:

[0780] The server integrates the acquired additional information into the friend information of the messaging application.

[0781] (Input) Additional information (occupation, company name)

[0782] (Output) Integrated friend information

[0783] Specifically, the server combines the acquired information with friend information in the messaging application.

[0784] Step 14:

[0785] The device updates the database.

[0786] (Input) Integrated friend information

[0787] (Output) Updated database

[0788] Specifically, the terminal reflects information from the server and keeps the database up to date.

[0789] Step 15:

[0790] A user selects the "Manage Groups" option from the friends list menu of a messaging application.

[0791] (Input) User action (selection of "Group Management")

[0792] (Output) Group management screen

[0793] Specifically, the user opens the friend list menu and selects "Group Management."

[0794] Step 16:

[0795] View all groups and friend information that your device is currently registered to.

[0796] (Input) User action

[0797] (Output) Displayed group and friend information

[0798] Specifically, the device retrieves all group and friend information from the database and displays it on the screen.

[0799] Step 17:

[0800] The user taps on the displayed group and edits the attribute information of the friend.

[0801] (Input) User action (tap on group, edit)

[0802] (Output) Edited attribute information

[0803] Specifically, the user taps the group he or she wants to edit and edits the attribute information.

[0804] Step 18:

[0805] Users can add new attribute categories and delete unnecessary groups.

[0806] (Input) User action (add, delete)

[0807] (Output) Updated attribute categories

[0808] As a specific operation, the user adds new attributes and deletes unnecessary groups.

[0809] Step 19:

[0810] The device saves the user's edits in a database.

[0811] (Input) Edited attribute information

[0812] (Output) Saved database

[0813] Specifically, the terminal records the edited content of the user in a database and updates it to the latest version.

[0814] (Application example 2)

[0815] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0816] Modern industrial and production sites require smooth communication between workers and robots. In particular, it is important to improve production efficiency and worker satisfaction by providing appropriate support based on the worker's emotions and state. However, existing systems lack robots that can properly understand the worker's voice instructions and respond flexibly based on emotion recognition, so new technologies are needed to solve this problem.

[0817] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice input means for inputting friend attribute information by voice, a voice recognition means for converting the voice-input attribute information into text data, a means for registering friends using QR codes or ID search, a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to those groups, an emotion recognition means for analyzing the acquired voice content and acquiring user emotion information, and a database storage means for storing the acquired emotion information in association with the friend attribute information. This makes it possible to realize a factory robot that appropriately analyzes a worker's voice instructions and provides flexible and optimal support based on the emotion information.

[0818] "Friend attribute information" is information set by the user to classify friends, and indicates categories such as "business partner" and "customer."

[0819] The "voice input means" is a device or system that allows a user to input attribute information of a friend through voice.

[0820] "Speech recognition means" refers to a technology or system that converts speech input by a speech input means into text data.

[0821] "Methods for registering friends using QR codes or ID searches" refers to methods in which users scan QR codes or enter IDs to identify and register friends.

[0822] The "database management means" is a system that has the function of analyzing text data, identifying corresponding groups, and assigning friends to those groups.

[0823] "Emotion recognition means" refers to a technique or device for analyzing the content of a voice and acquiring information about the user's emotions.

[0824] The "database storage means" is a system that has the function of storing the obtained emotional information in association with the attribute information of the friends.

[0825] "Means for scanning telephone directory data and comparing it with a database" refers to a technique for periodically checking existing telephone directory data and comparing it with information in a database.

[0826] The "data linking means" is a system that has the function of obtaining matching friend information from the phone book and adding it to the friend information in the database.

[0827] The "classification means" is a technology that uses a database management means to sort the attribute information of friends into new groups.

[0828] The "means for setting the support level" is a system that has the function of analyzing the voice instructions of the worker and acquiring emotional information to provide appropriate support according to the worker's condition.

[0829] The present invention provides a system for efficiently managing and classifying friend information and work support information using voice input and emotion recognition of a worker. Specific embodiments for carrying out the present invention will be described below.

[0830] Hardware and software used

[0831] This system uses the following hardware and software:

[0832] Audio input microphone (e.g., a general audio input device)

[0833] Industrial robots (e.g. Universal Robots)

[0834] Speech recognition API (e.g., Google Cloud Speech-to-Text)

[0835] Emotion recognition engine (e.g., Affectiva's Emotion Recognition SDK)

[0836] Data management software (e.g., SQLite database)

[0837] System configuration

[0838] The system consists of the following main components:

[0839] 1. Voice input means: A microphone device that allows workers to input their friends' attribute information and work instructions through voice.

[0840] 2. Speech recognition means: A speech recognition API is used to convert voice-input information into text data.

[0841] 3. How to register friends: Register friends using QR codes or ID searches.

[0842] 4. Database management means: Analyze the text data, identify the corresponding groups, and assign friends to those groups.

[0843] 5. Emotion recognition means: Analyzes the user's emotions from the voice content and obtains emotional information.

[0844] 6. Database storage means: The acquired emotional information is stored in association with the friend's attribute information.

[0845] 7. Data integration method: Scan existing phone book data and add it to the database.

[0846] 8. Support level setting means: Set an appropriate support level based on voice instructions and emotional information.

[0847] System Operation

[0848] First, the worker inputs the friend's attribute information and work instructions using the voice input means. This voice data is converted into text data by the voice recognition means. This text data is analyzed by the database management means and classified into appropriate groups.

[0849] Next, the emotion recognition means analyzes the voice data to obtain the user's emotion information, which is then stored in the database by the database storage means in association with the friend's attribute information.

[0850] Furthermore, the data linking means periodically scans the telephone book data, and the matching friend information is acquired from the telephone book and added to the database.

[0851] Finally, the support level setting means analyzes the voice instructions and emotional information of the worker and sets an appropriate support level, allowing the industrial robot to provide appropriate support according to the worker's workload.

[0852] Specific examples

[0853] For example, a worker may give a voice command such as "I'm tired today, so please increase my support." This voice is first converted into text data by a voice recognition API, and then emotional information indicating "fatigue" is obtained by an emotion recognition engine. Based on this information, the database management means increases the level of support for the worker, and the industrial robot automatically strengthens its support.

[0854] Prompt Sentence Examples

[0855] "Please create a program that analyzes the worker's voice instructions, obtains emotional information using an emotion recognition engine, and sets an appropriate assistance level. Please also provide specific details about the voice recognition API and emotion recognition engine you will use, as well as how you will save the data."

[0856] As described above, according to the embodiment of the invention, it is possible to efficiently manage and automate friend information and task support information based on the user's voice input and emotion recognition.

[0857] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0858] Step 1:

[0859] Audio Input:

[0860] The user inputs voice into the microphone device. The voice input includes attribute information of friends and work instructions. This input data is captured by the microphone device as an analog voice signal.

[0861] Step 2:

[0862] Voice Recognition:

[0863] The device sends voice input data to a speech recognition API (e.g., Google Cloud Speech-to-Text), which converts the analog voice signal into text data. In this case, the input is voice data and the output is text data.

[0864] Step 3:

[0865] Text data analysis:

[0866] The device analyzes the text data received from the voice recognition API using a database management means. The input is text data, and the data content is analyzed to identify related groups and assign the corresponding friend information to those groups. The output is the friend information categorized into groups.

[0867] Step 4:

[0868] Add as friend:

[0869] When a user registers a friend using a QR code or ID search, the device accepts the QR code scan or ID input and retrieves the corresponding friend information. The input is the QR code information or ID information, and the output is the friend information.

[0870] Step 5:

[0871] Emotion recognition:

[0872] The device uses an emotion recognition engine (e.g., Affectiva SDK) to recognize the user's emotions from text data. The emotion recognition engine receives text data as input and outputs emotional information. In this case, the input is text data and the output is emotional information.

[0873] Step 6:

[0874] Emotional information storage:

[0875] The emotion information acquired by the device from the emotion recognition engine is stored in a database by a database storage means in association with the friend's attribute information. The input is the emotion information and the friend's attribute information, and the output is the updated database.

[0876] Step 7:

[0877] Scan and match phone book data:

[0878] The device periodically scans its existing contact list and matches it with the friend information in its database. The input is the contact list data, and the output is the matching friend information.

[0879] Step 8:

[0880] Data integration:

[0881] The server adds the matching friend information retrieved from the phone book to the friend information in the database. The input is the phone book data and the database information, and the output is the updated friend information.

[0882] Step 9:

[0883] Support Level Settings:

[0884] The server uses the results of analysis by an emotion recognition engine based on the voice instructions and emotional information to set an appropriate support level. The input is the voice instructions and emotional information, and the output is the set support level.

[0885] Step 10:

[0886] Industrial robot control:

[0887] The terminal issues instructions to the industrial robot to assist in the work according to the set assistance level. This operation is performed using a robot motion control program. The input is assistance level information, and the output is robot motion control.

[0888] The system of this embodiment enables efficient management and automation of friend information and work support information based on the user's voice input and emotion recognition.

[0889] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0890] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0891] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0892] [Third embodiment]

[0893] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0894] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0895] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0896] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0897] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0898] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0899] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0900] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0901] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0902] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0903] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0904] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0905] This invention relates to a system for efficiently managing and classifying friend information in the LINE Messenger application. Specifically, it provides a function that allows users to register friends using QR codes or ID searches, and then classify the friends into appropriate groups using voice input. It also includes a function that links existing phone book data with LINE Messenger data and automatically updates friend attribute information.

[0906] Program processing

[0907] 1. Add friends

[0908] When a user opens the LINE Messenger application and scans a QR code or searches for an ID on the Add Friends screen, the device uses this information to store the friend's basic information (such as name, ID, and profile picture) in the LINE Messenger database.

[0909] 2. Audio grouping

[0910] After registering a friend, the user taps the microphone icon on the contact details screen to switch to voice input mode. When the user enters the friend's attribute information by voice, the device converts the voice into text data. The device then analyzes the converted text data to check whether a corresponding group exists. If no existing group exists, the device creates a new group, assigns the friend to it, and updates the database.

[0911] 3. Phonebook data integration

[0912] The device periodically scans the smartphone's contacts and compares them with the existing LINE Messenger database. If a match is found, additional information such as occupation and company name is retrieved from the contacts and added to the LINE Messenger contact information.

[0913] 4. Attribute Information Management

[0914] Users select the "Group Management" option from the friend list menu in the LINE Messenger app. The device displays all currently registered groups and friend information. Users can edit the displayed groups and attribute information, add new categories, or delete unnecessary groups. The device reflects the user's actions and updates the database accordingly.

[0915] Example: Classifying your friends' voices

[0916] For example, suppose a user registers a new friend using a QR code and then speaks "business partner." The device converts the speech into text data and assigns the friend to a group called "business partners." The device then scans the phone book to obtain the friend's detailed information (e.g., company name and occupation) and adds it to the LINE Messenger database.

[0917] This invention allows users to efficiently manage and classify information about their friends using only voice input, making it possible to quickly search and refer to the information they need, even when they have a large number of friends. Furthermore, the automatic linking function with phone book data significantly reduces the effort required for manually updating information.

[0918] The processing flow will be explained below.

[0919] Step 1:

[0920] The user opens the LINE Messenger application and scans the QR code on the Add Friends screen or searches for the ID. The device adds the friend information obtained from the QR code or ID to the LINE Messenger database.

[0921] Step 2:

[0922] After the registered friend information is displayed, the user taps the microphone icon on the contact details screen to switch to voice input mode, and the device enters a state waiting for the user's voice input.

[0923] Step 3:

[0924] The user inputs the group attribute by voice (e.g., "business partner"). The terminal receives the voice input and converts the voice into text data using a voice recognition means.

[0925] Step 4:

[0926] The device analyzes the converted text data and compares it with the LINE Messenger database. If there is no existing group, the device creates a new group. If there is an existing group, the device assigns friends to it.

[0927] Step 5:

[0928] The device scans the existing contacts data stored in the user's smartphone. The scan is performed periodically, for example, once a day.

[0929] Step 6:

[0930] The device compares the friend information registered in the LINE Messenger database with the data in the phone book. The server verifies the match and obtains additional information such as occupation and company name from the phone book.

[0931] Step 7:

[0932] The server integrates the additional information it has acquired into LINE Messenger's friend information, and the device updates the database to complete the friend's attribute information.

[0933] Step 8:

[0934] When a user selects the "Group Management" option from the friend list menu in the LINE Messenger application, all groups and friend information currently registered on the device is displayed.

[0935] Step 9:

[0936] Users can tap on a displayed group to edit their friends' attribute information. Users can add new attribute categories or delete unnecessary groups.

[0937] Step 10:

[0938] The device saves the user's edits to the database and reflects the latest status of group and attribute information, allowing the user to always manage the latest friend information.

[0939] Example 1

[0940] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0941] In conventional friend management systems, friend attribute information and group management are done manually, making management cumbersome, especially for users with many friends. Furthermore, since there is no function to update information in conjunction with phone book data, friend information is often out of date. For this reason, there is a demand for a system that can efficiently manage and classify friend information and automatically update the latest information.

[0942] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0943] In this invention, the server includes a voice input means for inputting friend attribute information by voice, a voice recognition means for converting the voice-input attribute information into text data, a means for registering friends using identification data, a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to those groups, and a data linking means for periodically scanning existing contact data, collating the information, acquiring additional information, and updating the database. This allows for efficient friend information management through voice input, and automatic updating of the latest information through linking with phone book data.

[0944] "Friend attribute information" is information relating to specific characteristics or categories of registered friends.

[0945] "Voice input means" refers to a device or function that allows a user to input information into a system using voice.

[0946] "Speech recognition means" refers to technology or devices for converting input speech into text data.

[0947] "Identification data" is data that uniquely identifies a friend, such as a QR code or ID.

[0948] "Database management means" refers to a system or device for efficiently managing, classifying, and storing friend information and related data.

[0949] "Data integration means" refers to the technology or function that compares existing contact data or other data sources with data within the system to obtain and supplement the necessary information.

[0950] A "group" is a category for managing friends who share common attribute information.

[0951] "Contact data" refers to data stored in a telephone directory or the like that includes information such as names, numbers, and company names.

[0952] This invention is a system for efficiently managing and classifying attribute information of friends. This system utilizes the LINE Messenger application, uses voice input to appropriately classify friends, and automatically links with phone book data to keep friend information up to date.

[0953] Hardware and software used

[0954] 1. LINE Messenger application: Used to register and manage friend information.

[0955] 2. Smartphone: The device on which the user runs the LINE Messenger app.

[0956] 3. Voice input engine (e.g. Google Speech-to-Text API): Used to convert voice into text data.

[0957] 4. Database system (e.g. SQLite): Used to store and update friend and group information.

[0958] Specific operation of the system

[0959] Add a friend

[0960] When a user opens the LINE Messenger application and scans a QR code or searches for an ID on the Add Friends screen, the device uses this information to store the friend's basic information (such as name, ID, and profile picture) in the LINE Messenger database.

[0961] Audio grouping

[0962] After registering a friend, the user taps the microphone icon on the contact details screen to switch to voice input mode. When the user enters attribute information such as "business partner" by voice, the device converts the voice into text data. The device analyzes the converted text data, and if there is no existing group, it creates a new group and assigns the friend to that group. The database management means processes this and updates the database.

[0963] Phone book data integration

[0964] The device periodically scans the smartphone's contacts and compares them with the existing LINE Messenger database. If a match is found, additional information such as occupation and company name is retrieved from the contacts and added to the LINE Messenger contacts. This keeps the contacts up to date.

[0965] Attribute information management

[0966] The user selects the "Group Management" option from the friend list menu in the LINE Messenger app. The device displays all registered groups and friend information. The user can edit the displayed groups and attribute information, add new categories, or delete unnecessary groups. The device reflects these operations and updates the database.

[0967] Examples of specific examples and prompts

[0968] For example, if a user registers a new friend using a QR code and speaks "business partner," the device converts the voice to text data and assigns the friend to a group called "business partners." The device then scans the phone book to obtain the friend's detailed information (e.g., company name and occupation) and adds it to the LINE Messenger database. This invention allows users to efficiently manage and categorize friend information using only voice input.

[0969] Example prompt sentence:

[0970] "How do I add new friends to the "Business Partners" group in the LINE Messenger app?"

[0971] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0972] Step 1:

[0973] Start friend registration

[0974] The user opens the LINE Messenger application and scans the QR code on the Add Friends screen or searches for the ID.

[0975] The device will retrieve the QR code information or ID search results based on this input and save the friend's basic information (name, ID, profile picture, etc.) in the LINE Messenger database.

[0976] Step 2:

[0977] Activating voice input mode

[0978] After users register friends using QR codes or ID searches, they can tap the microphone icon on the contact details screen to switch to voice input mode.

[0979] The device launches a voice input engine (e.g., Google Speech-to-Text API) and waits for the user's voice input.

[0980] Step 3:

[0981] Voice input of attribute information

[0982] The user inputs attribute information such as "business partner" by voice.

[0983] The device sends this voice data to the Google Speech-to-Text API and obtains the text data "Customer."

[0984] Input: Audio data

[0985] Output: Text data ("Customer")

[0986] Step 4:

[0987] Attribute information analysis and grouping

[0988] The text data "Customer" acquired by the device is analyzed using a natural language processing engine (e.g., NLTK) to check whether a corresponding group exists.

[0989] If the terminal does not have a group called "Customer", it will create a new group called "Customer".

[0990] The device assigns friends to the "Business Partners" group and updates the LINE Messenger database.

[0991] Input: Text data ("Customer")

[0992] Output: Updated database

[0993] Step 5:

[0994] Phone book data integration

[0995] The device periodically scans the phone book data on your smartphone.

[0996] The device compares the LINE Messenger database with the phone book data, and if a matching friend is found, it retrieves additional information from the phone book, such as occupation and company name.

[0997] The device will add the additional information it has acquired to the LINE Messenger friend information and update the database.

[0998] Input: Phonebook data

[0999] Output: Updated friend information

[1000] Step 6:

[1001] Attribute information management

[1002] The user selects the "Group Management" option from the friend list menu in the LINE Messenger app.

[1003] Displays all groups and friend information to which the device is registered.

[1004] Users can edit the displayed groups and attribute information, add new categories, or delete unnecessary groups.

[1005] The terminal will update the database with these changes.

[1006] Input: User operation (group edit, add, delete)

[1007] Output: Updated database

[1008] This system allows users to efficiently manage and categorize their friends' information using only voice input. In addition, the automatic linking function with phone book data significantly reduces the effort required to manually update information.

[1009] (Application example 1)

[1010] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1011] Traditionally, customer management in brick-and-mortar stores has been largely manual, making the classification and management of customer information particularly cumbersome in stores with large customer databases. Furthermore, classification based on voice input of customer information has not been realized, making efficient operation difficult. Furthermore, there was a need for a system that would reduce the workload of store employees in managing customer information and quickly and accurately reflect customer information.

[1012] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1013] In this invention, the server includes a voice input means for inputting friend attribute information by voice, a voice recognition means for converting the voice-input attribute information into text data, a means for registering friends using QR codes or ID search, a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to those groups, and a customer management means for store employees to manage customer information. This allows store employees to efficiently manage and classify customer information using QR codes or voice input.

[1014] "Voice input means" refers to a device or system that allows a user to input data through voice.

[1015] "Speech recognition means" refers to software or hardware for converting input voice data into text data.

[1016] "Means for registering friends using QR codes or ID search" refers to the functions or processes for registering friend information using QR code scanning or ID search.

[1017] The "database management means for analyzing text data, identifying corresponding groups, and assigning friends to those groups" is a system or program that analyzes text data, identifies appropriate groups, and assigns friend information to those groups.

[1018] "Customer management means" refers to the tools and systems that store employees use to efficiently manage customer information.

[1019] The "means for scanning communication data and comparing it with the communication application's database" is a function that reads existing contact data and compares it with the communication application's database to search for matching information.

[1020] "Data linkage means" refers to methods and technologies for exchanging information between different databases and integrating matching information.

[1021] The "means for editing, adding, and deleting attribute information" refers to an interface and method that allows a user to freely edit, add, or delete attribute information in a database.

[1022] A "means for categorizing customer information via voice input" is a system or function that uses voice input to automatically categorize customer information into specific categories.

[1023] The present invention relates to a system for improving the efficiency of customer management in a store and for classifying customer information through voice input.

[1024] System Overview

[1025] The system includes the following means:

[1026] 1. Voice input means: The server uses a voice input device that allows users to input customer information through voice, including the microphone of a smartphone.

[1027] 2. Speech recognition means: The server uses the speech_recognition library to convert the input voice data into text data.

[1028] 3. Registering friends by QR code or ID search: The server registers customer information by scanning QR code or ID search. This method includes qrcode library.

[1029] 4. Database management means: The server has a database for analyzing the converted text data, identifying the corresponding groups, and assigning customers to those groups. This database uses SQLite.

[1030] 5. Customer Management Tool: A management tool used by store employees to manage customer information. This tool supports editing, adding, and grouping customer information.

[1031] Program processing explanation

[1032] Processing of voice input means

[1033] When a user enters customer information by voice, the terminal captures the voice data through the microphone. This data is converted into text data by a voice recognition means. The specific hardware used is the microphone of a smartphone.

[1034] Processing of speech recognition means

[1035] The device uses the speech_recognition library to convert the recorded voice data into text data. This data conversion is done in real time, so the user can move on to the next operation without any delay.

[1036] Registration via QR code or ID search

[1037] When a user registers a new customer, they scan a QR code using their smartphone camera or enter their ID, which is sent to the server and stored in a SQLite database.

[1038] Database Management

[1039] The information obtained from the text data is analyzed on the server and classified into corresponding groups. Natural language processing technology is likely to be used for this process. SQLite is used for database management, allowing for efficient data retrieval and updating.

[1040] Examples of customer management

[1041] For example, imagine an employee registering a new customer with a QR code and typing "Regular Customer" by voice. This speech is converted to text by the speech_recognition library, and the customer's information is assigned to a group called "Regular Customer." This can all be done on a smartphone.

[1042] Prompt Sentence Examples

[1043] When using a generative AI model, you can use the following prompts:

[1044] "Create an application that categorizes customers into 'loyal customers' groups based on their voice."

[1045] This system allows store employees to efficiently manage and classify customer information, and makes it possible to quickly search and refer to necessary information, especially when there are a large number of customers. In addition, the combined use of voice input and QR codes makes customer information management even more efficient.

[1046] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1047] Step 1:

[1048] Users scan a QR code using their smartphone camera to register a new customer.

[1049] Input: QR code image

[1050] Data processing: Using the QR code library, the QR code is decoded, and basic customer information (name, contact details, etc.) is obtained.

[1051] Output: Customer basic information (name, contact details)

[1052] Step 2:

[1053] The basic customer information acquired by the terminal is sent to the server and stored in an SQLite database.

[1054] Input: Customer basic information

[1055] Data processing: Use SQL queries to register customer information in the database.

[1056] Output: New customer information registered in the database

[1057] Step 3:

[1058] Users enter customer attribute information through voice input, and the voice is recorded using the smartphone's microphone.

[1059] Input: Audio data

[1060] Data processing: Capture audio data through the microphone.

[1061] Output: Recorded audio data

[1062] Step 4:

[1063] The server converts the audio data into text data using the speech_recognition library.

[1064] Input: Audio data

[1065] Data processing: Voice data is converted into text data using a voice recognition algorithm.

[1066] Output: Text data (e.g., "Regular Customer")

[1067] Step 5:

[1068] The device analyzes the converted text data and classifies it into appropriate customer groups.

[1069] Input: Text data

[1070] Data processing: Using natural language processing technology, the text data is analyzed and group names are identified.

[1071] Output: Group information (e.g., "Regular Customers")

[1072] Step 6:

[1073] The server updates the customer and group information into a database and assigns customers to specific groups.

[1074] Input: Customer information, group information

[1075] Data manipulation: Using SQL queries to add new group information and update customer information in the database.

[1076] Output: Updated database

[1077] Step 7:

[1078] The user checks the group information reflected in the database and edits, adds, or deletes as necessary.

[1079] Input: User operation (edit, add, delete)

[1080] Data processing: The operations performed by the user are reflected in the database through the user interface.

[1081] Output: Updated customer and group information

[1082] This series of processes enables store employees to efficiently manage and classify customer information using QR codes and voice input.

[1083] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1084] This invention realizes more advanced friend management by combining a system that manages and classifies friend information using voice input means and registers friends using QR codes and ID searches with an emotion engine that recognizes the user's emotions. This system provides functions that allow users to register friends, group them using voice input, and acquire emotion information and reflect it in a database.

[1085] Program processing

[1086] Add friends and manage their basic information

[1087] 1. The user opens the LINE Messenger application and scans the QR code on the Add Friends screen or searches for an ID. The device adds the friend information obtained from the QR code or ID to the LINE Messenger database.

[1088] Audio grouping and emotion recognition

[1089] 2. The user taps the microphone icon on the contact details screen to switch to voice input mode, and the device enters a state where it is waiting for the user's voice input.

[1090] 3. The user inputs the group attribute by voice (e.g., "business partner"). The terminal receives the voice input and converts the voice into text data using a voice recognition means.

[1091] 4. The device analyzes the converted text data and compares it with the LINE Messenger database. If the group does not exist, the device creates a new group. If it does, the device assigns friends to the group.

[1092] 5. The device analyzes the voice content from the voice input means using an emotion engine to obtain the user's emotional information. For example, if a user speaks emotionally about an "important customer" when assigning a friend to the "business partner" group, that emotional information is obtained.

[1093] 6. The device records the acquired emotional information in a database and stores it in association with the friend's attribute information. This adds detailed information such as "business partner" and "important."

[1094] Integration with existing phone book data

[1095] 7. The device periodically scans the phone book data on the smartphone, for example, once a day.

[1096] 8. The device compares the friend information registered in the LINE Messenger database with the data in the phone book. The server verifies the match and obtains additional information such as occupation and company name from the phone book.

[1097] 9. The server integrates the additional information it has acquired into the friend information on LINE Messenger. The device updates the database and completes the friend's attribute information.

[1098] Managing and editing attribute information

[1099] 10. The user selects the "Group Management" option from the friend list menu in the LINE Messenger application. This displays all groups and friend information that the device is currently registered in.

[1100] 11. Users can tap on the displayed group to edit their friends' attribute information. Users can add new attribute categories or delete unnecessary groups.

[1101] 12. The device saves the user's edits to the database and reflects the latest status of group and attribute information, allowing the user to always manage the latest friend information.

[1102] Specific examples

[1103] Friend voice classification and emotion recognition

[1104] For example, after registering a new friend using a QR code, a user can enter "business partner" by voice. If the user speaks with emotion, the emotion engine analyzes the emotion. The device converts the voice into text data and assigns the friend to a group called "business partners." Furthermore, emotional information such as "important customer" is also acquired and recorded in the database. Next, the device scans the phone book to obtain the friend's detailed information (e.g., company name and occupation) and adds it to the friend information.

[1105] This invention allows users to efficiently manage and classify information about their friends through voice input and emotion recognition. This makes it possible to quickly search and refer to the information needed, even when there are a large number of friends. Furthermore, the automatic linking function of the emotion engine significantly reduces the effort required for manually updating information.

[1106] The processing flow will be explained below.

[1107] Step 1:

[1108] The user opens the LINE Messenger app and scans the QR code on the Add Friends screen or searches for the ID. The device adds the friend information obtained from the QR code or ID to the LINE Messenger database.

[1109] Step 2:

[1110] After the registered friend information is displayed, the user taps the microphone icon on the contact details screen to switch to voice input mode, and the device enters a state waiting for the user's voice input.

[1111] Step 3:

[1112] The user inputs the group attribute by voice (e.g., "business partner"). The terminal receives the voice input and converts the voice into text data using a voice recognition means.

[1113] Step 4:

[1114] The device inputs the text data obtained by voice recognition into an emotion engine, which analyzes the user's emotional information (e.g., the emphasis on "important business partner").

[1115] Step 5:

[1116] The device analyzes the converted text data and the analyzed emotional information and compares it with the LINE Messenger database. If no existing group exists, the device creates a new group. If an existing group exists, the device assigns friends to that group.

[1117] Step 6:

[1118] The device scans the existing contacts data in the user's smartphone. This scan operation is performed periodically (for example, once a day).

[1119] Step 7:

[1120] The device compares the friend information registered in the LINE Messenger database with the data in the phone book. The server verifies the match and obtains additional information such as occupation and company name from the phone book.

[1121] Step 8:

[1122] The server integrates the additional information it has acquired into LINE Messenger's friend information, and the device updates the database to complete the friend's attribute information.

[1123] Step 9:

[1124] When a user selects the "Group Management" option from the friend list menu in the LINE Messenger app, all groups and friend information currently registered on the device will be displayed.

[1125] Step 10:

[1126] Users can tap on a displayed group to edit their friends' attribute information. Users can add new attribute categories or delete unnecessary groups.

[1127] Step 11:

[1128] The device saves the user's edits to the database, reflecting the latest status of group and attribute information, allowing the user to always manage the latest friend information.

[1129] Specific examples

[1130] For example, a user registers a new friend using a QR code and then speaks to that friend, saying "business partner." If the user emphasizes "important business partner," the emotion engine analyzes that emphasis. The device converts the speech into text data, and based on the "important" information obtained from the emotion engine, creates a new group called "important business partner" and assigns the friend to it. The device then scans the phone book to obtain additional information about the friend (for example, the company name "XYZ Corporation" and the occupation "Sales Representative"), which is then integrated into the LINE Messenger database. This allows users to manage their friend information in detail.

[1131] Example 2

[1132] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1133] In today's world, many users need to manage and categorize their friend information. However, this management process is time-consuming, and adding emotional information is particularly difficult. Furthermore, there is a lack of efficient ways to integrate existing contact data with data from messaging applications. Therefore, there is a need for efficient and detailed management of friend information.

[1134] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1135] In this invention, the server includes a voice input means for acquiring friend attribute information input by voice, a voice recognition means for converting the voice-input attribute information into text data, a means for registering friends using identification information, a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to those groups, and an emotion engine means for analyzing the user's emotions and adding the acquired emotion information to the attribute information. This allows the user to efficiently register and classify friend information using voice, and further enables more detailed and accurate friend management by integrating the emotion information.

[1136] "Voice input means" refers to a device or system for receiving voice and analyzing the voice data.

[1137] "Speech recognition means" refers to the technology or algorithms used to convert voice data into text data.

[1138] "Identification information" refers to information that uniquely identifies a friend, such as a QR code or ID.

[1139] "Database management measures" refers to the systems and processes used to analyze text data and classify and assign friend information to appropriate groups.

[1140] The "emotion engine means" refers to an engine or software for analyzing the user's emotions from voice input and acquiring the emotion information.

[1141] "Data linkage means" refers to the means for collating and integrating data between different databases.

[1142] The present invention is a system that manages and classifies friend information by voice, registers friends using identification information, and analyzes the user's emotions and adds that information. This system includes a voice input means, a voice recognition means, a database management means, an emotion engine means, and a data linking means.

[1143] First, the user opens the appropriate application to register a friend. The user scans or enters identification information (e.g., QR code or ID) on the friend addition screen, and the device receives it and adds the friend information to the database.

[1144] The user taps the microphone icon on the contact details screen to switch to voice input mode. This puts the device into a voice input standby state, and the user voice-inputs the group attribute. For example, if the user voice-inputs "business partner," the device receives the voice and converts the voice into text data using the voice recognition means. The device then analyzes the converted text data, identifies the corresponding group using the database management means, and assigns friends to the group.

[1145] Furthermore, the terminal analyzes the voice content using the emotion engine means to acquire the user's emotion information. For example, if the user assigns a friend to a group called "business partners" and speaks with emotion such as "important customer," that emotion information is acquired. The acquired emotion information is added to the friend's attribute information by the database management means and recorded in the database.

[1146] Periodically, the device scans the smartphone's contact data and matches the existing contact data with the data in the messaging application. The server checks for matching friend information, extracts additional information from the contact data, such as occupation or company name, and integrates it into the friend information in the messaging application. The device then updates the database with this information.

[1147] Users can also edit their friends' attribute information by selecting the "Group Management" option from the friend list menu in the messaging application and tapping on a displayed group. Users can add new attribute categories or delete unnecessary groups. The device saves the user's edits to the database, ensuring that the latest friend information is always available.

[1148] Specific examples

[1149] For example, after registering a new friend using a QR code, a user can enter "business partner" by voice. If the user speaks with emotion, the emotion engine analyzes the emotion. The device converts the voice into text data and assigns the friend to a group called "business partners." Furthermore, the emotional information "important customer" is also acquired and recorded in the database. Next, the device scans the contact data to obtain the friend's detailed information (e.g., company name and occupation) and can add the friend to the friend information.

[1150] Prompt Sentence Examples

[1151] "Please register new friends using QR codes and group them using voice input. Also, please identify emotions and add appropriate emotional information when registering."

[1152] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1153] Step 1:

[1154] The user launches a messaging application.

[1155] (Input) User action (application launch)

[1156] (Output) The application starts and the home screen is displayed.

[1157] Specifically, a user taps an icon on their smartphone to launch the application, and the messaging application home screen appears on the screen.

[1158] Step 2:

[1159] The user scans the QR code on the Add Friend screen and enters their identification information.

[1160] (Input) Scan QR code or enter ID

[1161] (Output) Identification information is obtained.

[1162] Specifically, the user taps the camera icon on the Add Friend screen, scans the QR code, or enters the ID into the input field. The device then acquires this identification information.

[1163] Step 3:

[1164] The terminal adds the acquired identification information to a database.

[1165] (Input) Identification information (QR code or ID)

[1166] (Output) The friend information is added to the database.

[1167] Specifically, the device registers the LINE ID, name, and profile information obtained from the QR code in a database.

[1168] Step 4:

[1169] The user taps the microphone icon on the contact details screen to switch to voice input mode.

[1170] (Input) User action (tapping the microphone icon)

[1171] (Output) Enters voice input standby mode.

[1172] Specifically, the user taps the microphone icon on the contact details screen, causing the device to enter a voice input standby state.

[1173] Step 5:

[1174] The user speaks a group attribute (for example, "account").

[1175] (Input) User's voice (group attributes)

[1176] (Output) Audio data is obtained.

[1177] As a specific operation, the user speaks "business partner" and the terminal receives the voice data.

[1178] Step 6:

[1179] The terminal receives voice input and converts the voice into text data using a voice recognition means.

[1180] (Input) Audio data

[1181] (Output) Text data (e.g., "Business Partner")

[1182] Specifically, the device uses a voice recognition engine to analyze the voice and convert it into text data.

[1183] Step 7:

[1184] The terminal analyzes the converted text data, identifies the corresponding group by means of a database management means, and assigns friends to the group.

[1185] (Input) Text data (e.g., "Business Partner")

[1186] (Output) Friend information assigned to the group

[1187] Specifically, the device matches the text "business partner" with existing groups in the database, and creates new groups as needed, assigning friend information.

[1188] Step 8:

[1189] The device analyzes the voice content using an emotion engine and obtains the user's emotional information.

[1190] (Input) Audio data

[1191] (Output) Emotional information

[1192] Specifically, the device passes the voice recognition results to the emotion engine, which analyzes the emotions and obtains the emotion information.

[1193] Step 9:

[1194] The terminal records the acquired emotion information in a database.

[1195] (Input) Emotion information

[1196] (Output) Emotion information recorded in the database

[1197] Specifically, the device associates the acquired emotion information with friend information and stores it in a database.

[1198] Step 10:

[1199] The device periodically scans your smartphone for contact data.

[1200] (Input) Contact Data

[1201] (Output) Scan results

[1202] Specifically, the device will automatically scan your contact data every night.

[1203] Step 11:

[1204] The device checks the contact data against the messaging application's database.

[1205] (Input) Contact data, messaging application data

[1206] (Output) Matching result

[1207] Specifically, the device compares the contact data with the data in the messaging application to find matching information.

[1208] Step 12:

[1209] The server checks for a matching friend and retrieves additional information from the contact data, such as occupation and company name.

[1210] (Input) Matching results, contact information

[1211] (Output) Additional information (occupation, company name)

[1212] Specifically, the server checks for matching friend information and retrieves additional information.

[1213] Step 13:

[1214] The server integrates the acquired additional information into the friend information of the messaging application.

[1215] (Input) Additional information (occupation, company name)

[1216] (Output) Integrated friend information

[1217] Specifically, the server combines the acquired information with friend information in the messaging application.

[1218] Step 14:

[1219] The device updates the database.

[1220] (Input) Integrated friend information

[1221] (Output) Updated database

[1222] Specifically, the terminal reflects information from the server and keeps the database up to date.

[1223] Step 15:

[1224] A user selects the "Manage Groups" option from the friends list menu of a messaging application.

[1225] (Input) User action (selection of "Group Management")

[1226] (Output) Group management screen

[1227] Specifically, the user opens the friend list menu and selects "Group Management."

[1228] Step 16:

[1229] View all groups and friend information that your device is currently registered to.

[1230] (Input) User action

[1231] (Output) Displayed group and friend information

[1232] Specifically, the device retrieves all group and friend information from the database and displays it on the screen.

[1233] Step 17:

[1234] The user taps on the displayed group and edits the attribute information of the friend.

[1235] (Input) User action (tap on group, edit)

[1236] (Output) Edited attribute information

[1237] Specifically, the user taps the group he or she wants to edit and edits the attribute information.

[1238] Step 18:

[1239] Users can add new attribute categories and delete unnecessary groups.

[1240] (Input) User action (add, delete)

[1241] (Output) Updated attribute categories

[1242] As a specific operation, the user adds new attributes and deletes unnecessary groups.

[1243] Step 19:

[1244] The device saves the user's edits in a database.

[1245] (Input) Edited attribute information

[1246] (Output) Saved database

[1247] Specifically, the terminal records the edited content of the user in a database and updates it to the latest version.

[1248] (Application example 2)

[1249] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1250] Modern industrial and production sites require smooth communication between workers and robots. In particular, it is important to improve production efficiency and worker satisfaction by providing appropriate support based on the worker's emotions and state. However, existing systems lack robots that can properly understand the worker's voice instructions and respond flexibly based on emotion recognition, so new technologies are needed to solve this problem.

[1251] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice input means for inputting friend attribute information by voice, a voice recognition means for converting the voice-input attribute information into text data, a means for registering friends using QR codes or ID search, a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to those groups, an emotion recognition means for analyzing the acquired voice content and acquiring user emotion information, and a database storage means for storing the acquired emotion information in association with the friend attribute information. This makes it possible to realize a factory robot that appropriately analyzes a worker's voice instructions and provides flexible and optimal support based on the emotion information.

[1252] "Friend attribute information" is information set by the user to classify friends, and indicates categories such as "business partner" and "customer."

[1253] The "voice input means" is a device or system that allows a user to input attribute information of a friend through voice.

[1254] "Speech recognition means" refers to a technology or system that converts speech input by a speech input means into text data.

[1255] "Methods for registering friends using QR codes or ID searches" refers to methods in which users scan QR codes or enter IDs to identify and register friends.

[1256] The "database management means" is a system that has the function of analyzing text data, identifying corresponding groups, and assigning friends to those groups.

[1257] "Emotion recognition means" refers to a technique or device for analyzing the content of a voice and acquiring information about the user's emotions.

[1258] The "database storage means" is a system that has the function of storing the obtained emotional information in association with the attribute information of the friends.

[1259] "Means for scanning telephone directory data and comparing it with a database" refers to a technique for periodically checking existing telephone directory data and comparing it with information in a database.

[1260] The "data linking means" is a system that has the function of obtaining matching friend information from the phone book and adding it to the friend information in the database.

[1261] The "classification means" is a technology that uses a database management means to sort the attribute information of friends into new groups.

[1262] The "means for setting the support level" is a system that has the function of analyzing the voice instructions of the worker and acquiring emotional information to provide appropriate support according to the worker's condition.

[1263] The present invention provides a system for efficiently managing and classifying friend information and work support information using voice input and emotion recognition of a worker. Specific embodiments for carrying out the present invention will be described below.

[1264] Hardware and software used

[1265] This system uses the following hardware and software:

[1266] Audio input microphone (e.g., a general audio input device)

[1267] Industrial robots (e.g. Universal Robots)

[1268] Speech recognition API (e.g., Google Cloud Speech-to-Text)

[1269] Emotion recognition engine (e.g., Affectiva's Emotion Recognition SDK)

[1270] Data management software (e.g., SQLite database)

[1271] System configuration

[1272] The system consists of the following main components:

[1273] 1. Voice input means: A microphone device that allows workers to input their friends' attribute information and work instructions through voice.

[1274] 2. Speech recognition means: A speech recognition API is used to convert voice-input information into text data.

[1275] 3. How to register friends: Register friends using QR codes or ID searches.

[1276] 4. Database management means: Analyze the text data, identify the corresponding groups, and assign friends to those groups.

[1277] 5. Emotion recognition means: Analyzes the user's emotions from the voice content and obtains emotional information.

[1278] 6. Database storage means: The acquired emotional information is stored in association with the friend's attribute information.

[1279] 7. Data integration method: Scan existing phone book data and add it to the database.

[1280] 8. Support level setting means: Set an appropriate support level based on voice instructions and emotional information.

[1281] System Operation

[1282] First, the worker inputs the friend's attribute information and work instructions using the voice input means. This voice data is converted into text data by the voice recognition means. This text data is analyzed by the database management means and classified into appropriate groups.

[1283] Next, the emotion recognition means analyzes the voice data to obtain the user's emotion information, which is then stored in the database by the database storage means in association with the friend's attribute information.

[1284] Furthermore, the data linking means periodically scans the telephone book data, and the matching friend information is acquired from the telephone book and added to the database.

[1285] Finally, the support level setting means analyzes the voice instructions and emotional information of the worker and sets an appropriate support level, allowing the industrial robot to provide appropriate support according to the worker's workload.

[1286] Specific examples

[1287] For example, a worker may give a voice command such as "I'm tired today, so please increase my support." This voice is first converted into text data by a voice recognition API, and then emotional information indicating "fatigue" is obtained by an emotion recognition engine. Based on this information, the database management means increases the level of support for the worker, and the industrial robot automatically strengthens its support.

[1288] Prompt Sentence Examples

[1289] "Please create a program that analyzes the worker's voice instructions, obtains emotional information using an emotion recognition engine, and sets an appropriate assistance level. Please also provide specific details about the voice recognition API and emotion recognition engine you will use, as well as how you will save the data."

[1290] As described above, according to the embodiment of the invention, it is possible to efficiently manage and automate friend information and task support information based on the user's voice input and emotion recognition.

[1291] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1292] Step 1:

[1293] Audio Input:

[1294] The user inputs voice into the microphone device. The voice input includes attribute information of friends and work instructions. This input data is captured by the microphone device as an analog voice signal.

[1295] Step 2:

[1296] Voice Recognition:

[1297] The device sends voice input data to a speech recognition API (e.g., Google Cloud Speech-to-Text), which converts the analog voice signal into text data. In this case, the input is voice data and the output is text data.

[1298] Step 3:

[1299] Text data analysis:

[1300] The device analyzes the text data received from the voice recognition API using a database management means. The input is text data, and the data content is analyzed to identify related groups and assign the corresponding friend information to those groups. The output is the friend information categorized into groups.

[1301] Step 4:

[1302] Add as friend:

[1303] When a user registers a friend using a QR code or ID search, the device accepts the QR code scan or ID input and retrieves the corresponding friend information. The input is the QR code information or ID information, and the output is the friend information.

[1304] Step 5:

[1305] Emotion recognition:

[1306] The device uses an emotion recognition engine (e.g., Affectiva SDK) to recognize the user's emotions from text data. The emotion recognition engine receives text data as input and outputs emotional information. In this case, the input is text data and the output is emotional information.

[1307] Step 6:

[1308] Emotional information storage:

[1309] The emotion information acquired by the device from the emotion recognition engine is stored in a database by a database storage means in association with the friend's attribute information. The input is the emotion information and the friend's attribute information, and the output is the updated database.

[1310] Step 7:

[1311] Scan and match phone book data:

[1312] The device periodically scans its existing contact list and matches it with the friend information in its database. The input is the contact list data, and the output is the matching friend information.

[1313] Step 8:

[1314] Data integration:

[1315] The server adds the matching friend information retrieved from the phone book to the friend information in the database. The input is the phone book data and the database information, and the output is the updated friend information.

[1316] Step 9:

[1317] Support Level Settings:

[1318] The server uses the results of analysis by an emotion recognition engine based on the voice instructions and emotional information to set an appropriate support level. The input is the voice instructions and emotional information, and the output is the set support level.

[1319] Step 10:

[1320] Industrial robot control:

[1321] The terminal issues instructions to the industrial robot to assist in the work according to the set assistance level. This operation is performed using a robot motion control program. The input is assistance level information, and the output is robot motion control.

[1322] The system of this embodiment enables efficient management and automation of friend information and work support information based on the user's voice input and emotion recognition.

[1323] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1324] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1325] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1326] [Fourth embodiment]

[1327] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1328] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1329] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1330] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1331] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1332] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1333] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1334] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1335] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1336] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1337] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1338] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1339] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1340] This invention relates to a system for efficiently managing and classifying friend information in the LINE Messenger application. Specifically, it provides a function that allows users to register friends using QR codes or ID searches, and then classify the friends into appropriate groups using voice input. It also includes a function that links existing phone book data with LINE Messenger data and automatically updates friend attribute information.

[1341] Program processing

[1342] 1. Add friends

[1343] When a user opens the LINE Messenger application and scans a QR code or searches for an ID on the Add Friends screen, the device uses this information to store the friend's basic information (such as name, ID, and profile picture) in the LINE Messenger database.

[1344] 2. Audio grouping

[1345] After registering a friend, the user taps the microphone icon on the contact details screen to switch to voice input mode. When the user enters the friend's attribute information by voice, the device converts the voice into text data. The device then analyzes the converted text data to check whether a corresponding group exists. If no existing group exists, the device creates a new group, assigns the friend to it, and updates the database.

[1346] 3. Phonebook data integration

[1347] The device periodically scans the smartphone's contacts and compares them with the existing LINE Messenger database. If a match is found, additional information such as occupation and company name is retrieved from the contacts and added to the LINE Messenger contact information.

[1348] 4. Attribute Information Management

[1349] Users select the "Group Management" option from the friend list menu in the LINE Messenger app. The device displays all currently registered groups and friend information. Users can edit the displayed groups and attribute information, add new categories, or delete unnecessary groups. The device reflects the user's actions and updates the database accordingly.

[1350] Example: Classifying your friends' voices

[1351] For example, suppose a user registers a new friend using a QR code and then speaks "business partner." The device converts the speech into text data and assigns the friend to a group called "business partners." The device then scans the phone book to obtain the friend's detailed information (e.g., company name and occupation) and adds it to the LINE Messenger database.

[1352] This invention allows users to efficiently manage and classify information about their friends using only voice input, making it possible to quickly search and refer to the information they need, even when they have a large number of friends. Furthermore, the automatic linking function with phone book data significantly reduces the effort required for manually updating information.

[1353] The processing flow will be explained below.

[1354] Step 1:

[1355] The user opens the LINE Messenger application and scans the QR code on the Add Friends screen or searches for the ID. The device adds the friend information obtained from the QR code or ID to the LINE Messenger database.

[1356] Step 2:

[1357] After the registered friend information is displayed, the user taps the microphone icon on the contact details screen to switch to voice input mode, and the device enters a state waiting for the user's voice input.

[1358] Step 3:

[1359] The user inputs the group attribute by voice (e.g., "business partner"). The terminal receives the voice input and converts the voice into text data using a voice recognition means.

[1360] Step 4:

[1361] The device analyzes the converted text data and compares it with the LINE Messenger database. If there is no existing group, the device creates a new group. If there is an existing group, the device assigns friends to it.

[1362] Step 5:

[1363] The device scans the existing contacts data stored in the user's smartphone. The scan is performed periodically, for example, once a day.

[1364] Step 6:

[1365] The device compares the friend information registered in the LINE Messenger database with the data in the phone book. The server verifies the match and obtains additional information such as occupation and company name from the phone book.

[1366] Step 7:

[1367] The server integrates the additional information it has acquired into LINE Messenger's friend information, and the device updates the database to complete the friend's attribute information.

[1368] Step 8:

[1369] When a user selects the "Group Management" option from the friend list menu in the LINE Messenger application, all groups and friend information currently registered on the device is displayed.

[1370] Step 9:

[1371] Users can tap on a displayed group to edit their friends' attribute information. Users can add new attribute categories or delete unnecessary groups.

[1372] Step 10:

[1373] The device saves the user's edits to the database and reflects the latest status of group and attribute information, allowing the user to always manage the latest friend information.

[1374] Example 1

[1375] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1376] In conventional friend management systems, friend attribute information and group management are done manually, making management cumbersome, especially for users with many friends. Furthermore, since there is no function to update information in conjunction with phone book data, friend information is often out of date. For this reason, there is a demand for a system that can efficiently manage and classify friend information and automatically update the latest information.

[1377] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1378] In this invention, the server includes a voice input means for inputting friend attribute information by voice, a voice recognition means for converting the voice-input attribute information into text data, a means for registering friends using identification data, a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to those groups, and a data linking means for periodically scanning existing contact data, collating the information, acquiring additional information, and updating the database. This allows for efficient friend information management through voice input, and automatic updating of the latest information through linking with phone book data.

[1379] "Friend attribute information" is information relating to specific characteristics or categories of registered friends.

[1380] "Voice input means" refers to a device or function that allows a user to input information into a system using voice.

[1381] "Speech recognition means" refers to technology or devices for converting input speech into text data.

[1382] "Identification data" is data that uniquely identifies a friend, such as a QR code or ID.

[1383] "Database management means" refers to a system or device for efficiently managing, classifying, and storing friend information and related data.

[1384] "Data integration means" refers to the technology or function that compares existing contact data or other data sources with data within the system to obtain and supplement the necessary information.

[1385] A "group" is a category for managing friends who share common attribute information.

[1386] "Contact data" refers to data stored in a telephone directory or the like that includes information such as names, numbers, and company names.

[1387] This invention is a system for efficiently managing and classifying attribute information of friends. This system utilizes the LINE Messenger application, uses voice input to appropriately classify friends, and automatically links with phone book data to keep friend information up to date.

[1388] Hardware and software used

[1389] 1. LINE Messenger application: Used to register and manage friend information.

[1390] 2. Smartphone: The device on which the user runs the LINE Messenger app.

[1391] 3. Voice input engine (e.g. Google Speech-to-Text API): Used to convert voice into text data.

[1392] 4. Database system (e.g. SQLite): Used to store and update friend and group information.

[1393] Specific operation of the system

[1394] Add a friend

[1395] When a user opens the LINE Messenger application and scans a QR code or searches for an ID on the Add Friends screen, the device uses this information to store the friend's basic information (such as name, ID, and profile picture) in the LINE Messenger database.

[1396] Audio grouping

[1397] After registering a friend, the user taps the microphone icon on the contact details screen to switch to voice input mode. When the user enters attribute information such as "business partner" by voice, the device converts the voice into text data. The device analyzes the converted text data, and if there is no existing group, it creates a new group and assigns the friend to that group. The database management means processes this and updates the database.

[1398] Phone book data integration

[1399] The device periodically scans the smartphone's contacts and compares them with the existing LINE Messenger database. If a match is found, additional information such as occupation and company name is retrieved from the contacts and added to the LINE Messenger contacts. This keeps the contacts up to date.

[1400] Attribute information management

[1401] The user selects the "Group Management" option from the friend list menu in the LINE Messenger app. The device displays all registered groups and friend information. The user can edit the displayed groups and attribute information, add new categories, or delete unnecessary groups. The device reflects these operations and updates the database.

[1402] Examples of specific examples and prompts

[1403] For example, if a user registers a new friend using a QR code and speaks "business partner," the device converts the voice to text data and assigns the friend to a group called "business partners." The device then scans the phone book to obtain the friend's detailed information (e.g., company name and occupation) and adds it to the LINE Messenger database. This invention allows users to efficiently manage and categorize friend information using only voice input.

[1404] Example prompt sentence:

[1405] "How do I add new friends to the "Business Partners" group in the LINE Messenger app?"

[1406] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1407] Step 1:

[1408] Start friend registration

[1409] The user opens the LINE Messenger application and scans the QR code on the Add Friends screen or searches for the ID.

[1410] The device will retrieve the QR code information or ID search results based on this input and save the friend's basic information (name, ID, profile picture, etc.) in the LINE Messenger database.

[1411] Step 2:

[1412] Activating voice input mode

[1413] After users register friends using QR codes or ID searches, they can tap the microphone icon on the contact details screen to switch to voice input mode.

[1414] The device launches a voice input engine (e.g., Google Speech-to-Text API) and waits for the user's voice input.

[1415] Step 3:

[1416] Voice input of attribute information

[1417] The user inputs attribute information such as "business partner" by voice.

[1418] The device sends this voice data to the Google Speech-to-Text API and obtains the text data "Customer."

[1419] Input: Audio data

[1420] Output: Text data ("Customer")

[1421] Step 4:

[1422] Attribute information analysis and grouping

[1423] The text data "Customer" acquired by the device is analyzed using a natural language processing engine (e.g., NLTK) to check whether a corresponding group exists.

[1424] If the terminal does not have a group called "Customer", it will create a new group called "Customer".

[1425] The device assigns friends to the "Business Partners" group and updates the LINE Messenger database.

[1426] Input: Text data ("Customer")

[1427] Output: Updated database

[1428] Step 5:

[1429] Phone book data integration

[1430] The device periodically scans the phone book data on your smartphone.

[1431] The device compares the LINE Messenger database with the phone book data, and if a matching friend is found, it retrieves additional information from the phone book, such as occupation and company name.

[1432] The device will add the additional information it has acquired to the LINE Messenger friend information and update the database.

[1433] Input: Phonebook data

[1434] Output: Updated friend information

[1435] Step 6:

[1436] Attribute information management

[1437] The user selects the "Group Management" option from the friend list menu in the LINE Messenger app.

[1438] Displays all groups and friend information to which the device is registered.

[1439] Users can edit the displayed groups and attribute information, add new categories, or delete unnecessary groups.

[1440] The terminal will update the database with these changes.

[1441] Input: User operation (group edit, add, delete)

[1442] Output: Updated database

[1443] This system allows users to efficiently manage and categorize their friends' information using only voice input. In addition, the automatic linking function with phone book data significantly reduces the effort required to manually update information.

[1444] (Application example 1)

[1445] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1446] Traditionally, customer management in brick-and-mortar stores has been largely manual, making the classification and management of customer information particularly cumbersome in stores with large customer databases. Furthermore, classification based on voice input of customer information has not been realized, making efficient operation difficult. Furthermore, there was a need for a system that would reduce the workload of store employees in managing customer information and quickly and accurately reflect customer information.

[1447] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1448] In this invention, the server includes a voice input means for inputting friend attribute information by voice, a voice recognition means for converting the voice-input attribute information into text data, a means for registering friends using QR codes or ID search, a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to those groups, and a customer management means for store employees to manage customer information. This allows store employees to efficiently manage and classify customer information using QR codes or voice input.

[1449] "Voice input means" refers to a device or system that allows a user to input data through voice.

[1450] "Speech recognition means" refers to software or hardware for converting input voice data into text data.

[1451] "Means for registering friends using QR codes or ID search" refers to the functions or processes for registering friend information using QR code scanning or ID search.

[1452] The "database management means for analyzing text data, identifying corresponding groups, and assigning friends to those groups" is a system or program that analyzes text data, identifies appropriate groups, and assigns friend information to those groups.

[1453] "Customer management means" refers to the tools and systems that store employees use to efficiently manage customer information.

[1454] The "means for scanning communication data and comparing it with the communication application's database" is a function that reads existing contact data and compares it with the communication application's database to search for matching information.

[1455] "Data linkage means" refers to methods and technologies for exchanging information between different databases and integrating matching information.

[1456] The "means for editing, adding, and deleting attribute information" refers to an interface and method that allows a user to freely edit, add, or delete attribute information in a database.

[1457] A "means for categorizing customer information via voice input" is a system or function that uses voice input to automatically categorize customer information into specific categories.

[1458] The present invention relates to a system for improving the efficiency of customer management in a store and for classifying customer information through voice input.

[1459] System Overview

[1460] The system includes the following means:

[1461] 1. Voice input means: The server uses a voice input device that allows users to input customer information through voice, including the microphone of a smartphone.

[1462] 2. Speech recognition means: The server uses the speech_recognition library to convert the input voice data into text data.

[1463] 3. Registering friends by QR code or ID search: The server registers customer information by scanning QR code or ID search. This method includes qrcode library.

[1464] 4. Database management means: The server has a database for analyzing the converted text data, identifying the corresponding groups, and assigning customers to those groups. This database uses SQLite.

[1465] 5. Customer Management Tool: A management tool used by store employees to manage customer information. This tool supports editing, adding, and grouping customer information.

[1466] Program processing explanation

[1467] Processing of voice input means

[1468] When a user enters customer information by voice, the terminal captures the voice data through the microphone. This data is converted into text data by a voice recognition means. The specific hardware used is the microphone of a smartphone.

[1469] Processing of speech recognition means

[1470] The device uses the speech_recognition library to convert the recorded voice data into text data. This data conversion is done in real time, so the user can move on to the next operation without any delay.

[1471] Registration via QR code or ID search

[1472] When a user registers a new customer, they scan a QR code using their smartphone camera or enter their ID, which is sent to the server and stored in a SQLite database.

[1473] Database Management

[1474] The information obtained from the text data is analyzed on the server and classified into corresponding groups. Natural language processing technology is likely to be used for this process. SQLite is used for database management, allowing for efficient data retrieval and updating.

[1475] Examples of customer management

[1476] For example, imagine an employee registering a new customer with a QR code and typing "Regular Customer" by voice. This speech is converted to text by the speech_recognition library, and the customer's information is assigned to a group called "Regular Customer." This can all be done on a smartphone.

[1477] Prompt Sentence Examples

[1478] When using a generative AI model, you can use the following prompts:

[1479] "Create an application that categorizes customers into 'loyal customers' groups based on their voice."

[1480] This system allows store employees to efficiently manage and classify customer information, and makes it possible to quickly search and refer to necessary information, especially when there are a large number of customers. In addition, the combined use of voice input and QR codes makes customer information management even more efficient.

[1481] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1482] Step 1:

[1483] Users scan a QR code using their smartphone camera to register a new customer.

[1484] Input: QR code image

[1485] Data processing: Using the QR code library, the QR code is decoded, and basic customer information (name, contact details, etc.) is obtained.

[1486] Output: Customer basic information (name, contact details)

[1487] Step 2:

[1488] The basic customer information acquired by the terminal is sent to the server and stored in an SQLite database.

[1489] Input: Customer basic information

[1490] Data processing: Use SQL queries to register customer information in the database.

[1491] Output: New customer information registered in the database

[1492] Step 3:

[1493] Users enter customer attribute information through voice input, and the voice is recorded using the smartphone's microphone.

[1494] Input: Audio data

[1495] Data processing: Capture audio data through the microphone.

[1496] Output: Recorded audio data

[1497] Step 4:

[1498] The server converts the audio data into text data using the speech_recognition library.

[1499] Input: Audio data

[1500] Data processing: Voice data is converted into text data using a voice recognition algorithm.

[1501] Output: Text data (e.g., "Regular Customer")

[1502] Step 5:

[1503] The device analyzes the converted text data and classifies it into appropriate customer groups.

[1504] Input: Text data

[1505] Data processing: Using natural language processing technology, the text data is analyzed and group names are identified.

[1506] Output: Group information (e.g., "Regular Customers")

[1507] Step 6:

[1508] The server updates the customer and group information into a database and assigns customers to specific groups.

[1509] Input: Customer information, group information

[1510] Data manipulation: Using SQL queries to add new group information and update customer information in the database.

[1511] Output: Updated database

[1512] Step 7:

[1513] The user checks the group information reflected in the database and edits, adds, or deletes as necessary.

[1514] Input: User operation (edit, add, delete)

[1515] Data processing: The operations performed by the user are reflected in the database through the user interface.

[1516] Output: Updated customer and group information

[1517] This series of processes enables store employees to efficiently manage and classify customer information using QR codes and voice input.

[1518] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1519] This invention realizes more advanced friend management by combining a system that manages and classifies friend information using voice input means and registers friends using QR codes and ID searches with an emotion engine that recognizes the user's emotions. This system provides functions that allow users to register friends, group them using voice input, and acquire emotion information and reflect it in a database.

[1520] Program processing

[1521] Add friends and manage their basic information

[1522] 1. The user opens the LINE Messenger application and scans the QR code on the Add Friends screen or searches for an ID. The device adds the friend information obtained from the QR code or ID to the LINE Messenger database.

[1523] Audio grouping and emotion recognition

[1524] 2. The user taps the microphone icon on the contact details screen to switch to voice input mode, and the device enters a state where it is waiting for the user's voice input.

[1525] 3. The user inputs the group attribute by voice (e.g., "business partner"). The terminal receives the voice input and converts the voice into text data using a voice recognition means.

[1526] 4. The device analyzes the converted text data and compares it with the LINE Messenger database. If the group does not exist, the device creates a new group. If it does, the device assigns friends to the group.

[1527] 5. The device analyzes the voice content from the voice input means using an emotion engine to obtain the user's emotional information. For example, if a user speaks emotionally about an "important customer" when assigning a friend to the "business partner" group, that emotional information is obtained.

[1528] 6. The device records the acquired emotional information in a database and stores it in association with the friend's attribute information. This adds detailed information such as "business partner" and "important."

[1529] Integration with existing phone book data

[1530] 7. The device periodically scans the phone book data on the smartphone, for example, once a day.

[1531] 8. The device compares the friend information registered in the LINE Messenger database with the data in the phone book. The server verifies the match and obtains additional information such as occupation and company name from the phone book.

[1532] 9. The server integrates the additional information it has acquired into the friend information on LINE Messenger. The device updates the database and completes the friend's attribute information.

[1533] Managing and editing attribute information

[1534] 10. The user selects the "Group Management" option from the friend list menu in the LINE Messenger application. This displays all groups and friend information that the device is currently registered in.

[1535] 11. Users can tap on the displayed group to edit their friends' attribute information. Users can add new attribute categories or delete unnecessary groups.

[1536] 12. The device saves the user's edits to the database and reflects the latest status of group and attribute information, allowing the user to always manage the latest friend information.

[1537] Specific examples

[1538] Friend voice classification and emotion recognition

[1539] For example, after registering a new friend using a QR code, a user can enter "business partner" by voice. If the user speaks with emotion, the emotion engine analyzes the emotion. The device converts the voice into text data and assigns the friend to a group called "business partners." Furthermore, emotional information such as "important customer" is also acquired and recorded in the database. Next, the device scans the phone book to obtain the friend's detailed information (e.g., company name and occupation) and adds it to the friend information.

[1540] This invention allows users to efficiently manage and classify information about their friends through voice input and emotion recognition. This makes it possible to quickly search and refer to the information needed, even when there are a large number of friends. Furthermore, the automatic linking function of the emotion engine significantly reduces the effort required for manually updating information.

[1541] The processing flow will be explained below.

[1542] Step 1:

[1543] The user opens the LINE Messenger app and scans the QR code on the Add Friends screen or searches for the ID. The device adds the friend information obtained from the QR code or ID to the LINE Messenger database.

[1544] Step 2:

[1545] After the registered friend information is displayed, the user taps the microphone icon on the contact details screen to switch to voice input mode, and the device enters a state waiting for the user's voice input.

[1546] Step 3:

[1547] The user inputs the group attribute by voice (e.g., "business partner"). The terminal receives the voice input and converts the voice into text data using a voice recognition means.

[1548] Step 4:

[1549] The device inputs the text data obtained by voice recognition into an emotion engine, which analyzes the user's emotional information (e.g., the emphasis on "important business partner").

[1550] Step 5:

[1551] The device analyzes the converted text data and the analyzed emotional information and compares it with the LINE Messenger database. If no existing group exists, the device creates a new group. If an existing group exists, the device assigns friends to that group.

[1552] Step 6:

[1553] The device scans the existing contacts data in the user's smartphone. This scan operation is performed periodically (for example, once a day).

[1554] Step 7:

[1555] The device compares the friend information registered in the LINE Messenger database with the data in the phone book. The server verifies the match and obtains additional information such as occupation and company name from the phone book.

[1556] Step 8:

[1557] The server integrates the additional information it has acquired into LINE Messenger's friend information, and the device updates the database to complete the friend's attribute information.

[1558] Step 9:

[1559] When a user selects the "Group Management" option from the friend list menu in the LINE Messenger app, all groups and friend information currently registered on the device will be displayed.

[1560] Step 10:

[1561] Users can tap on a displayed group to edit their friends' attribute information. Users can add new attribute categories or delete unnecessary groups.

[1562] Step 11:

[1563] The device saves the user's edits to the database, reflecting the latest status of group and attribute information, allowing the user to always manage the latest friend information.

[1564] Specific examples

[1565] For example, a user registers a new friend using a QR code and then speaks to that friend, saying "business partner." If the user emphasizes "important business partner," the emotion engine analyzes that emphasis. The device converts the speech into text data, and based on the "important" information obtained from the emotion engine, creates a new group called "important business partner" and assigns the friend to it. The device then scans the phone book to obtain additional information about the friend (for example, the company name "XYZ Corporation" and the occupation "Sales Representative"), which is then integrated into the LINE Messenger database. This allows users to manage their friend information in detail.

[1566] Example 2

[1567] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1568] In today's world, many users need to manage and categorize their friend information. However, this management process is time-consuming, and adding emotional information is particularly difficult. Furthermore, there is a lack of efficient ways to integrate existing contact data with data from messaging applications. Therefore, there is a need for efficient and detailed management of friend information.

[1569] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1570] In this invention, the server includes a voice input means for acquiring friend attribute information input by voice, a voice recognition means for converting the voice-input attribute information into text data, a means for registering friends using identification information, a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to those groups, and an emotion engine means for analyzing the user's emotions and adding the acquired emotion information to the attribute information. This allows the user to efficiently register and classify friend information using voice, and further enables more detailed and accurate friend management by integrating the emotion information.

[1571] "Voice input means" refers to a device or system for receiving voice and analyzing the voice data.

[1572] "Speech recognition means" refers to the technology or algorithms used to convert voice data into text data.

[1573] "Identification information" refers to information that uniquely identifies a friend, such as a QR code or ID.

[1574] "Database management measures" refers to the systems and processes used to analyze text data and classify and assign friend information to appropriate groups.

[1575] The "emotion engine means" refers to an engine or software for analyzing the user's emotions from voice input and acquiring the emotion information.

[1576] "Data linkage means" refers to the means for collating and integrating data between different databases.

[1577] The present invention is a system that manages and classifies friend information by voice, registers friends using identification information, and analyzes the user's emotions and adds that information. This system includes a voice input means, a voice recognition means, a database management means, an emotion engine means, and a data linking means.

[1578] First, the user opens the appropriate application to register a friend. The user scans or enters identification information (e.g., QR code or ID) on the friend addition screen, and the device receives it and adds the friend information to the database.

[1579] The user taps the microphone icon on the contact details screen to switch to voice input mode. This puts the device into a voice input standby state, and the user voice-inputs the group attribute. For example, if the user voice-inputs "business partner," the device receives the voice and converts the voice into text data using the voice recognition means. The device then analyzes the converted text data, identifies the corresponding group using the database management means, and assigns friends to the group.

[1580] Furthermore, the terminal analyzes the voice content using the emotion engine means to acquire the user's emotion information. For example, if the user assigns a friend to a group called "business partners" and speaks with emotion such as "important customer," that emotion information is acquired. The acquired emotion information is added to the friend's attribute information by the database management means and recorded in the database.

[1581] Periodically, the device scans the smartphone's contact data and matches the existing contact data with the data in the messaging application. The server checks for matching friend information, extracts additional information from the contact data, such as occupation or company name, and integrates it into the friend information in the messaging application. The device then updates the database with this information.

[1582] Users can also edit their friends' attribute information by selecting the "Group Management" option from the friend list menu in the messaging application and tapping on a displayed group. Users can add new attribute categories or delete unnecessary groups. The device saves the user's edits to the database, ensuring that the latest friend information is always available.

[1583] Specific examples

[1584] For example, after registering a new friend using a QR code, a user can enter "business partner" by voice. If the user speaks with emotion, the emotion engine analyzes the emotion. The device converts the voice into text data and assigns the friend to a group called "business partners." Furthermore, the emotional information "important customer" is also acquired and recorded in the database. Next, the device scans the contact data to obtain the friend's detailed information (e.g., company name and occupation) and can add the friend to the friend information.

[1585] Prompt Sentence Examples

[1586] "Please register new friends using QR codes and group them using voice input. Also, please identify emotions and add appropriate emotional information when registering."

[1587] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1588] Step 1:

[1589] The user launches a messaging application.

[1590] (Input) User action (application launch)

[1591] (Output) The application starts and the home screen is displayed.

[1592] Specifically, a user taps an icon on their smartphone to launch the application, and the messaging application home screen appears on the screen.

[1593] Step 2:

[1594] The user scans the QR code on the Add Friend screen and enters their identification information.

[1595] (Input) Scan QR code or enter ID

[1596] (Output) Identification information is obtained.

[1597] Specifically, the user taps the camera icon on the Add Friend screen, scans the QR code, or enters the ID into the input field. The device then acquires this identification information.

[1598] Step 3:

[1599] The terminal adds the acquired identification information to a database.

[1600] (Input) Identification information (QR code or ID)

[1601] (Output) The friend information is added to the database.

[1602] Specifically, the device registers the LINE ID, name, and profile information obtained from the QR code in a database.

[1603] Step 4:

[1604] The user taps the microphone icon on the contact details screen to switch to voice input mode.

[1605] (Input) User action (tapping the microphone icon)

[1606] (Output) Enters voice input standby mode.

[1607] Specifically, the user taps the microphone icon on the contact details screen, causing the device to enter a voice input standby state.

[1608] Step 5:

[1609] The user speaks a group attribute (for example, "account").

[1610] (Input) User's voice (group attributes)

[1611] (Output) Audio data is obtained.

[1612] As a specific operation, the user speaks "business partner" and the terminal receives the voice data.

[1613] Step 6:

[1614] The terminal receives voice input and converts the voice into text data using a voice recognition means.

[1615] (Input) Audio data

[1616] (Output) Text data (e.g., "Business Partner")

[1617] Specifically, the device uses a voice recognition engine to analyze the voice and convert it into text data.

[1618] Step 7:

[1619] The terminal analyzes the converted text data, identifies the corresponding group by means of a database management means, and assigns friends to the group.

[1620] (Input) Text data (e.g., "Business Partner")

[1621] (Output) Friend information assigned to the group

[1622] Specifically, the device matches the text "business partner" with existing groups in the database, and creates new groups as needed, assigning friend information.

[1623] Step 8:

[1624] The device analyzes the voice content using an emotion engine and obtains the user's emotional information.

[1625] (Input) Audio data

[1626] (Output) Emotional information

[1627] Specifically, the device passes the voice recognition results to the emotion engine, which analyzes the emotions and obtains the emotion information.

[1628] Step 9:

[1629] The terminal records the acquired emotion information in a database.

[1630] (Input) Emotion information

[1631] (Output) Emotion information recorded in the database

[1632] Specifically, the device associates the acquired emotion information with friend information and stores it in a database.

[1633] Step 10:

[1634] The device periodically scans your smartphone for contact data.

[1635] (Input) Contact Data

[1636] (Output) Scan results

[1637] Specifically, the device will automatically scan your contact data every night.

[1638] Step 11:

[1639] The device checks the contact data against the messaging application's database.

[1640] (Input) Contact data, messaging application data

[1641] (Output) Matching result

[1642] Specifically, the device compares the contact data with the data in the messaging application to find matching information.

[1643] Step 12:

[1644] The server checks for a matching friend and retrieves additional information from the contact data, such as occupation and company name.

[1645] (Input) Matching results, contact information

[1646] (Output) Additional information (occupation, company name)

[1647] Specifically, the server checks for matching friend information and retrieves additional information.

[1648] Step 13:

[1649] The server integrates the acquired additional information into the friend information of the messaging application.

[1650] (Input) Additional information (occupation, company name)

[1651] (Output) Integrated friend information

[1652] Specifically, the server combines the acquired information with friend information in the messaging application.

[1653] Step 14:

[1654] The device updates the database.

[1655] (Input) Integrated friend information

[1656] (Output) Updated database

[1657] Specifically, the terminal reflects information from the server and keeps the database up to date.

[1658] Step 15:

[1659] A user selects the "Manage Groups" option from the friends list menu of a messaging application.

[1660] (Input) User action (selection of "Group Management")

[1661] (Output) Group management screen

[1662] Specifically, the user opens the friend list menu and selects "Group Management."

[1663] Step 16:

[1664] View all groups and friend information that your device is currently registered to.

[1665] (Input) User action

[1666] (Output) Displayed group and friend information

[1667] Specifically, the device retrieves all group and friend information from the database and displays it on the screen.

[1668] Step 17:

[1669] The user taps on the displayed group and edits the attribute information of the friend.

[1670] (Input) User action (tap on group, edit)

[1671] (Output) Edited attribute information

[1672] Specifically, the user taps the group he or she wants to edit and edits the attribute information.

[1673] Step 18:

[1674] Users can add new attribute categories and delete unnecessary groups.

[1675] (Input) User action (add, delete)

[1676] (Output) Updated attribute categories

[1677] As a specific operation, the user adds new attributes and deletes unnecessary groups.

[1678] Step 19:

[1679] The device saves the user's edits in a database.

[1680] (Input) Edited attribute information

[1681] (Output) Saved database

[1682] Specifically, the terminal records the edited content of the user in a database and updates it to the latest version.

[1683] (Application example 2)

[1684] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1685] Modern industrial and production sites require smooth communication between workers and robots. In particular, it is important to improve production efficiency and worker satisfaction by providing appropriate support based on the worker's emotions and state. However, existing systems lack robots that can properly understand the worker's voice instructions and respond flexibly based on emotion recognition, so new technologies are needed to solve this problem.

[1686] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a voice input means for inputting friend attribute information by voice, a voice recognition means for converting the voice-input attribute information into text data, a means for registering friends using QR codes or ID search, a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to those groups, an emotion recognition means for analyzing the acquired voice content and acquiring user emotion information, and a database storage means for storing the acquired emotion information in association with the friend attribute information. This makes it possible to realize a factory robot that appropriately analyzes a worker's voice instructions and provides flexible and optimal support based on the emotion information.

[1687] "Friend attribute information" is information set by the user to classify friends, and indicates categories such as "business partner" and "customer."

[1688] The "voice input means" is a device or system that allows a user to input attribute information of a friend through voice.

[1689] "Speech recognition means" refers to a technology or system that converts speech input by a speech input means into text data.

[1690] "Methods for registering friends using QR codes or ID searches" refers to methods in which users scan QR codes or enter IDs to identify and register friends.

[1691] The "database management means" is a system that has the function of analyzing text data, identifying corresponding groups, and assigning friends to those groups.

[1692] "Emotion recognition means" refers to a technique or device for analyzing the content of a voice and acquiring information about the user's emotions.

[1693] The "database storage means" is a system that has the function of storing the obtained emotional information in association with the attribute information of the friends.

[1694] "Means for scanning telephone directory data and comparing it with a database" refers to a technique for periodically checking existing telephone directory data and comparing it with information in a database.

[1695] The "data linking means" is a system that has the function of obtaining matching friend information from the phone book and adding it to the friend information in the database.

[1696] The "classification means" is a technology that uses a database management means to sort the attribute information of friends into new groups.

[1697] The "means for setting the support level" is a system that has the function of analyzing the voice instructions of the worker and acquiring emotional information to provide appropriate support according to the worker's condition.

[1698] The present invention provides a system for efficiently managing and classifying friend information and work support information using voice input and emotion recognition of a worker. Specific embodiments for carrying out the present invention will be described below.

[1699] Hardware and software used

[1700] This system uses the following hardware and software:

[1701] Audio input microphone (e.g., a general audio input device)

[1702] Industrial robots (e.g. Universal Robots)

[1703] Speech recognition API (e.g., Google Cloud Speech-to-Text)

[1704] Emotion recognition engine (e.g., Affectiva's Emotion Recognition SDK)

[1705] Data management software (e.g., SQLite database)

[1706] System configuration

[1707] The system consists of the following main components:

[1708] 1. Voice input means: A microphone device that allows workers to input their friends' attribute information and work instructions through voice.

[1709] 2. Speech recognition means: A speech recognition API is used to convert voice-input information into text data.

[1710] 3. How to register friends: Register friends using QR codes or ID searches.

[1711] 4. Database management means: Analyze the text data, identify the corresponding groups, and assign friends to those groups.

[1712] 5. Emotion recognition means: Analyzes the user's emotions from the voice content and obtains emotional information.

[1713] 6. Database storage means: The acquired emotional information is stored in association with the friend's attribute information.

[1714] 7. Data integration method: Scan existing phone book data and add it to the database.

[1715] 8. Support level setting means: Set an appropriate support level based on voice instructions and emotional information.

[1716] System Operation

[1717] First, the worker inputs the friend's attribute information and work instructions using the voice input means. This voice data is converted into text data by the voice recognition means. This text data is analyzed by the database management means and classified into appropriate groups.

[1718] Next, the emotion recognition means analyzes the voice data to obtain the user's emotion information, which is then stored in the database by the database storage means in association with the friend's attribute information.

[1719] Furthermore, the data linking means periodically scans the telephone book data, and the matching friend information is acquired from the telephone book and added to the database.

[1720] Finally, the support level setting means analyzes the voice instructions and emotional information of the worker and sets an appropriate support level, allowing the industrial robot to provide appropriate support according to the worker's workload.

[1721] Specific examples

[1722] For example, a worker may give a voice command such as "I'm tired today, so please increase my support." This voice is first converted into text data by a voice recognition API, and then emotional information indicating "fatigue" is obtained by an emotion recognition engine. Based on this information, the database management means increases the level of support for the worker, and the industrial robot automatically strengthens its support.

[1723] Prompt Sentence Examples

[1724] "Please create a program that analyzes the worker's voice instructions, obtains emotional information using an emotion recognition engine, and sets an appropriate assistance level. Please also provide specific details about the voice recognition API and emotion recognition engine you will use, as well as how you will save the data."

[1725] As described above, according to the embodiment of the invention, it is possible to efficiently manage and automate friend information and task support information based on the user's voice input and emotion recognition.

[1726] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1727] Step 1:

[1728] Audio Input:

[1729] The user inputs voice into the microphone device. The voice input includes attribute information of friends and work instructions. This input data is captured by the microphone device as an analog voice signal.

[1730] Step 2:

[1731] Voice Recognition:

[1732] The device sends voice input data to a speech recognition API (e.g., Google Cloud Speech-to-Text), which converts the analog voice signal into text data. In this case, the input is voice data and the output is text data.

[1733] Step 3:

[1734] Text data analysis:

[1735] The device analyzes the text data received from the voice recognition API using a database management means. The input is text data, and the data content is analyzed to identify related groups and assign the corresponding friend information to those groups. The output is the friend information categorized into groups.

[1736] Step 4:

[1737] Add as friend:

[1738] When a user registers a friend using a QR code or ID search, the device accepts the QR code scan or ID input and retrieves the corresponding friend information. The input is the QR code information or ID information, and the output is the friend information.

[1739] Step 5:

[1740] Emotion recognition:

[1741] The device uses an emotion recognition engine (e.g., Affectiva SDK) to recognize the user's emotions from text data. The emotion recognition engine receives text data as input and outputs emotional information. In this case, the input is text data and the output is emotional information.

[1742] Step 6:

[1743] Emotional information storage:

[1744] The emotion information acquired by the device from the emotion recognition engine is stored in a database by a database storage means in association with the friend's attribute information. The input is the emotion information and the friend's attribute information, and the output is the updated database.

[1745] Step 7:

[1746] Scan and match phone book data:

[1747] The device periodically scans its existing contact list and matches it with the friend information in its database. The input is the contact list data, and the output is the matching friend information.

[1748] Step 8:

[1749] Data integration:

[1750] The server adds the matching friend information retrieved from the phone book to the friend information in the database. The input is the phone book data and the database information, and the output is the updated friend information.

[1751] Step 9:

[1752] Support Level Settings:

[1753] The server uses the results of analysis by an emotion recognition engine based on the voice instructions and emotional information to set an appropriate support level. The input is the voice instructions and emotional information, and the output is the set support level.

[1754] Step 10:

[1755] Industrial robot control:

[1756] The terminal issues instructions to the industrial robot to assist in the work according to the set assistance level. This operation is performed using a robot motion control program. The input is assistance level information, and the output is robot motion control.

[1757] The system of this embodiment enables efficient management and automation of friend information and work support information based on the user's voice input and emotion recognition.

[1758] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1759] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1760] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1761] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1762] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1763] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1764] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1765] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1766] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1767] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1768] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1769] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1770] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1771] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1772] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1773] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1774] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1775] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1776] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1777] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1778] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1779] The following is further disclosed regarding the above embodiment.

[1780] (Claim 1)

[1781] a voice input means for inputting attribute information of friends by voice;

[1782] a voice recognition means for converting voice-input attribute information into text data;

[1783] You can register friends using QR codes or ID searches,

[1784] a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to the groups;

[1785] A system including:

[1786] (Claim 2)

[1787] Scan your existing phone book data and compare it with the LINE Messenger database.

[1788] A data integration method to retrieve matching friend information from the phone book and add it to LINE Messenger's friend information,

[1789] The system of claim 1 further comprising:

[1790] (Claim 3)

[1791] A means for classifying attribute information of friends into new groups such as "business partners" and "customers" by the database management means;

[1792] A means for displaying classified groups and allowing users to edit, add, or delete attribute information;

[1793] The system of claim 1 further comprising:

[1794]

[1795] "Example 1"

[1796] (Claim 1)

[1797] a voice input means for inputting attribute information of friends by voice;

[1798] a voice recognition means for converting voice-input attribute information into text data;

[1799] A means for registering friends using identification data;

[1800] a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to the groups;

[1801] A data integration method that periodically scans existing contact data, collates the information, retrieves additional information, and updates the database;

[1802] A system including:

[1803] (Claim 2)

[1804] A means for creating new groups based on attribute information input by the user through voice and classifying friends into those groups;

[1805] A means for displaying classified group and friend information and allowing users to edit, add, or delete attribute information;

[1806] 10. The system of claim 1, comprising:

[1807] (Claim 3)

[1808] A means for users to manage their contact information in an integrated manner and efficiently update their friends' attribute information;

[1809] 10. The system of claim 1, comprising:

[1810] "Application Example 1"

[1811] (Claim 1)

[1812] a voice input means for inputting attribute information of friends by voice;

[1813] a voice recognition means for converting voice-input attribute information into text data;

[1814] You can register friends using QR codes or ID searches,

[1815] a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to the groups;

[1816] a customer management means for store employees to manage customer information;

[1817] A system including:

[1818] (Claim 2)

[1819] means for scanning existing communications data and matching it with a database of communications applications;

[1820] a data linking means for acquiring the matching friend information from the communication data and adding the information to the friend information of the communication application;

[1821] The system of claim 1 further comprising:

[1822] (Claim 3)

[1823] A means for classifying attribute information of friends into new groups such as "business partners" and "customers" by the database management means;

[1824] A means for displaying classified groups and allowing users to edit, add, or delete attribute information;

[1825] A means for categorizing customer information into categories such as "regular customers" and "new customers" through voice input;

[1826] The system of claim 1 further comprising:

[1827] "Example 2: Combining Emotion Engines"

[1828] (Claim 1)

[1829] A voice input means for acquiring attribute information of a friend input by voice;

[1830] a voice recognition means for converting voice-input attribute information into text data;

[1831] A means for registering friends using identification information;

[1832] a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to the groups;

[1833] emotion engine means for analyzing the emotion of the user and adding the acquired emotion information to the attribute information;

[1834] A system including:

[1835] (Claim 2)

[1836] a means for scanning existing contact data and matching it with the messaging application's database;

[1837] a data linking means for retrieving matching friend information from the contact data and adding the matching friend information to the friend information of the messaging application;

[1838] The system of claim 1 further comprising:

[1839] (Claim 3)

[1840] means for classifying attribute information of friends into new groups by the database management means;

[1841] A means for displaying classified groups and allowing users to edit, add, or delete attribute information;

[1842] The system of claim 1 further comprising:

[1843] "Application example 2 when combining emotion engines"

[1844] (Claim 1)

[1845] a voice input means for inputting attribute information of friends by voice;

[1846] a voice recognition means for converting voice-input attribute information into text data;

[1847] You can register friends using QR codes or ID searches,

[1848] a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to the groups;

[1849] emotion recognition means for analyzing the acquired voice content and acquiring emotion information of the user;

[1850] a database storage means for storing the obtained emotion information in association with attribute information of the friend;

[1851] A system including:

[1852] (Claim 2)

[1853] A means of scanning existing telephone directory data and matching it with a database;

[1854] A data linking means for acquiring matching friend information from the phone book and adding it to the friend information in the database;

[1855] The system of claim 1 further comprising:

[1856] (Claim 3)

[1857] A means for classifying attribute information of friends into new groups such as "business partners" and "customers" by the database management means;

[1858] A means for displaying classified groups and allowing users to edit, add, or delete attribute information;

[1859] A means for analyzing voice instructions from a worker and acquiring emotional information to set an appropriate support level;

[1860] The system of claim 1 further comprising: [Explanation of symbols]

[1861] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a voice input means for inputting attribute information of friends by voice; a voice recognition means for converting voice-input attribute information into text data; You can register friends using QR codes or ID searches, a database management means for analyzing the text data, identifying corresponding groups, and assigning friends to the groups; A system including:

2. Scan your existing phone book data and compare it with the LINE Messenger database. A data integration method to retrieve matching friend information from the phone book and add it to LINE Messenger's friend information, The system of claim 1 further comprising:

3. A means for classifying attribute information of friends into new groups such as "business partners" and "customers" by the database management means; A means for displaying classified groups and allowing users to edit, add, or delete attribute information; The system of claim 1 further comprising:

Images