System

Abstract

A system is provided.SOLUTION: The system includes a means for receiving input data of a user, a means for constructing a sign language database of sign language and storing additional information related to each sign language, a means for retrieving corresponding sign language information from the sign language database based on the input data of the user, and a means for visually displaying retrieval results to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Sign languages ​​vary by region and generation, making it difficult to understand and learn them in a unified way. Furthermore, while it is effective to know the background and meaning of sign languages ​​in order to develop an interest in them, there is a lack of means to provide this information. Given these circumstances, there is a need for a system that makes it easier to understand the diversity of sign languages ​​and fosters a sense of familiarity with them. [Means for solving the problem]

[0005] The present invention provides a system that receives user input data, searches a sign language database for appropriate sign language information and its additional information based on that data, and visually displays it to the user. Specifically, the system collects information about the user's region and age and uses it to search for sign languages. By also displaying additional information about the meaning and origin of each sign, the system allows the user to intuitively understand the diversity of sign languages ​​and stimulates interest in sign languages.

[0006] A "sign language database" is a database that systematically stores information about sign language and allows it to be searched and retrieved as needed.

[0007] "Additional information" refers not only to information about the sign language itself, but also to detailed information such as the "origin" and "meaning" of the sign language, the "region" in which it originated, and the "era."

[0008] "User input data" refers to information entered by a user into the system, specifically search keywords and sign language-related questions.

[0009] A "visual display" is a means for displaying search results in a user interface, including visual elements such as text, images, and video.

[0010] "Region information" refers to information about the geographic location of a user, which allows regional variations of sign language to be identified.

[0011] "Era information" refers to information about the era to which the user belongs, which allows for identifying variations in sign language for each era.

[0012] "Search results" refer to sign languages ​​and their additional information obtained from a sign language database based on user input data.

[0013] "Sign language information" is information about the gestures and actions of a particular sign language, showing how that sign language is performed.

[0014] "User" refers to a person who uses the system to search for and view sign language information. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] MODE FOR CARRYING OUT THE INVENTION

[0037] The present invention relates to a system that provides "cues" and "meanings" related to sign language to help users understand sign language. This system provides a means for efficiently learning sign languages ​​that vary by region and generation. Specific embodiments of the present invention are described below.

[0038] Overall system configuration

[0039] This system mainly consists of the following three main components:

[0040] 1. A server that manages the sign language database and stores additional information on sign languages ​​and their "cues" and "meanings."

[0041] 2. A terminal that accepts user input, sends requests to a server, and displays the results.

[0042] 3. Users who search for, learn, and use information about sign language.

[0043] Sign Language Database

[0044] server:

[0045] A sign language database systematically stores multiple pieces of information about sign languages. This includes not only information about the gestures and actions of sign languages, but also additional information such as the background, region, and era in which the sign language originated. The server manages this information and returns appropriate data in response to user search requests.

[0046] Handling User Input

[0047] User:

[0048] Users input information about a particular sign language through their device. For example, to find out about the sign for "thank you," they type "thank you" into the search bar. Information about the user's region and age is also collected. This information is important for identifying regional and age-specific variations in sign language.

[0049] Submitting a search request

[0050] Device:

[0051] The device receives the user's input data and sends a request for sign language information to the server based on that data. The request includes not only the keywords entered by the user, but also regional and historical information. This data is used to improve the accuracy of the search.

[0052] Database search and result formatting

[0053] server:

[0054] The server receives a user request and searches for the corresponding sign language information from a sign language database. Specifically, it retrieves each sign language variation and its additional information based on the input keywords. The search results are then formatted appropriately before being returned to the user. This formatted data includes details such as the sign gesture, the region where it originated, the era, and its meaning.

[0055] Displaying search results

[0056] Device:

[0057] The device analyzes the search results received from the server and displays them on the user interface. The display includes information about specific sign language gestures and their "cues" and "meanings," making it easier for users to understand the background and meaning of the sign language.

[0058] Specific examples

[0059] As a specific example, consider the case where a user searches for the sign language for "thank you." The user enters "thank you" in the search bar and presses the search button. At this time, the device sends the user's input data, along with region information (e.g., Tokyo) and age information (e.g., 30-40 years old) to the server.

[0060] The server searches the database for sign language information related to "thank you," searching for different variations such as "in the Kansai region, the thumb and index finger form a circle" and "in Tokyo, the palms are placed together," and obtaining the meaning and background information for each.

[0061] The search results are formatted and sent back to the device, where they are displayed on the user interface. Users can visually check the sign language and its meanings specific to each region and era. This helps users understand the diversity of sign language and stimulates their interest in sign language.

[0062] The embodiment of the present invention has been specifically described above. This system makes it easier to understand the diversity of sign languages ​​and makes people feel closer to sign languages.

[0063] The processing flow will be explained below.

[0064] Program processing steps

[0065] Step 1: Receiving User Input

[0066] User:

[0067] If you want to find information about sign language, enter a keyword in the search bar, for example, "thank you."

[0068] Step 2: Collect user information

[0069] Device:

[0070] It receives user input data (keywords) and also collects the user's location information (e.g., identifying "Tokyo" using GPS) and age information (e.g., identifying "30-40 years old" from the profile).

[0071] Step 3: Create and submit a request

[0072] Device:

[0073] The collected user input data, region information, and age information are sent to the server as a request. The request format is as follows: { keyword: 'Thank you', region: 'Tokyo', age_group: '30-40'}.

[0074] Step 4: Receiving and Parsing the Request

[0075] server:

[0076] The received request data is analyzed to identify the sign language the user wants to search for and its related information.

[0077] Step 5: Retrieving information from the database

[0078] server:

[0079] The system searches the sign language database for sign language information and additional information that matches the keywords in the request. Specifically, it searches for sign language variations related to "thank you" and their respective "trigger" and "meaning."

[0080] Step 6: Formatting the search results

[0081] server:

[0082] The search results are formatted and converted into a format to be returned to the user. For example, in JSON format, they are formatted as "{ sign: 'Thank you', variations: [{ region: 'Kansai', gesture: 'Make a circle with your thumb and index finger', meaning: 'Convey gratitude'}, { region: 'Tokyo', gesture: 'Put your palms together', meaning: 'Meaning of gratitude'}]}".

[0083] Step 7: Submit search results

[0084] server:

[0085] The formatted search results are returned to the terminal. The returned data is in a format that has already been formatted.

[0086] Step 8: Parse and display search results

[0087] Device:

[0088] The search results received from the server are analyzed and displayed on the user interface, visually displaying specific sign language gestures and information about their "cues" and "meanings."

[0089] Step 9: Verify the user and request additional information

[0090] User:

[0091] You can review the displayed sign language information and request more detailed information if necessary. For example, if you want to learn more about the development and historical background of the sign language, click the button for more information.

[0092] The above are the specific program processing steps in the embodiment of the present invention, which provide users with information about the diversity of sign languages ​​and their background information, thereby increasing their understanding and interest in sign languages.

[0093] Example 1

[0094] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0095] While there are many systems to promote the learning and understanding of sign language, few systems efficiently and accurately provide the different sign language variations by region and age. Furthermore, conventional systems have difficulty providing detailed background information and meanings of sign language. Therefore, there is a need for a system that allows users to easily obtain sign language information according to specific conditions and understand its meaning and background.

[0096] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0097] In this invention, the server includes means for receiving a user's search keywords and regional and era information, means for managing a database that systematically stores various information related to sign language, means for searching the database for information such as gestures, background, region, era, and meaning of the relevant sign language based on data entered by the user, means for formatting the search results in an appropriate format such as JSON and returning the results, and means for analyzing the search results received from the server and displaying them on a user interface. This enables a user to efficiently and accurately obtain sign language information tailored to a specific region or era and understand its background and meaning.

[0098] 1. "Means" refers to methods or devices used to achieve a particular purpose.

[0099] 2. "Search Keywords" refers to words or phrases that users enter to obtain specific sign language information.

[0100] 3. "Region information" refers to geographic information where a user lives or is associated, and is used to identify sign language variations.

[0101] 4. "Date information" refers to the age of a user or temporal information necessary to identify a sign language variation of a particular era.

[0102] 5. "Database" refers to a collection of information for the systematic storage and management of information related to sign languages.

[0103] 6. "Gesture" refers to the specific movements of the hands and body when signing.

[0104] 7. "Context" refers to information about how a particular sign language originated and what its cultural and historical background is.

[0105] 8. "Meaning" refers to the content or intent that a particular sign language is intended to convey to the user.

[0106] 9. "JSON format" refers to a lightweight data exchange format for representing data in a structured manner.

[0107] 10. "User interface" refers to the display and input devices that allow a user to directly interact with a system.

[0108] 11. "Parsing" refers to the process of understanding data received from the server and converting it into a user-friendly format.

[0109] MODE FOR CARRYING OUT THE INVENTION

[0110] The present invention relates to a system that allows users to learn sign language efficiently and accurately. To implement this system, a server that manages a sign language database and provides information based on user search requests, a terminal for users to perform searches, and users who actually learn sign language are required.

[0111] Server Configuration

[0112] The server has a database that systematically stores and manages various information related to sign language. For example, a relational database such as MySQL (registered trademark) can be used. The database includes information such as sign language gestures, background, region, era, and meaning. Based on a search request received from a user, the server searches for the relevant sign language information, formats it in an appropriate format (e.g., JSON format), and returns it.

[0113] Device configuration

[0114] The terminal provides an interface for the user to input search keywords, region information, and era information. For example, a PC, smartphone, tablet, or other device can be used. The terminal receives the data entered by the user and sends it to the server in the form of an HTTP POST request, for example. The terminal analyzes the data returned from the server and displays it on the user interface.

[0115] User operations

[0116] Users enter keywords related to a specific sign language into the device's search bar. For example, if they want to find the sign for "thank you," they enter this keyword. In addition, the system also collects information about the user's region and age. For example, "Tokyo" or "30-40 years old." This information is important for improving search accuracy.

[0117] Specific example explanation

[0118] As a specific example of use, consider the case where a user searches for the sign for "thank you."

[0119] The user enters "Thank you" in the search bar of the device and presses the search button. The device sends the user's input data, region information (e.g., Tokyo), and age information (e.g., 30-40 years old) to the server.

[0120] The server searches the sign language database for sign language information related to "thank you." For example, different variations such as "in the Kansai region, the thumb and index finger form a circle" and "in Tokyo, the palms are placed together" are searched for, and the meanings and background information of each are also obtained. These search results are then formatted and sent back to the device.

[0121] The device analyzes the search results and displays them on the user interface. Users can visually check the sign language and its meanings specific to each region and era, which helps them understand the diversity of sign language and stimulates their interest in sign language.

[0122] Prompt Sentence Examples

[0123] Below are some examples of specific prompt sentences.

[0124] If a user wants to find out about the sign language for "thank you," they type "thank you" into the search bar and press the search button. The device sends the user's input data, along with region information (e.g., Tokyo) and age information (e.g., 30-40 years old), to the server. The server searches the sign language database for information related to "thank you," and obtains different variations, such as a circular thumb and index finger in the Kansai region and a palm-to-palm gesture in Tokyo. The server also provides information about the meaning and background of each variation. The formatted search results are sent back to the device and displayed on the user interface.

[0125] The above describes a specific embodiment of the present invention. This system allows users to efficiently learn the diversity of sign languages ​​and gain a deeper understanding of their backgrounds and meanings.

[0126] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0127] Specific flow of program processing

[0128] Step 1:

[0129] The user enters search keywords into the device

[0130] The user enters a specific sign language keyword into the device's search bar, for example, "thank you," along with region information (e.g., Tokyo) and age information (e.g., 30-40 years old).

[0131] Input: Search keyword "Thank you", region information "Tokyo", age information "30-40 years old"

[0132] Output: User input data

[0133] Step 2:

[0134] The device receives user input data and sends it to the server

[0135] The terminal creates an HTTP POST request based on the received user input data and sends it to a specific API endpoint on the server.

[0136] Input: User-entered data

[0137] Data transformation: Converting user-entered data into an HTTP POST request

[0138] Output: Request sent to server

[0139] Step 3:

[0140] The server receives the request and performs a database lookup.

[0141] The server receives the request from the device and queries the sign language database based on the keyword "thank you" to retrieve relevant information. The query searches for information such as the sign gesture, background, region, era, and meaning.

[0142] Input: HTTP POST request (search keyword "Thank you", region information "Tokyo", age information "30-40 years old")

[0143] Data manipulation: performing database queries

[0144] Output: Search result data (sign language information)

[0145] Step 4:

[0146] The server formats the search results and returns them to the device

[0147] The server formats the search results into JSON format, converts them into the appropriate format, and returns them to the terminal.

[0148] Input: Search result data

[0149] Data processing: Conversion to JSON format

[0150] Output: Formatted search results data

[0151] Step 5:

[0152] The device receives the search results and displays them on the user interface.

[0153] The device receives the formatted data returned from the server, analyzes it, and displays it on the user interface, allowing the user to visually check the various variations of sign language and their meanings and backgrounds.

[0154] Input: Formatted search result data

[0155] Data Calculation: Data Analysis

[0156] Output: User interface display of search results

[0157] The processing steps of the system have been described above in detail.

[0158] (Application example 1)

[0159] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0160] Learning sign language and using it efficiently in specific tasks is difficult because the background and meaning of sign language vary by region and age. Additionally, certain tasks, such as food delivery, require the accurate use of sign language when communicating with hearing-impaired customers, but there are a lack of ways for employees to quickly learn that sign language. To solve this issue, a system is needed that allows employees to efficiently learn and use sign language related to specific tasks in a timely manner.

[0161] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0162] In this invention, the server includes means for receiving user input data, means for constructing a sign language database and storing additional information related to each sign language, means for searching the sign language database for corresponding sign language information based on the user input data, means for visually displaying the search results to the user, and means for enabling efficient learning and use of sign languages ​​in specific tasks, thereby enabling accurate learning and rapid use of sign languages.

[0163] A "means for receiving user input" is the interface a user uses to provide information about a particular sign language, including speech, keyboard input, touch, etc.

[0164] "Means for building a database of sign languages ​​and storing additional information related to each sign language" refers to a system or server for systematically storing and managing detailed information such as sign language gestures, regional information, historical information, and their background and meanings.

[0165] "Means for searching for corresponding sign language information from a sign language database based on user input data" refers to algorithms or processes for efficiently searching and extracting relevant sign language information from a sign language database based on keywords, regional information, and era information provided by the user.

[0166] "Means for visually displaying search results to the user" refers to a graphical user interface or display device that displays the searched sign language information in a format that is easy for the user to understand.

[0167] "Means to enable efficient learning and use of sign language in specific jobs" refers to training modules and guidelines to support employees in specific jobs, such as food delivery, in quickly learning the necessary sign language and using it effectively in their work.

[0168] "When an employee performing a specific task uses sign language, collect regional and age information and use it to search for sign language information" refers to the process of inputting regional and age information and selecting the most appropriate sign language based on that information in order to provide the appropriate sign language according to the context when the employee uses sign language.

[0169] "Additional information related to the meaning or trigger of sign language" is supplementary information for understanding sign language, such as the situations in which a particular sign language is used, its background and history, etc.

[0170] This invention provides a system that enables employees of food delivery services to easily learn and use sign language. This system mainly consists of the following components:

[0171] Overall system configuration

[0172] This system mainly consists of the following five main components:

[0173] 1. A means of receiving user input data

[0174] 2. A means of building a database of sign languages ​​and storing additional information associated with each sign language.

[0175] 3. A means to search for corresponding sign language information from a sign language database based on user input data

[0176] 4. A means of visually displaying search results to the user

[0177] 5. Means of enabling people to learn and use sign language efficiently in specific jobs

[0178] Handling User Input

[0179] Device:

[0180] Users use their smartphones to input information about a specific sign language. For example, if a delivery person wants to know the sign for "thank you," they can enter "thank you" into the application's search bar. At the same time, the user's region and age information are also collected. This information is important for identifying regional and age-specific variations in sign language.

[0181] Submitting a search request

[0182] Device:

[0183] The device sends the user's input data, along with regional and historical information, to the server, which is used to improve search accuracy and identify sign language variations.

[0184] Database search and result formatting

[0185] server:

[0186] The server receives a request from a user and searches for the corresponding sign language information from a sign language database. Specifically, it retrieves each sign language variation and its additional information based on the input keyword. For example, for the sign expression for "thank you," it searches for sign language variations for the Tokyo area and the 30-40 age group, and also retrieves the meaning and background information of each.

[0187] Displaying search results

[0188] Device:

[0189] The terminal analyzes the search results received from the server and displays them on the user interface. The display includes information about the specific sign language gestures and their "cues" and "meanings," making it easier for employees to understand the background and meaning of the sign language.

[0190] Specific examples

[0191] As a specific use case, consider a food delivery person who wants to learn the sign language for "thank you." The delivery person enters "thank you" into the application's search bar and presses the search button. At this time, the device sends the user's input data, along with region information (e.g., Tokyo) and age information (e.g., 30-40 years old) to the server. The server then searches the database for sign language information related to "thank you" and obtains detailed variations in sign language for different regions and age groups. The search results are formatted and sent back to the device, where the delivery person can view this information on the user interface. This allows the delivery person to understand the sign language and its meaning specific to their region and age group, and use sign language appropriately in their actual delivery work.

[0192] Prompt Sentence Examples

[0193] An example of a prompt to input to a generative AI model is as follows:

[0194] "We will create a food delivery application program for the hearing impaired. It will run on smartphones and have a function that allows users to learn sign language efficiently. When a user enters keywords for the sign language for "thank you" and provides region and era information, the corresponding sign language instructions will be displayed. Specifically, search results will be displayed that include detailed information such as sign language gestures, region, era, and meaning. Please use the following data processing flow. Input data: Keywords, region, era. Output data: Sign language gestures, region, era, meaning."

[0195] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0196] Step 1:

[0197] The user operates the delivery person app and inputs information about sign language. For example, if the delivery person wants to know the sign language for "thank you," they enter "thank you" in the application's search bar and press the search button. At this point, region information (e.g., Tokyo) and age information (e.g., 30-40 years old) are also entered at the same time. The input data is the keyword, region, and age. The output data is the information entered by the user, and the device retains this as input.

[0198] Step 2:

[0199] The device collects the user's input data and sends it to the server. The input data includes keywords, region, and age information. Based on this, the device sends a request to the server. This request includes the keywords entered by the user, such as "Thank you," as well as region and age information. The output data is the search request received by the server.

[0200] Step 3:

[0201] The server searches the sign language database based on the received search request. Specifically, it searches for entries in the database that match keywords such as "thank you," and also searches for associated regional and historical information. The input data is the search request, and the output data is the search results. The server processes this using a database search algorithm to obtain the corresponding sign language information and its additional information.

[0202] Step 4:

[0203] The server formats the acquired sign language information, prepares it in a format that can be sent, and sends it back to the user's device. The search results include detailed information such as the sign language gesture, region, era, and meaning. The input data are the search results, and the output data are the formatted search results. The server uses a data formatting algorithm to format the results and send them to the device.

[0204] Step 5:

[0205] The terminal analyzes the received search results and displays them visually. Specifically, it displays specific sign language gestures and information about their "cues" and "meanings" on the user interface. This makes it easier for users to understand the background and meaning of the sign language. The input data are the formatted search results, and the output data is the content displayed to the user. The terminal's analysis and display processing allows the delivery person to understand and use sign language accurately.

[0206] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0207] MODE FOR CARRYING OUT THE INVENTION

[0208] The present invention relates to a system for aiding in the understanding of sign language and fostering familiarity with sign language. In particular, the system provides a more personalized learning experience by incorporating an emotion engine that recognizes a user's emotions and dynamically adjusts the method and content of providing sign language information. Specific embodiments of the present invention are described below.

[0209] Overall system configuration

[0210] This system mainly consists of the following four main components:

[0211] 1. A device that incorporates an emotion engine to receive user input data and recognize emotions.

[0212] 2. A server that manages the sign language database and stores the sign language and additional information such as its "trigger" and "meaning."

[0213] 3. Users who search for, learn, and use information about sign language.

[0214] 4. A means for providing sign language information based on the analysis results of the emotion engine.

[0215] Sign Language Database

[0216] server:

[0217] The server manages a sign language database and stores detailed information about sign languages, including information about the gestures and actions of the sign language, as well as additional information such as the background, region, and era in which the sign language originated. The server efficiently searches and provides this information.

[0218] Emotion Engine

[0219] Device:

[0220] The emotion engine is a software module for analyzing the user's emotions in real time. It recognizes the user's emotional state by analyzing the user's facial expressions and tone of voice through a camera and microphone. Based on this information, the system provides sign language information that matches the user's current emotion.

[0221] Handling User Input

[0222] User:

[0223] The user inputs information about a specific sign language through the device. For example, to find out about the sign for "thank you," the user enters "thank you" into the search bar. The emotion engine then detects the user's facial expressions and voice and analyzes their emotional state.

[0224] Submitting a search request

[0225] Device:

[0226] The device sends a request to the server containing the user's input data and the emotion information recognized by the emotion engine, including information about the region and age.

[0227] Database search and result formatting

[0228] server:

[0229] The server analyzes the received request and searches for the corresponding sign language information from the sign language database. Taking into account the results of the emotion engine, the server adjusts the priority of the sign language information appropriate to the user's emotion. For example, if the user is relaxed, it provides more detailed information, but if the user is feeling impatient, it prioritizes concise information.

[0230] Displaying search results

[0231] Device:

[0232] The device analyzes the search results received from the server and adjusts the display content based on the emotional information obtained from the emotion engine. For example, it displays messages emphasizing success to users who show positive emotions, and adds encouraging messages to users who show negative emotions.

[0233] Specific examples

[0234] Consider a specific use case. When a user searches for the sign language for "thank you," they enter "thank you" in the search bar and press the search button. At this time, the emotion engine analyzes the user's facial expressions and tone of voice and recognizes that the user is relaxed.

[0235] The device sends a request containing the user's input data, region information, age information, and emotion information to the server. The server searches for sign language information corresponding to "thank you" and, taking into account the results of the emotion engine, provides detailed sign language information that is appropriate for when the user is relaxed. It also adds encouraging and positive feedback as support messages as needed.

[0236] The device analyzes the received search results and displays them on the user interface, visually displaying specific sign gestures and their "cues" and "meanings." This helps users understand the background and meaning of sign language, and by providing emotional feedback during the learning process, it increases interest in and familiarity with sign language.

[0237] The above is a specific description of an embodiment of the present invention. This system provides the diversity of sign languages ​​and their background information, and also provides a personalized sign language learning experience that responds to the user's emotions.

[0238] The processing flow will be explained below.

[0239] MODE FOR CARRYING OUT THE INVENTION

[0240] The processing flow of the system based on the present invention will be specifically explained by dividing it into the following steps.

[0241] Step 1: Receiving User Input

[0242] User:

[0243] If you want to find information about sign language, enter a keyword in the search bar, for example, "thank you."

[0244] Step 2: Emotion analysis using the emotion engine

[0245] Device:

[0246] The emotion engine is activated and analyzes the user's facial expressions and voice via the camera and microphone, recognizing their emotional state in real time, such as whether they are relaxed or anxious.

[0247] Step 3: Collect user information

[0248] Device:

[0249] Collects user input data (keywords), emotional information, regional information (e.g., identifying "Tokyo" using GPS), and age information (e.g., identifying "30-40 years old" from the profile).

[0250] Step 4: Create and submit a request

[0251] Device:

[0252] The request data is created based on the collected information and sent to the server. The request contains the following data: { keyword: 'Thank you', region: 'Tokyo', age_group: '30-40', emotion: 'Relax'}.

[0253] Step 5: Receiving and Parsing the Request

[0254] server:

[0255] The received request data is analyzed to identify the sign language the user wants to search for and related information. Emotional information is also used in the analysis.

[0256] Step 6: Retrieving information from the database

[0257] server:

[0258] Searches for sign language information and additional information that matches the entered keywords from a sign language database. For example, sign language variations related to "thank you" and their "trigger" and "meaning."

[0259] Step 7: Formatting the search results

[0260] server:

[0261] It shapes search results, adjusting the level of detail and content of the information provided based on the user's emotional state, for example prioritizing detailed information for a relaxed user and concise information for a rushed user.

[0262] Step 8: Submit search results

[0263] server:

[0264] The formatted search results are sent back to the device. The returned data can be in the following format: "{ sign: 'Thank you', variations: [{ region: 'Kansai', gesture: 'Circle your thumb and index finger', meaning: 'Expresses gratitude'}, { region: 'Tokyo', gesture: 'Put your palms together', meaning: 'Means thank you'}], message: 'You are relaxed. Show more information.'}".

[0265] Step 9: Parse and display search results

[0266] Device:

[0267] The search results received from the server are analyzed and displayed on the user interface. At this time, appropriate encouragement and feedback messages are also displayed based on the analysis results of the emotion engine. For example, a message such as "You are relaxed. Please check for more information" may be displayed.

[0268] Step 10: Verify the user and request additional information

[0269] User:

[0270] View the displayed sign language information, its supplementary information, and emotion-based feedback. If necessary, you can request more information. For example, if you want to learn more about the historical background of the sign language, you can click the button for more information.

[0271] The above are the specific program processing steps of an embodiment of the present invention, which combines an emotion engine that recognizes the user's emotions. This system provides the diversity of sign languages ​​and their background information, enabling a personalized sign language learning experience that responds to the user's emotions.

[0272] Example 2

[0273] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0274] Conventional systems for supporting sign language learning rely on user input data and basic search functions, and are unable to provide a personalized learning experience that takes into account the user's emotional state. This makes it difficult to elicit the user's interest and affinity and promote effective learning. Furthermore, optimization of search and provided information that takes into account the user's region and age is also insufficient. To solve this problem, a system is needed that analyzes the user's emotional state in real time and provides optimal sign language information based on that analysis.

[0275] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving input data and emotional state of a user, means for constructing a sign language database and storing additional information related to each sign language, means for searching for corresponding sign language information from the sign language database based on the user's input data and emotional information, and means for displaying the search results and the emotional information in a visually and emotionally adjusted form to the user. This makes it possible to provide optimal sign language information according to the user's emotional state, thereby realizing a personalized learning experience. Furthermore, by taking into account regional and era information, it is possible to provide more accurate information.

[0276] "User-entered data" refers to information entered by a user using a terminal, such as keywords and phrases related to sign language searches.

[0277] "Emotional state" refers to a psychological state analyzed from the user's facial expression, tone of voice, etc., and is classified into categories such as relaxation, impatience, and joy.

[0278] A "sign language database" refers to a database that constructs and stores data including sign language gestures, meanings, background information, etc.

[0279] "Additional information" is detailed information related to a sign language, including additional information such as the background, region, and era in which the sign language originated.

[0280] The "search means" refers to a function that searches for corresponding sign language information from a sign language database based on the user's input data and emotion information.

[0281] "Visual display" refers to a method of visually presenting sign language information to a user on a screen, including sign language gestures and additional information.

[0282] "Emotional adjustment" refers to a method of adjusting the priority of information based on the user's emotional state and providing the information to the user in an optimal manner.

[0283] "Region information" is information about the user's current location, and is used for filtering when searching for sign language information.

[0284] "Age information" is information about the user's age and birth decade, and is used for filtering when searching for sign language.

[0285] This invention is a system for providing a personalized experience for users learning sign language. The system has the ability to analyze the user's emotional state in real time and dynamically provide sign language information based on that analysis.

[0286] Overall system configuration

[0287] 1. Receiving user input data

[0288] User: The user enters the sign language keyword they want to search for in the search bar of their device. For example, if they want to learn the sign language for "thank you," they enter "thank you" in the search bar and click the search button.

[0289] 2. Acquiring emotional information

[0290] Device: The device uses a built-in camera and microphone to capture the user's facial expressions and tone of voice, which are then analyzed by an emotion engine. This data is then classified as the user's emotional state (e.g., relaxed, anxious, happy, etc.).

[0291] 3. Generating a search request

[0292] Device: The device generates a search request that includes the keywords and emotion information entered by the user, as well as location and age information. The generated request is sent to the server in the next step.

[0293] 4. Sending a request to the server

[0294] Device: The device generates a search request and sends it to the server, using an internet connection to transfer data using HTTP or HTTPS protocols.

[0295] 5. Database Search

[0296] Server: The server analyzes the received request and searches for the corresponding sign language information from a sign language database, which contains information about the gestures, meanings, and context of the sign language. The search results are prioritized based on the emotional information.

[0297] 6. Formatting Sign Language Information

[0298] Server: The server formats the search results and generates a response in a user-friendly format. If the user is relaxed, the response will contain more information.

[0299] 7. Receiving and Displaying Search Results

[0300] Device: The device analyzes the search results received from the server and displays them in a visually and emotionally tailored way to the user's emotions. For users who show positive emotions, it displays messages that highlight success, and for users who show negative emotions, it adds encouraging messages.

[0301] Specific use cases

[0302] As a specific example, consider the case where a user wants to search for the sign language for "thank you." The user types "thank you" into the search bar and presses the search button. At this time, the device's emotion engine analyzes the user's facial expressions and voice and recognizes that the user is relaxed.

[0303] The device generates a request including the input data "Thank you," region information, age information, and emotional information "Relax," and sends it to the server. The server searches for the corresponding sign language information in a sign language database, prioritizes providing detailed information, and also generates a response including positive feedback.

[0304] The device displays the received search results in a user interface. For example, not only are the sign gestures, background information, and meanings visually displayed, but detailed information such as "This sign means thank you and is used in these situations" is also provided. Furthermore, because the user is relaxed, a positive feedback message is also displayed.

[0305] Example of input prompt for generative AI model

[0306] "Suppose a user wants to know the sign for 'thank you'. You have emotional information that the user is relaxed. In this case, provide detailed information about the sign, including the gesture, background information, and geographic and historical information."

[0307] In this way, a sign language learning support system that reflects emotional information can provide users with a personalized learning experience.

[0308] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0309] Step 1: Receiving User Input

[0310] User:

[0311] The user enters the sign language keyword they want to search for in the search bar of their device and presses the search button. For example, if they want to search for the sign language for "thank you," they enter "thank you" in the search bar.

[0312] Input: Keyboard input ("Thank you")

[0313] Output: Keyword data ("Thank you")

[0314] Specific behavior:

[0315] The user enters keywords into the device's search bar and clicks the "Search" button.

[0316] Step 2: Acquiring emotional information

[0317] Device:

[0318] The device uses a built-in camera and microphone to capture the user's facial expressions and tone of voice, which are then analyzed by an emotion engine.

[0319] Input: Camera video data, microphone audio data

[0320] Output: Emotion data (e.g. "Relaxed")

[0321] Specific behavior:

[0322] The camera captures the user's face and the microphone records the user's voice.

[0323] An emotion engine analyzes this data to determine the user's emotional state.

[0324] Step 3: Generate a search request

[0325] Device:

[0326] The terminal generates a search request by combining the keywords, emotional information, and regional and age information entered by the user.

[0327] Input: Keyword data ("Thank you"), emotion data ("Relax"), region information, age information

[0328] Output: Search request data

[0329] Specific behavior:

[0330] Based on the input data, the data is formatted into a search request format.

[0331] Create request data including region and age information.

[0332] Step 4: Sending a request to the server

[0333] Device:

[0334] The terminal transmits the generated search request to the server.

[0335] Input: Search request data

[0336] Output: HTTP request to the server

[0337] Specific behavior:

[0338] Using the HTTP protocol, a request is sent to the specified URL on the server using the POST method.

[0339] Step 5: Database Search

[0340] server:

[0341] The server analyzes the received request and searches for the corresponding sign language information from the sign language database.

[0342] Input: Search request data

[0343] Output: Sign language information data

[0344] Specific behavior:

[0345] A database query is generated and searched against a sign language database.

[0346] Take search results and adjust priorities based on sentiment information.

[0347] Step 6: Formatting the sign language information

[0348] server:

[0349] The server formats the search results and prepares the sign language information in a user-friendly format.

[0350] Input: Sign language information data

[0351] Output: Formatted display data

[0352] Specific behavior:

[0353] Format sign language gestures, meanings and related information.

[0354] Add positive feedback messages based on emotions.

[0355] Step 7: Receiving and displaying search results

[0356] Device:

[0357] The terminal analyzes the search results received from the server and generates content to be displayed to the user.

[0358] Input: Formatted display data

[0359] Output: User interface display

[0360] Specific behavior:

[0361] The data received from the server is analyzed and sign language gestures and background information are displayed on the user interface.

[0362] It also displays emotion-based feedback messages.

[0363] (Application example 2)

[0364] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0365] There is a need to improve and streamline communication within factories, but there are challenges such as difficulty in communicating smoothly with hearing-impaired employees and responding to employees' emotions in stressful environments. In particular, when communicating using sign language, it is difficult to provide information that corresponds to the emotional state of employees, and an effective system for this purpose is needed.

[0366] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0367] In this invention, the server includes means for receiving user input data, means for building a sign language database and storing additional information related to each sign language, emotion recognition means using an emotion engine that analyzes the user's emotions in real time, means for searching the sign language database for corresponding sign language information based on the user's input data and emotion information, and means for visually displaying the search results in a form that corresponds to the user's emotional state. This enables smooth communication using sign language in factory communications and personalized support that corresponds to the emotions of employees.

[0368] The "means for receiving user input data" is a hardware or software mechanism for obtaining information provided by a user.

[0369] "Means for building a sign language database and storing additional information related to each sign language" refers to a method for comprehensively organizing and storing information about sign languages, as well as their background and meanings.

[0370] The "emotion recognition means" is a mechanism for analyzing the user's emotional state in real time and obtaining the results.

[0371] The "means for searching for corresponding sign language information from a sign language database" is a method for searching for appropriate sign language information from a sign language database based on the user's input data and emotion information.

[0372] "Visual display means" refers to the method of presenting search results to the user through a visual interface such as a display or screen.

[0373] "Means for collecting regional and generational information" refers to methods for acquiring and accumulating data relating to the geographical location and generation of users.

[0374] The "means for dynamically personalizing and providing additional information" is a mechanism for individually optimizing and providing additional information about sign language based on the user's emotional state and other factors.

[0375] The present invention provides a sign language compatible robot assistant system for realizing efficient communication within a factory. Specific embodiments are described below.

[0376] Overall system configuration

[0377] The system mainly consists of the following main components:

[0378] 1. Hardware and software for receiving user input data

[0379] 2. A server that builds a sign language database and stores additional information related to each sign language.

[0380] 3. A device that uses an emotion engine to analyze user emotions in real time

[0381] 4. A device that visually displays search results in a format that corresponds to the user's emotional state

[0382] Receiving user input data

[0383] User input data is collected by a device equipped with a camera and microphone, with the Python OpenCV library installed to capture video frames and collect audio data.

[0384] Sign Language Database

[0385] The server manages a sign language database and stores detailed information about sign languages. This includes information about the gestures and actions of the sign language, as well as additional information such as the background, region, and era in which the sign language originated. The server uses the Python sign_language_database module to efficiently search and provide this information.

[0386] Emotion Engine

[0387] The emotion engine is a software module for analyzing the user's emotions in real time. This module uses a library called EmotionRecognition to analyze the user's facial expressions and tone of voice. The level of detail of the information provided changes depending on whether the user is feeling stressed or not.

[0388] Processing user input data and emotional information

[0389] The device sends the user's input data and the emotional information recognized by the emotion engine together to the server. The server analyzes this request and searches the database for corresponding sign language information. The search results are returned by the server with either a detailed explanation or a concise explanation selected based on the user's emotional state.

[0390] Displaying search results

[0391] The device analyzes the search results received from the server and adjusts the display content based on the emotional information obtained from the emotion engine. For example, if the device detects that the user is feeling stressed, it will prioritize displaying a concise explanation.

[0392] Specific examples

[0393] If a user signs "OK," the emotion engine recognizes that the user is feeling stressed. The device sends this information to the server, which searches for the sign information corresponding to "OK" and returns a brief explanation. It also adds positive feedback as a support message. The device visually displays this to the user, helping them to reduce their stress.

[0394] Prompt Sentence Examples

[0395] Examples of input prompts for a generative AI model might include:

[0396] "Recognize sign language used in factories and provide dynamic support according to employees' emotions."

[0397] This will enable smooth communication within the factory using sign language and personalized support that responds to employees' emotions.

[0398] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0399] Step 1:

[0400] The user signs.

[0401] Input: User's sign language actions, facial expressions, and voice

[0402] Output: Video frames and audio data captured by the camera and microphone

[0403] Specific operation: When a user signs, the device's camera records the sign and the microphone collects the audio.

[0404] Step 2:

[0405] The device processes the input data to analyze the user's emotions in real time.

[0406] Input: Video frames and audio data

[0407] Output: User's emotional state (e.g., relaxed, stressed, etc.)

[0408] How it works: The device uses the EmotionRecognition library to analyze facial expressions from video frames and tone of voice from audio data, and then determines the user's emotional state based on the results of this analysis.

[0409] Step 3:

[0410] The device recognizes sign language.

[0411] Input: Video frame

[0412] Output: Sign language commands (e.g. OK, Thank you, etc.)

[0413] Specific actions: The device uses a sign language recognition algorithm to analyze the sign language actions in the video frames and extract specific sign language commands.

[0414] Step 4:

[0415] The terminal transmits the user's input data and emotion information to the server.

[0416] Input: sign language commands, emotional states, video frames, audio data

[0417] Output: Request data to the server

[0418] Specific operation: The terminal assembles the user's input data (sign language commands, video frames, and voice data) and emotional state into packets and sends them to the server.

[0419] Step 5:

[0420] The server searches for the corresponding sign language information from the sign language database.

[0421] Input: Request data

[0422] Output: Search results (sign meaning, trigger, additional information, etc.)

[0423] Specific operation: The server receives the request data and queries the sign language database, searching for sign language information corresponding to the sign language command and adjusting the level of detail of the information according to the emotional state.

[0424] Step 6:

[0425] The server sends the search results to the terminal.

[0426] Input: Search results

[0427] Output: Response data to the terminal

[0428] Specific operation: The server organizes the search results, converts them into an appropriate format, and sends them to the terminal.

[0429] Step 7:

[0430] The terminal visually displays the search results to the user.

[0431] Input: Response data

[0432] Output: What is displayed on the user interface

[0433] Specific operation: The device receives the response data and adjusts the display content according to the user's emotional state, for example, displaying a brief explanation to a stressed user and providing detailed information to a relaxed user.

[0434] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0435] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0436] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0437] [Second embodiment]

[0438] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0439] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0440] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0441] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0442] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0443] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0444] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0445] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0446] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0447] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0448] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0449] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0450] MODE FOR CARRYING OUT THE INVENTION

[0451] The present invention relates to a system that provides "cues" and "meanings" related to sign language to help users understand sign language. This system provides a means for efficiently learning sign languages ​​that vary by region and generation. Specific embodiments of the present invention are described below.

[0452] Overall system configuration

[0453] This system mainly consists of the following three main components:

[0454] 1. A server that manages the sign language database and stores additional information on sign languages ​​and their "cues" and "meanings."

[0455] 2. A terminal that accepts user input, sends requests to a server, and displays the results.

[0456] 3. Users who search for, learn, and use information about sign language.

[0457] Sign Language Database

[0458] server:

[0459] A sign language database systematically stores multiple pieces of information about sign languages. This includes not only information about the gestures and actions of sign languages, but also additional information such as the background, region, and era in which the sign language originated. The server manages this information and returns appropriate data in response to user search requests.

[0460] Handling User Input

[0461] User:

[0462] Users input information about a particular sign language through their device. For example, to find out about the sign for "thank you," they type "thank you" into the search bar. Information about the user's region and age is also collected. This information is important for identifying regional and age-specific variations in sign language.

[0463] Submitting a search request

[0464] Device:

[0465] The device receives the user's input data and sends a request for sign language information to the server based on that data. The request includes not only the keywords entered by the user, but also regional and historical information. This data is used to improve the accuracy of the search.

[0466] Database search and result formatting

[0467] server:

[0468] The server receives a user request and searches for the corresponding sign language information from a sign language database. Specifically, it retrieves each sign language variation and its additional information based on the input keywords. The search results are then formatted appropriately before being returned to the user. This formatted data includes details such as the sign gesture, the region where it originated, the era, and its meaning.

[0469] Displaying search results

[0470] Device:

[0471] The device analyzes the search results received from the server and displays them on the user interface. The display includes information about specific sign language gestures and their "cues" and "meanings," making it easier for users to understand the background and meaning of the sign language.

[0472] Specific examples

[0473] As a specific example, consider the case where a user searches for the sign language for "thank you." The user enters "thank you" in the search bar and presses the search button. At this time, the device sends the user's input data, along with region information (e.g., Tokyo) and age information (e.g., 30-40 years old) to the server.

[0474] The server searches the database for sign language information related to "thank you," searching for different variations such as "in the Kansai region, the thumb and index finger form a circle" and "in Tokyo, the palms are placed together," and obtaining the meaning and background information for each.

[0475] The search results are formatted and sent back to the device, where they are displayed on the user interface. Users can visually check the sign language and its meanings specific to each region and era. This helps users understand the diversity of sign language and stimulates their interest in sign language.

[0476] The embodiment of the present invention has been specifically described above. This system makes it easier to understand the diversity of sign languages ​​and makes people feel closer to sign languages.

[0477] The processing flow will be explained below.

[0478] Program processing steps

[0479] Step 1: Receiving User Input

[0480] User:

[0481] If you want to find information about sign language, enter a keyword in the search bar, for example, "thank you."

[0482] Step 2: Collect user information

[0483] Device:

[0484] It receives user input data (keywords) and also collects the user's location information (e.g., identifying "Tokyo" using GPS) and age information (e.g., identifying "30-40 years old" from the profile).

[0485] Step 3: Create and submit a request

[0486] Device:

[0487] The collected user input data, region information, and age information are sent to the server as a request. The request format is as follows: { keyword: 'Thank you', region: 'Tokyo', age_group: '30-40'}.

[0488] Step 4: Receiving and Parsing the Request

[0489] server:

[0490] The received request data is analyzed to identify the sign language the user wants to search for and its related information.

[0491] Step 5: Retrieving information from the database

[0492] server:

[0493] The system searches the sign language database for sign language information and additional information that matches the keywords in the request. Specifically, it searches for sign language variations related to "thank you" and their respective "trigger" and "meaning."

[0494] Step 6: Formatting the search results

[0495] server:

[0496] The search results are formatted and converted into a format to be returned to the user. For example, in JSON format, they are formatted as "{ sign: 'Thank you', variations: [{ region: 'Kansai', gesture: 'Make a circle with your thumb and index finger', meaning: 'Convey gratitude'}, { region: 'Tokyo', gesture: 'Put your palms together', meaning: 'Meaning of gratitude'}]}".

[0497] Step 7: Submit search results

[0498] server:

[0499] The formatted search results are returned to the terminal. The returned data is in a format that has already been formatted.

[0500] Step 8: Parse and display search results

[0501] Device:

[0502] The search results received from the server are analyzed and displayed on the user interface, visually displaying specific sign language gestures and information about their "cues" and "meanings."

[0503] Step 9: Verify the user and request additional information

[0504] User:

[0505] You can review the displayed sign language information and request more detailed information if necessary. For example, if you want to learn more about the development and historical background of the sign language, click the button for more information.

[0506] The above are the specific program processing steps in the embodiment of the present invention, which provide users with information about the diversity of sign languages ​​and their background information, thereby increasing their understanding and interest in sign languages.

[0507] Example 1

[0508] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0509] While there are many systems to promote the learning and understanding of sign language, few systems efficiently and accurately provide the different sign language variations by region and age. Furthermore, conventional systems have difficulty providing detailed background information and meanings of sign language. Therefore, there is a need for a system that allows users to easily obtain sign language information according to specific conditions and understand its meaning and background.

[0510] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0511] In this invention, the server includes means for receiving a user's search keywords and regional and era information, means for managing a database that systematically stores various information related to sign language, means for searching the database for information such as gestures, background, region, era, and meaning of the relevant sign language based on data entered by the user, means for formatting the search results in an appropriate format such as JSON and returning the results, and means for analyzing the search results received from the server and displaying them on a user interface. This enables a user to efficiently and accurately obtain sign language information tailored to a specific region or era and understand its background and meaning.

[0512] 1. "Means" refers to methods or devices used to achieve a particular purpose.

[0513] 2. "Search Keywords" refers to words or phrases that users enter to obtain specific sign language information.

[0514] 3. "Region information" refers to geographic information where a user lives or is associated, and is used to identify sign language variations.

[0515] 4. "Date information" refers to the age of a user or temporal information necessary to identify a sign language variation of a particular era.

[0516] 5. "Database" refers to a collection of information for the systematic storage and management of information related to sign languages.

[0517] 6. "Gesture" refers to the specific movements of the hands and body when signing.

[0518] 7. "Context" refers to information about how a particular sign language originated and what its cultural and historical background is.

[0519] 8. "Meaning" refers to the content or intent that a particular sign language is intended to convey to the user.

[0520] 9. "JSON format" refers to a lightweight data exchange format for representing data in a structured manner.

[0521] 10. "User interface" refers to the display and input devices that allow a user to directly interact with a system.

[0522] 11. "Parsing" refers to the process of understanding data received from the server and converting it into a user-friendly format.

[0523] MODE FOR CARRYING OUT THE INVENTION

[0524] The present invention relates to a system that allows users to learn sign language efficiently and accurately. To implement this system, a server that manages a sign language database and provides information based on user search requests, a terminal for users to perform searches, and users who actually learn sign language are required.

[0525] Server Configuration

[0526] The server has a database that systematically stores and manages various information related to sign language. For example, a relational database such as MySQL can be used. The database includes information such as sign language gestures, background, region, era, and meaning. Based on a search request received from a user, the server searches for the relevant sign language information, formats it in an appropriate format (e.g., JSON format), and returns it.

[0527] Device configuration

[0528] The terminal provides an interface for the user to input search keywords, region information, and era information. For example, a PC, smartphone, tablet, or other device can be used. The terminal receives the data entered by the user and sends it to the server in the form of an HTTP POST request, for example. The terminal analyzes the data returned from the server and displays it on the user interface.

[0529] User operations

[0530] Users enter keywords related to a specific sign language into the device's search bar. For example, if they want to find the sign for "thank you," they enter this keyword. In addition, the system also collects information about the user's region and age. For example, "Tokyo" or "30-40 years old." This information is important for improving search accuracy.

[0531] Specific example explanation

[0532] As a specific example of use, consider the case where a user searches for the sign for "thank you."

[0533] The user enters "Thank you" in the search bar of the device and presses the search button. The device sends the user's input data, region information (e.g., Tokyo), and age information (e.g., 30-40 years old) to the server.

[0534] The server searches the sign language database for sign language information related to "thank you." For example, different variations such as "in the Kansai region, the thumb and index finger form a circle" and "in Tokyo, the palms are placed together" are searched for, and the meanings and background information of each are also obtained. These search results are then formatted and sent back to the device.

[0535] The device analyzes the search results and displays them on the user interface. Users can visually check the sign language and its meanings specific to each region and era, which helps them understand the diversity of sign language and stimulates their interest in sign language.

[0536] Prompt Sentence Examples

[0537] Below are some examples of specific prompt sentences.

[0538] If a user wants to find out about the sign language for "thank you," they type "thank you" into the search bar and press the search button. The device sends the user's input data, along with region information (e.g., Tokyo) and age information (e.g., 30-40 years old), to the server. The server searches the sign language database for information related to "thank you," and obtains different variations, such as a circular thumb and index finger in the Kansai region and a palm-to-palm gesture in Tokyo. The server also provides information about the meaning and background of each variation. The formatted search results are sent back to the device and displayed on the user interface.

[0539] The above describes a specific embodiment of the present invention. This system allows users to efficiently learn the diversity of sign languages ​​and gain a deeper understanding of their backgrounds and meanings.

[0540] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0541] Specific flow of program processing

[0542] Step 1:

[0543] The user enters search keywords into the device

[0544] The user enters a specific sign language keyword into the device's search bar, for example, "thank you," along with region information (e.g., Tokyo) and age information (e.g., 30-40 years old).

[0545] Input: Search keyword "Thank you", region information "Tokyo", age information "30-40 years old"

[0546] Output: User input data

[0547] Step 2:

[0548] The device receives user input data and sends it to the server

[0549] The terminal creates an HTTP POST request based on the received user input data and sends it to a specific API endpoint on the server.

[0550] Input: User-entered data

[0551] Data transformation: Converting user-entered data into an HTTP POST request

[0552] Output: Request sent to server

[0553] Step 3:

[0554] The server receives the request and performs a database lookup.

[0555] The server receives the request from the device and queries the sign language database based on the keyword "thank you" to retrieve relevant information. The query searches for information such as the sign gesture, background, region, era, and meaning.

[0556] Input: HTTP POST request (search keyword "Thank you", region information "Tokyo", age information "30-40 years old")

[0557] Data manipulation: performing database queries

[0558] Output: Search result data (sign language information)

[0559] Step 4:

[0560] The server formats the search results and returns them to the device

[0561] The server formats the search results into JSON format, converts them into the appropriate format, and returns them to the terminal.

[0562] Input: Search result data

[0563] Data processing: Conversion to JSON format

[0564] Output: Formatted search results data

[0565] Step 5:

[0566] The device receives the search results and displays them on the user interface.

[0567] The device receives the formatted data returned from the server, analyzes it, and displays it on the user interface, allowing the user to visually check the various variations of sign language and their meanings and backgrounds.

[0568] Input: Formatted search result data

[0569] Data Calculation: Data Analysis

[0570] Output: User interface display of search results

[0571] The processing steps of the system have been described above in detail.

[0572] (Application example 1)

[0573] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0574] Learning sign language and using it efficiently in specific tasks is difficult because the background and meaning of sign language vary by region and age. Additionally, certain tasks, such as food delivery, require the accurate use of sign language when communicating with hearing-impaired customers, but there are a lack of ways for employees to quickly learn that sign language. To solve this issue, a system is needed that allows employees to efficiently learn and use sign language related to specific tasks in a timely manner.

[0575] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0576] In this invention, the server includes means for receiving user input data, means for constructing a sign language database and storing additional information related to each sign language, means for searching the sign language database for corresponding sign language information based on the user input data, means for visually displaying the search results to the user, and means for enabling efficient learning and use of sign languages ​​in specific tasks, thereby enabling accurate learning and rapid use of sign languages.

[0577] A "means for receiving user input" is the interface a user uses to provide information about a particular sign language, including speech, keyboard input, touch, etc.

[0578] "Means for building a database of sign languages ​​and storing additional information related to each sign language" refers to a system or server for systematically storing and managing detailed information such as sign language gestures, regional information, historical information, and their background and meanings.

[0579] "Means for searching for corresponding sign language information from a sign language database based on user input data" refers to algorithms or processes for efficiently searching and extracting relevant sign language information from a sign language database based on keywords, regional information, and era information provided by the user.

[0580] "Means for visually displaying search results to the user" refers to a graphical user interface or display device that displays the searched sign language information in a format that is easy for the user to understand.

[0581] "Means to enable efficient learning and use of sign language in specific jobs" refers to training modules and guidelines to support employees in specific jobs, such as food delivery, in quickly learning the necessary sign language and using it effectively in their work.

[0582] "When an employee performing a specific task uses sign language, collect regional and age information and use it to search for sign language information" refers to the process of inputting regional and age information and selecting the most appropriate sign language based on that information in order to provide the appropriate sign language according to the context when the employee uses sign language.

[0583] "Additional information related to the meaning or trigger of sign language" is supplementary information for understanding sign language, such as the situations in which a particular sign language is used, its background and history, etc.

[0584] This invention provides a system that enables employees of food delivery services to easily learn and use sign language. This system mainly consists of the following components:

[0585] Overall system configuration

[0586] This system mainly consists of the following five main components:

[0587] 1. A means of receiving user input data

[0588] 2. A means of building a database of sign languages ​​and storing additional information associated with each sign language.

[0589] 3. A means to search for corresponding sign language information from a sign language database based on user input data

[0590] 4. A means of visually displaying search results to the user

[0591] 5. Means of enabling people to learn and use sign language efficiently in specific jobs

[0592] Handling User Input

[0593] Device:

[0594] Users use their smartphones to input information about a specific sign language. For example, if a delivery person wants to know the sign for "thank you," they can enter "thank you" into the application's search bar. At the same time, the user's region and age information are also collected. This information is important for identifying regional and age-specific variations in sign language.

[0595] Submitting a search request

[0596] Device:

[0597] The device sends the user's input data, along with regional and historical information, to the server, which is used to improve search accuracy and identify sign language variations.

[0598] Database search and result formatting

[0599] server:

[0600] The server receives a request from a user and searches for the corresponding sign language information from a sign language database. Specifically, it retrieves each sign language variation and its additional information based on the input keyword. For example, for the sign expression for "thank you," it searches for sign language variations for the Tokyo area and the 30-40 age group, and also retrieves the meaning and background information of each.

[0601] Displaying search results

[0602] Device:

[0603] The terminal analyzes the search results received from the server and displays them on the user interface. The display includes information about the specific sign language gestures and their "cues" and "meanings," making it easier for employees to understand the background and meaning of the sign language.

[0604] Specific examples

[0605] As a specific use case, consider a food delivery person who wants to learn the sign language for "thank you." The delivery person enters "thank you" into the application's search bar and presses the search button. At this time, the device sends the user's input data, along with region information (e.g., Tokyo) and age information (e.g., 30-40 years old) to the server. The server then searches the database for sign language information related to "thank you" and obtains detailed variations in sign language for different regions and age groups. The search results are formatted and sent back to the device, where the delivery person can view this information on the user interface. This allows the delivery person to understand the sign language and its meaning specific to their region and age group, and use sign language appropriately in their actual delivery work.

[0606] Prompt Sentence Examples

[0607] An example of a prompt to input to a generative AI model is as follows:

[0608] "We will create a food delivery application program for the hearing impaired. It will run on smartphones and have a function that allows users to learn sign language efficiently. When a user enters keywords for the sign language for "thank you" and provides region and era information, the corresponding sign language instructions will be displayed. Specifically, search results will be displayed that include detailed information such as sign language gestures, region, era, and meaning. Please use the following data processing flow. Input data: Keywords, region, era. Output data: Sign language gestures, region, era, meaning."

[0609] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0610] Step 1:

[0611] The user operates the delivery person app and inputs information about sign language. For example, if the delivery person wants to know the sign language for "thank you," they enter "thank you" in the application's search bar and press the search button. At this point, region information (e.g., Tokyo) and age information (e.g., 30-40 years old) are also entered at the same time. The input data is the keyword, region, and age. The output data is the information entered by the user, and the device retains this as input.

[0612] Step 2:

[0613] The device collects the user's input data and sends it to the server. The input data includes keywords, region, and age information. Based on this, the device sends a request to the server. This request includes the keywords entered by the user, such as "Thank you," as well as region and age information. The output data is the search request received by the server.

[0614] Step 3:

[0615] The server searches the sign language database based on the received search request. Specifically, it searches for entries in the database that match keywords such as "thank you," and also searches for associated regional and historical information. The input data is the search request, and the output data is the search results. The server processes this using a database search algorithm to obtain the corresponding sign language information and its additional information.

[0616] Step 4:

[0617] The server formats the acquired sign language information, prepares it in a format that can be sent, and sends it back to the user's device. The search results include detailed information such as the sign language gesture, region, era, and meaning. The input data are the search results, and the output data are the formatted search results. The server uses a data formatting algorithm to format the results and send them to the device.

[0618] Step 5:

[0619] The terminal analyzes the received search results and displays them visually. Specifically, it displays specific sign language gestures and information about their "cues" and "meanings" on the user interface. This makes it easier for users to understand the background and meaning of the sign language. The input data are the formatted search results, and the output data is the content displayed to the user. The terminal's analysis and display processing allows the delivery person to understand and use sign language accurately.

[0620] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0621] MODE FOR CARRYING OUT THE INVENTION

[0622] The present invention relates to a system for aiding in the understanding of sign language and fostering familiarity with sign language. In particular, the system provides a more personalized learning experience by incorporating an emotion engine that recognizes a user's emotions and dynamically adjusts the method and content of providing sign language information. Specific embodiments of the present invention are described below.

[0623] Overall system configuration

[0624] This system mainly consists of the following four main components:

[0625] 1. A device that incorporates an emotion engine to receive user input data and recognize emotions.

[0626] 2. A server that manages the sign language database and stores the sign language and additional information such as its "trigger" and "meaning."

[0627] 3. Users who search for, learn, and use information about sign language.

[0628] 4. A means for providing sign language information based on the analysis results of the emotion engine.

[0629] Sign Language Database

[0630] server:

[0631] The server manages a sign language database and stores detailed information about sign languages, including information about the gestures and actions of the sign language, as well as additional information such as the background, region, and era in which the sign language originated. The server efficiently searches and provides this information.

[0632] Emotion Engine

[0633] Device:

[0634] The emotion engine is a software module for analyzing the user's emotions in real time. It recognizes the user's emotional state by analyzing the user's facial expressions and tone of voice through a camera and microphone. Based on this information, the system provides sign language information that matches the user's current emotion.

[0635] Handling User Input

[0636] User:

[0637] The user inputs information about a specific sign language through the device. For example, to find out about the sign for "thank you," the user enters "thank you" into the search bar. The emotion engine then detects the user's facial expressions and voice and analyzes their emotional state.

[0638] Submitting a search request

[0639] Device:

[0640] The device sends a request to the server containing the user's input data and the emotion information recognized by the emotion engine, including information about the region and age.

[0641] Database search and result formatting

[0642] server:

[0643] The server analyzes the received request and searches for the corresponding sign language information from the sign language database. Taking into account the results of the emotion engine, the server adjusts the priority of the sign language information appropriate to the user's emotion. For example, if the user is relaxed, it provides more detailed information, but if the user is feeling impatient, it prioritizes concise information.

[0644] Displaying search results

[0645] Device:

[0646] The device analyzes the search results received from the server and adjusts the display content based on the emotional information obtained from the emotion engine. For example, it displays messages emphasizing success to users who show positive emotions, and adds encouraging messages to users who show negative emotions.

[0647] Specific examples

[0648] Consider a specific use case. When a user searches for the sign language for "thank you," they enter "thank you" in the search bar and press the search button. At this time, the emotion engine analyzes the user's facial expressions and tone of voice and recognizes that the user is relaxed.

[0649] The device sends a request containing the user's input data, region information, age information, and emotion information to the server. The server searches for sign language information corresponding to "thank you" and, taking into account the results of the emotion engine, provides detailed sign language information that is appropriate for when the user is relaxed. It also adds encouraging and positive feedback as support messages as needed.

[0650] The device analyzes the received search results and displays them on the user interface, visually displaying specific sign gestures and their "cues" and "meanings." This helps users understand the background and meaning of sign language, and by providing emotional feedback during the learning process, it increases interest in and familiarity with sign language.

[0651] The above is a specific description of an embodiment of the present invention. This system provides the diversity of sign languages ​​and their background information, and also provides a personalized sign language learning experience that responds to the user's emotions.

[0652] The processing flow will be explained below.

[0653] MODE FOR CARRYING OUT THE INVENTION

[0654] The processing flow of the system based on the present invention will be specifically explained by dividing it into the following steps.

[0655] Step 1: Receiving User Input

[0656] User:

[0657] If you want to find information about sign language, enter a keyword in the search bar, for example, "thank you."

[0658] Step 2: Emotion analysis using the emotion engine

[0659] Device:

[0660] The emotion engine is activated and analyzes the user's facial expressions and voice via the camera and microphone, recognizing their emotional state in real time, such as whether they are relaxed or anxious.

[0661] Step 3: Collect user information

[0662] Device:

[0663] Collects user input data (keywords), emotional information, regional information (e.g., identifying "Tokyo" using GPS), and age information (e.g., identifying "30-40 years old" from the profile).

[0664] Step 4: Create and submit a request

[0665] Device:

[0666] The request data is created based on the collected information and sent to the server. The request contains the following data: { keyword: 'Thank you', region: 'Tokyo', age_group: '30-40', emotion: 'Relax'}.

[0667] Step 5: Receiving and Parsing the Request

[0668] server:

[0669] The received request data is analyzed to identify the sign language the user wants to search for and related information. Emotional information is also used in the analysis.

[0670] Step 6: Retrieving information from the database

[0671] server:

[0672] Searches for sign language information and additional information that matches the entered keywords from a sign language database. For example, sign language variations related to "thank you" and their "trigger" and "meaning."

[0673] Step 7: Formatting the search results

[0674] server:

[0675] It shapes search results, adjusting the level of detail and content of the information provided based on the user's emotional state, for example prioritizing detailed information for a relaxed user and concise information for a rushed user.

[0676] Step 8: Submit search results

[0677] server:

[0678] The formatted search results are sent back to the device. The returned data can be in the following format: "{ sign: 'Thank you', variations: [{ region: 'Kansai', gesture: 'Circle your thumb and index finger', meaning: 'Expresses gratitude'}, { region: 'Tokyo', gesture: 'Put your palms together', meaning: 'Means thank you'}], message: 'You are relaxed. Show more information.'}".

[0679] Step 9: Parse and display search results

[0680] Device:

[0681] The search results received from the server are analyzed and displayed on the user interface. At this time, appropriate encouragement and feedback messages are also displayed based on the analysis results of the emotion engine. For example, a message such as "You are relaxed. Please check for more information" may be displayed.

[0682] Step 10: Verify the user and request additional information

[0683] User:

[0684] View the displayed sign language information, its supplementary information, and emotion-based feedback. If necessary, you can request more information. For example, if you want to learn more about the historical background of the sign language, you can click the button for more information.

[0685] The above are the specific program processing steps of an embodiment of the present invention, which combines an emotion engine that recognizes the user's emotions. This system provides the diversity of sign languages ​​and their background information, enabling a personalized sign language learning experience that responds to the user's emotions.

[0686] Example 2

[0687] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0688] Conventional systems for supporting sign language learning rely on user input data and basic search functions, and are unable to provide a personalized learning experience that takes into account the user's emotional state. This makes it difficult to elicit the user's interest and affinity and promote effective learning. Furthermore, optimization of search and provided information that takes into account the user's region and age is also insufficient. To solve this problem, a system is needed that analyzes the user's emotional state in real time and provides optimal sign language information based on that analysis.

[0689] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving input data and emotional state of a user, means for constructing a sign language database and storing additional information related to each sign language, means for searching for corresponding sign language information from the sign language database based on the user's input data and emotional information, and means for displaying the search results and the emotional information in a visually and emotionally adjusted form to the user. This makes it possible to provide optimal sign language information according to the user's emotional state, thereby realizing a personalized learning experience. Furthermore, by taking into account regional and era information, it is possible to provide more accurate information.

[0690] "User-entered data" refers to information entered by a user using a terminal, such as keywords and phrases related to sign language searches.

[0691] "Emotional state" refers to a psychological state analyzed from the user's facial expression, tone of voice, etc., and is classified into categories such as relaxation, impatience, and joy.

[0692] A "sign language database" refers to a database that constructs and stores data including sign language gestures, meanings, background information, etc.

[0693] "Additional information" is detailed information related to a sign language, including additional information such as the background, region, and era in which the sign language originated.

[0694] The "search means" refers to a function that searches for corresponding sign language information from a sign language database based on the user's input data and emotion information.

[0695] "Visual display" refers to a method of visually presenting sign language information to a user on a screen, including sign language gestures and additional information.

[0696] "Emotional adjustment" refers to a method of adjusting the priority of information based on the user's emotional state and providing the information to the user in an optimal manner.

[0697] "Region information" is information about the user's current location, and is used for filtering when searching for sign language information.

[0698] "Age information" is information about the user's age and birth decade, and is used for filtering when searching for sign language.

[0699] This invention is a system for providing a personalized experience for users learning sign language. The system has the ability to analyze the user's emotional state in real time and dynamically provide sign language information based on that analysis.

[0700] Overall system configuration

[0701] 1. Receiving user input data

[0702] User: The user enters the sign language keyword they want to search for in the search bar of their device. For example, if they want to learn the sign language for "thank you," they enter "thank you" in the search bar and click the search button.

[0703] 2. Acquiring emotional information

[0704] Device: The device uses a built-in camera and microphone to capture the user's facial expressions and tone of voice, which are then analyzed by an emotion engine. This data is then classified as the user's emotional state (e.g., relaxed, anxious, happy, etc.).

[0705] 3. Generating a search request

[0706] Device: The device generates a search request that includes the keywords and emotion information entered by the user, as well as location and age information. The generated request is sent to the server in the next step.

[0707] 4. Sending a request to the server

[0708] Device: The device generates a search request and sends it to the server, using an internet connection to transfer data using HTTP or HTTPS protocols.

[0709] 5. Database Search

[0710] Server: The server analyzes the received request and searches for the corresponding sign language information from a sign language database, which contains information about the gestures, meanings, and context of the sign language. The search results are prioritized based on the emotional information.

[0711] 6. Formatting Sign Language Information

[0712] Server: The server formats the search results and generates a response in a user-friendly format. If the user is relaxed, the response will contain more information.

[0713] 7. Receiving and Displaying Search Results

[0714] Device: The device analyzes the search results received from the server and displays them in a visually and emotionally tailored way to the user's emotions. For users who show positive emotions, it displays messages that highlight success, and for users who show negative emotions, it adds encouraging messages.

[0715] Specific use cases

[0716] As a specific example, consider the case where a user wants to search for the sign language for "thank you." The user types "thank you" into the search bar and presses the search button. At this time, the device's emotion engine analyzes the user's facial expressions and voice and recognizes that the user is relaxed.

[0717] The device generates a request including the input data "Thank you," region information, age information, and emotional information "Relax," and sends it to the server. The server searches for the corresponding sign language information in a sign language database, prioritizes providing detailed information, and also generates a response including positive feedback.

[0718] The device displays the received search results in a user interface. For example, not only are the sign gestures, background information, and meanings visually displayed, but detailed information such as "This sign means thank you and is used in these situations" is also provided. Furthermore, because the user is relaxed, a positive feedback message is also displayed.

[0719] Example of input prompt for generative AI model

[0720] "Suppose a user wants to know the sign for 'thank you'. You have emotional information that the user is relaxed. In this case, provide detailed information about the sign, including the gesture, background information, and geographic and historical information."

[0721] In this way, a sign language learning support system that reflects emotional information can provide users with a personalized learning experience.

[0722] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0723] Step 1: Receiving User Input

[0724] User:

[0725] The user enters the sign language keyword they want to search for in the search bar of their device and presses the search button. For example, if they want to search for the sign language for "thank you," they enter "thank you" in the search bar.

[0726] Input: Keyboard input ("Thank you")

[0727] Output: Keyword data ("Thank you")

[0728] Specific behavior:

[0729] The user enters keywords into the device's search bar and clicks the "Search" button.

[0730] Step 2: Acquiring emotional information

[0731] Device:

[0732] The device uses a built-in camera and microphone to capture the user's facial expressions and tone of voice, which are then analyzed by an emotion engine.

[0733] Input: Camera video data, microphone audio data

[0734] Output: Emotion data (e.g. "Relaxed")

[0735] Specific behavior:

[0736] The camera captures the user's face and the microphone records the user's voice.

[0737] An emotion engine analyzes this data to determine the user's emotional state.

[0738] Step 3: Generate a search request

[0739] Device:

[0740] The terminal generates a search request by combining the keywords, emotional information, and regional and age information entered by the user.

[0741] Input: Keyword data ("Thank you"), emotion data ("Relax"), region information, age information

[0742] Output: Search request data

[0743] Specific behavior:

[0744] Based on the input data, the data is formatted into a search request format.

[0745] Create request data including region and age information.

[0746] Step 4: Sending a request to the server

[0747] Device:

[0748] The terminal transmits the generated search request to the server.

[0749] Input: Search request data

[0750] Output: HTTP request to the server

[0751] Specific behavior:

[0752] Using the HTTP protocol, a request is sent to the specified URL on the server using the POST method.

[0753] Step 5: Database Search

[0754] server:

[0755] The server analyzes the received request and searches for the corresponding sign language information from the sign language database.

[0756] Input: Search request data

[0757] Output: Sign language information data

[0758] Specific behavior:

[0759] A database query is generated and searched against a sign language database.

[0760] Take search results and adjust priorities based on sentiment information.

[0761] Step 6: Formatting the sign language information

[0762] server:

[0763] The server formats the search results and prepares the sign language information in a user-friendly format.

[0764] Input: Sign language information data

[0765] Output: Formatted display data

[0766] Specific behavior:

[0767] Format sign language gestures, meanings and related information.

[0768] Add positive feedback messages based on emotions.

[0769] Step 7: Receiving and displaying search results

[0770] Device:

[0771] The terminal analyzes the search results received from the server and generates content to be displayed to the user.

[0772] Input: Formatted display data

[0773] Output: User interface display

[0774] Specific behavior:

[0775] The data received from the server is analyzed and sign language gestures and background information are displayed on the user interface.

[0776] It also displays emotion-based feedback messages.

[0777] (Application example 2)

[0778] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0779] There is a need to improve and streamline communication within factories, but there are challenges such as difficulty in communicating smoothly with hearing-impaired employees and responding to employees' emotions in stressful environments. In particular, when communicating using sign language, it is difficult to provide information that corresponds to the emotional state of employees, and an effective system for this purpose is needed.

[0780] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0781] In this invention, the server includes means for receiving user input data, means for building a sign language database and storing additional information related to each sign language, emotion recognition means using an emotion engine that analyzes the user's emotions in real time, means for searching the sign language database for corresponding sign language information based on the user's input data and emotion information, and means for visually displaying the search results in a form that corresponds to the user's emotional state. This enables smooth communication using sign language in factory communications and personalized support that corresponds to the emotions of employees.

[0782] The "means for receiving user input data" is a hardware or software mechanism for obtaining information provided by a user.

[0783] "Means for building a sign language database and storing additional information related to each sign language" refers to a method for comprehensively organizing and storing information about sign languages, as well as their background and meanings.

[0784] The "emotion recognition means" is a mechanism for analyzing the user's emotional state in real time and obtaining the results.

[0785] The "means for searching for corresponding sign language information from a sign language database" is a method for searching for appropriate sign language information from a sign language database based on the user's input data and emotion information.

[0786] "Visual display means" refers to the method of presenting search results to the user through a visual interface such as a display or screen.

[0787] "Means for collecting regional and generational information" refers to methods for acquiring and accumulating data relating to the geographical location and generation of users.

[0788] The "means for dynamically personalizing and providing additional information" is a mechanism for individually optimizing and providing additional information about sign language based on the user's emotional state and other factors.

[0789] The present invention provides a sign language compatible robot assistant system for realizing efficient communication within a factory. Specific embodiments are described below.

[0790] Overall system configuration

[0791] The system mainly consists of the following main components:

[0792] 1. Hardware and software for receiving user input data

[0793] 2. A server that builds a sign language database and stores additional information related to each sign language.

[0794] 3. A device that uses an emotion engine to analyze user emotions in real time

[0795] 4. A device that visually displays search results in a format that corresponds to the user's emotional state

[0796] Receiving user input data

[0797] User input data is collected by a device equipped with a camera and microphone, with the Python OpenCV library installed to capture video frames and collect audio data.

[0798] Sign Language Database

[0799] The server manages a sign language database and stores detailed information about sign languages. This includes information about the gestures and actions of the sign language, as well as additional information such as the background, region, and era in which the sign language originated. The server uses the Python sign_language_database module to efficiently search and provide this information.

[0800] Emotion Engine

[0801] The emotion engine is a software module for analyzing the user's emotions in real time. This module uses a library called EmotionRecognition to analyze the user's facial expressions and tone of voice. The level of detail of the information provided changes depending on whether the user is feeling stressed or not.

[0802] Processing user input data and emotional information

[0803] The device sends the user's input data and the emotional information recognized by the emotion engine together to the server. The server analyzes this request and searches the database for corresponding sign language information. The search results are returned by the server with either a detailed explanation or a concise explanation selected based on the user's emotional state.

[0804] Displaying search results

[0805] The device analyzes the search results received from the server and adjusts the display content based on the emotional information obtained from the emotion engine. For example, if the device detects that the user is feeling stressed, it will prioritize displaying a concise explanation.

[0806] Specific examples

[0807] If a user signs "OK," the emotion engine recognizes that the user is feeling stressed. The device sends this information to the server, which searches for the sign information corresponding to "OK" and returns a brief explanation. It also adds positive feedback as a support message. The device visually displays this to the user, helping them to reduce their stress.

[0808] Prompt Sentence Examples

[0809] Examples of input prompts for a generative AI model might include:

[0810] "Recognize sign language used in factories and provide dynamic support according to employees' emotions."

[0811] This will enable smooth communication within the factory using sign language and personalized support that responds to employees' emotions.

[0812] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0813] Step 1:

[0814] The user signs.

[0815] Input: User's sign language actions, facial expressions, and voice

[0816] Output: Video frames and audio data captured by the camera and microphone

[0817] Specific operation: When a user signs, the device's camera records the sign and the microphone collects the audio.

[0818] Step 2:

[0819] The device processes the input data to analyze the user's emotions in real time.

[0820] Input: Video frames and audio data

[0821] Output: User's emotional state (e.g., relaxed, stressed, etc.)

[0822] How it works: The device uses the EmotionRecognition library to analyze facial expressions from video frames and tone of voice from audio data, and then determines the user's emotional state based on the results of this analysis.

[0823] Step 3:

[0824] The device recognizes sign language.

[0825] Input: Video frame

[0826] Output: Sign language commands (e.g. OK, Thank you, etc.)

[0827] Specific actions: The device uses a sign language recognition algorithm to analyze the sign language actions in the video frames and extract specific sign language commands.

[0828] Step 4:

[0829] The terminal transmits the user's input data and emotion information to the server.

[0830] Input: sign language commands, emotional states, video frames, audio data

[0831] Output: Request data to the server

[0832] Specific operation: The terminal assembles the user's input data (sign language commands, video frames, and voice data) and emotional state into packets and sends them to the server.

[0833] Step 5:

[0834] The server searches for the corresponding sign language information from the sign language database.

[0835] Input: Request data

[0836] Output: Search results (sign meaning, trigger, additional information, etc.)

[0837] Specific operation: The server receives the request data and queries the sign language database, searching for sign language information corresponding to the sign language command and adjusting the level of detail of the information according to the emotional state.

[0838] Step 6:

[0839] The server sends the search results to the terminal.

[0840] Input: Search results

[0841] Output: Response data to the terminal

[0842] Specific operation: The server organizes the search results, converts them into an appropriate format, and sends them to the terminal.

[0843] Step 7:

[0844] The terminal visually displays the search results to the user.

[0845] Input: Response data

[0846] Output: What is displayed on the user interface

[0847] Specific operation: The device receives the response data and adjusts the display content according to the user's emotional state, for example, displaying a brief explanation to a stressed user and providing detailed information to a relaxed user.

[0848] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0849] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0850] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0851] [Third embodiment]

[0852] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0853] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0854] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0855] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0856] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0857] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0858] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0859] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0860] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0861] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0862] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0863] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0864] MODE FOR CARRYING OUT THE INVENTION

[0865] The present invention relates to a system that provides "cues" and "meanings" related to sign language to help users understand sign language. This system provides a means for efficiently learning sign languages ​​that vary by region and generation. Specific embodiments of the present invention are described below.

[0866] Overall system configuration

[0867] This system mainly consists of the following three main components:

[0868] 1. A server that manages the sign language database and stores additional information on sign languages ​​and their "cues" and "meanings."

[0869] 2. A terminal that accepts user input, sends requests to a server, and displays the results.

[0870] 3. Users who search for, learn, and use information about sign language.

[0871] Sign Language Database

[0872] server:

[0873] A sign language database systematically stores multiple pieces of information about sign languages. This includes not only information about the gestures and actions of sign languages, but also additional information such as the background, region, and era in which the sign language originated. The server manages this information and returns appropriate data in response to user search requests.

[0874] Handling User Input

[0875] User:

[0876] Users input information about a particular sign language through their device. For example, to find out about the sign for "thank you," they type "thank you" into the search bar. Information about the user's region and age is also collected. This information is important for identifying regional and age-specific variations in sign language.

[0877] Submitting a search request

[0878] Device:

[0879] The device receives the user's input data and sends a request for sign language information to the server based on that data. The request includes not only the keywords entered by the user, but also regional and historical information. This data is used to improve the accuracy of the search.

[0880] Database search and result formatting

[0881] server:

[0882] The server receives a user request and searches for the corresponding sign language information from a sign language database. Specifically, it retrieves each sign language variation and its additional information based on the input keywords. The search results are then formatted appropriately before being returned to the user. This formatted data includes details such as the sign gesture, the region where it originated, the era, and its meaning.

[0883] Displaying search results

[0884] Device:

[0885] The device analyzes the search results received from the server and displays them on the user interface. The display includes information about specific sign language gestures and their "cues" and "meanings," making it easier for users to understand the background and meaning of the sign language.

[0886] Specific examples

[0887] As a specific example, consider the case where a user searches for the sign language for "thank you." The user enters "thank you" in the search bar and presses the search button. At this time, the device sends the user's input data, along with region information (e.g., Tokyo) and age information (e.g., 30-40 years old) to the server.

[0888] The server searches the database for sign language information related to "thank you," searching for different variations such as "in the Kansai region, the thumb and index finger form a circle" and "in Tokyo, the palms are placed together," and obtaining the meaning and background information for each.

[0889] The search results are formatted and sent back to the device, where they are displayed on the user interface. Users can visually check the sign language and its meanings specific to each region and era. This helps users understand the diversity of sign language and stimulates their interest in sign language.

[0890] The embodiment of the present invention has been specifically described above. This system makes it easier to understand the diversity of sign languages ​​and makes people feel closer to sign languages.

[0891] The processing flow will be explained below.

[0892] Program processing steps

[0893] Step 1: Receiving User Input

[0894] User:

[0895] If you want to find information about sign language, enter a keyword in the search bar, for example, "thank you."

[0896] Step 2: Collect user information

[0897] Device:

[0898] It receives user input data (keywords) and also collects the user's location information (e.g., identifying "Tokyo" using GPS) and age information (e.g., identifying "30-40 years old" from the profile).

[0899] Step 3: Create and submit a request

[0900] Device:

[0901] The collected user input data, region information, and age information are sent to the server as a request. The request format is as follows: { keyword: 'Thank you', region: 'Tokyo', age_group: '30-40'}.

[0902] Step 4: Receiving and Parsing the Request

[0903] server:

[0904] The received request data is analyzed to identify the sign language the user wants to search for and its related information.

[0905] Step 5: Retrieving information from the database

[0906] server:

[0907] The system searches the sign language database for sign language information and additional information that matches the keywords in the request. Specifically, it searches for sign language variations related to "thank you" and their respective "trigger" and "meaning."

[0908] Step 6: Formatting the search results

[0909] server:

[0910] The search results are formatted and converted into a format to be returned to the user. For example, in JSON format, they are formatted as "{ sign: 'Thank you', variations: [{ region: 'Kansai', gesture: 'Make a circle with your thumb and index finger', meaning: 'Convey gratitude'}, { region: 'Tokyo', gesture: 'Put your palms together', meaning: 'Meaning of gratitude'}]}".

[0911] Step 7: Submit search results

[0912] server:

[0913] The formatted search results are returned to the terminal. The returned data is in a format that has already been formatted.

[0914] Step 8: Parse and display search results

[0915] Device:

[0916] The search results received from the server are analyzed and displayed on the user interface, visually displaying specific sign language gestures and information about their "cues" and "meanings."

[0917] Step 9: Verify the user and request additional information

[0918] User:

[0919] You can review the displayed sign language information and request more detailed information if necessary. For example, if you want to learn more about the development and historical background of the sign language, click the button for more information.

[0920] The above are the specific program processing steps in the embodiment of the present invention, which provide users with information about the diversity of sign languages ​​and their background information, thereby increasing their understanding and interest in sign languages.

[0921] Example 1

[0922] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0923] While there are many systems to promote the learning and understanding of sign language, few systems efficiently and accurately provide the different sign language variations by region and age. Furthermore, conventional systems have difficulty providing detailed background information and meanings of sign language. Therefore, there is a need for a system that allows users to easily obtain sign language information according to specific conditions and understand its meaning and background.

[0924] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0925] In this invention, the server includes means for receiving a user's search keywords and regional and era information, means for managing a database that systematically stores various information related to sign language, means for searching the database for information such as gestures, background, region, era, and meaning of the relevant sign language based on data entered by the user, means for formatting the search results in an appropriate format such as JSON and returning the results, and means for analyzing the search results received from the server and displaying them on a user interface. This enables a user to efficiently and accurately obtain sign language information tailored to a specific region or era and understand its background and meaning.

[0926] 1. "Means" refers to methods or devices used to achieve a particular purpose.

[0927] 2. "Search Keywords" refers to words or phrases that users enter to obtain specific sign language information.

[0928] 3. "Region information" refers to geographic information where a user lives or is associated, and is used to identify sign language variations.

[0929] 4. "Date information" refers to the age of a user or temporal information necessary to identify a sign language variation of a particular era.

[0930] 5. "Database" refers to a collection of information for the systematic storage and management of information related to sign languages.

[0931] 6. "Gesture" refers to the specific movements of the hands and body when signing.

[0932] 7. "Context" refers to information about how a particular sign language originated and what its cultural and historical background is.

[0933] 8. "Meaning" refers to the content or intent that a particular sign language is intended to convey to the user.

[0934] 9. "JSON format" refers to a lightweight data exchange format for representing data in a structured manner.

[0935] 10. "User interface" refers to the display and input devices that allow a user to directly interact with a system.

[0936] 11. "Parsing" refers to the process of understanding data received from the server and converting it into a user-friendly format.

[0937] MODE FOR CARRYING OUT THE INVENTION

[0938] The present invention relates to a system that allows users to learn sign language efficiently and accurately. To implement this system, a server that manages a sign language database and provides information based on user search requests, a terminal for users to perform searches, and users who actually learn sign language are required.

[0939] Server Configuration

[0940] The server has a database that systematically stores and manages various information related to sign language. For example, a relational database such as MySQL can be used. The database includes information such as sign language gestures, background, region, era, and meaning. Based on a search request received from a user, the server searches for the relevant sign language information, formats it in an appropriate format (e.g., JSON format), and returns it.

[0941] Device configuration

[0942] The terminal provides an interface for the user to input search keywords, region information, and era information. For example, a PC, smartphone, tablet, or other device can be used. The terminal receives the data entered by the user and sends it to the server in the form of an HTTP POST request, for example. The terminal analyzes the data returned from the server and displays it on the user interface.

[0943] User operations

[0944] Users enter keywords related to a specific sign language into the device's search bar. For example, if they want to find the sign for "thank you," they enter this keyword. In addition, the system also collects information about the user's region and age. For example, "Tokyo" or "30-40 years old." This information is important for improving search accuracy.

[0945] Specific example explanation

[0946] As a specific example of use, consider the case where a user searches for the sign for "thank you."

[0947] The user enters "Thank you" in the search bar of the device and presses the search button. The device sends the user's input data, region information (e.g., Tokyo), and age information (e.g., 30-40 years old) to the server.

[0948] The server searches the sign language database for sign language information related to "thank you." For example, different variations such as "in the Kansai region, the thumb and index finger form a circle" and "in Tokyo, the palms are placed together" are searched for, and the meanings and background information of each are also obtained. These search results are then formatted and sent back to the device.

[0949] The device analyzes the search results and displays them on the user interface. Users can visually check the sign language and its meanings specific to each region and era, which helps them understand the diversity of sign language and stimulates their interest in sign language.

[0950] Prompt Sentence Examples

[0951] Below are some examples of specific prompt sentences.

[0952] If a user wants to find out about the sign language for "thank you," they type "thank you" into the search bar and press the search button. The device sends the user's input data, along with region information (e.g., Tokyo) and age information (e.g., 30-40 years old), to the server. The server searches the sign language database for information related to "thank you," and obtains different variations, such as a circular thumb and index finger in the Kansai region and a palm-to-palm gesture in Tokyo. The server also provides information about the meaning and background of each variation. The formatted search results are sent back to the device and displayed on the user interface.

[0953] The above describes a specific embodiment of the present invention. This system allows users to efficiently learn the diversity of sign languages ​​and gain a deeper understanding of their backgrounds and meanings.

[0954] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0955] Specific flow of program processing

[0956] Step 1:

[0957] The user enters search keywords into the device

[0958] The user enters a specific sign language keyword into the device's search bar, for example, "thank you," along with region information (e.g., Tokyo) and age information (e.g., 30-40 years old).

[0959] Input: Search keyword "Thank you", region information "Tokyo", age information "30-40 years old"

[0960] Output: User input data

[0961] Step 2:

[0962] The device receives user input data and sends it to the server

[0963] The terminal creates an HTTP POST request based on the received user input data and sends it to a specific API endpoint on the server.

[0964] Input: User-entered data

[0965] Data transformation: Converting user-entered data into an HTTP POST request

[0966] Output: Request sent to server

[0967] Step 3:

[0968] The server receives the request and performs a database lookup.

[0969] The server receives the request from the device and queries the sign language database based on the keyword "thank you" to retrieve relevant information. The query searches for information such as the sign gesture, background, region, era, and meaning.

[0970] Input: HTTP POST request (search keyword "Thank you", region information "Tokyo", age information "30-40 years old")

[0971] Data manipulation: performing database queries

[0972] Output: Search result data (sign language information)

[0973] Step 4:

[0974] The server formats the search results and returns them to the device

[0975] The server formats the search results into JSON format, converts them into the appropriate format, and returns them to the terminal.

[0976] Input: Search result data

[0977] Data processing: Conversion to JSON format

[0978] Output: Formatted search results data

[0979] Step 5:

[0980] The device receives the search results and displays them on the user interface.

[0981] The device receives the formatted data returned from the server, analyzes it, and displays it on the user interface, allowing the user to visually check the various variations of sign language and their meanings and backgrounds.

[0982] Input: Formatted search result data

[0983] Data Calculation: Data Analysis

[0984] Output: User interface display of search results

[0985] The processing steps of the system have been described above in detail.

[0986] (Application example 1)

[0987] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0988] Learning sign language and using it efficiently in specific tasks is difficult because the background and meaning of sign language vary by region and age. Additionally, certain tasks, such as food delivery, require the accurate use of sign language when communicating with hearing-impaired customers, but there are a lack of ways for employees to quickly learn that sign language. To solve this issue, a system is needed that allows employees to efficiently learn and use sign language related to specific tasks in a timely manner.

[0989] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0990] In this invention, the server includes means for receiving user input data, means for constructing a sign language database and storing additional information related to each sign language, means for searching the sign language database for corresponding sign language information based on the user input data, means for visually displaying the search results to the user, and means for enabling efficient learning and use of sign languages ​​in specific tasks, thereby enabling accurate learning and rapid use of sign languages.

[0991] A "means for receiving user input" is the interface a user uses to provide information about a particular sign language, including speech, keyboard input, touch, etc.

[0992] "Means for building a database of sign languages ​​and storing additional information related to each sign language" refers to a system or server for systematically storing and managing detailed information such as sign language gestures, regional information, historical information, and their background and meanings.

[0993] "Means for searching for corresponding sign language information from a sign language database based on user input data" refers to algorithms or processes for efficiently searching and extracting relevant sign language information from a sign language database based on keywords, regional information, and era information provided by the user.

[0994] "Means for visually displaying search results to the user" refers to a graphical user interface or display device that displays the searched sign language information in a format that is easy for the user to understand.

[0995] "Means to enable efficient learning and use of sign language in specific jobs" refers to training modules and guidelines to support employees in specific jobs, such as food delivery, in quickly learning the necessary sign language and using it effectively in their work.

[0996] "When an employee performing a specific task uses sign language, collect regional and age information and use it to search for sign language information" refers to the process of inputting regional and age information and selecting the most appropriate sign language based on that information in order to provide the appropriate sign language according to the context when the employee uses sign language.

[0997] "Additional information related to the meaning or trigger of sign language" is supplementary information for understanding sign language, such as the situations in which a particular sign language is used, its background and history, etc.

[0998] This invention provides a system that enables employees of food delivery services to easily learn and use sign language. This system mainly consists of the following components:

[0999] Overall system configuration

[1000] This system mainly consists of the following five main components:

[1001] 1. A means of receiving user input data

[1002] 2. A means of building a database of sign languages ​​and storing additional information associated with each sign language.

[1003] 3. A means to search for corresponding sign language information from a sign language database based on user input data

[1004] 4. A means of visually displaying search results to the user

[1005] 5. Means of enabling people to learn and use sign language efficiently in specific jobs

[1006] Handling User Input

[1007] Device:

[1008] Users use their smartphones to input information about a specific sign language. For example, if a delivery person wants to know the sign for "thank you," they can enter "thank you" into the application's search bar. At the same time, the user's region and age information are also collected. This information is important for identifying regional and age-specific variations in sign language.

[1009] Submitting a search request

[1010] Device:

[1011] The device sends the user's input data, along with regional and historical information, to the server, which is used to improve search accuracy and identify sign language variations.

[1012] Database search and result formatting

[1013] server:

[1014] The server receives a request from a user and searches for the corresponding sign language information from a sign language database. Specifically, it retrieves each sign language variation and its additional information based on the input keyword. For example, for the sign expression for "thank you," it searches for sign language variations for the Tokyo area and the 30-40 age group, and also retrieves the meaning and background information of each.

[1015] Displaying search results

[1016] Device:

[1017] The terminal analyzes the search results received from the server and displays them on the user interface. The display includes information about the specific sign language gestures and their "cues" and "meanings," making it easier for employees to understand the background and meaning of the sign language.

[1018] Specific examples

[1019] As a specific use case, consider a food delivery person who wants to learn the sign language for "thank you." The delivery person enters "thank you" into the application's search bar and presses the search button. At this time, the device sends the user's input data, along with region information (e.g., Tokyo) and age information (e.g., 30-40 years old) to the server. The server then searches the database for sign language information related to "thank you" and obtains detailed variations in sign language for different regions and age groups. The search results are formatted and sent back to the device, where the delivery person can view this information on the user interface. This allows the delivery person to understand the sign language and its meaning specific to their region and age group, and use sign language appropriately in their actual delivery work.

[1020] Prompt Sentence Examples

[1021] An example of a prompt to input to a generative AI model is as follows:

[1022] "We will create a food delivery application program for the hearing impaired. It will run on smartphones and have a function that allows users to learn sign language efficiently. When a user enters keywords for the sign language for "thank you" and provides region and era information, the corresponding sign language instructions will be displayed. Specifically, search results will be displayed that include detailed information such as sign language gestures, region, era, and meaning. Please use the following data processing flow. Input data: Keywords, region, era. Output data: Sign language gestures, region, era, meaning."

[1023] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1024] Step 1:

[1025] The user operates the delivery person app and inputs information about sign language. For example, if the delivery person wants to know the sign language for "thank you," they enter "thank you" in the application's search bar and press the search button. At this point, region information (e.g., Tokyo) and age information (e.g., 30-40 years old) are also entered at the same time. The input data is the keyword, region, and age. The output data is the information entered by the user, and the device retains this as input.

[1026] Step 2:

[1027] The device collects the user's input data and sends it to the server. The input data includes keywords, region, and age information. Based on this, the device sends a request to the server. This request includes the keywords entered by the user, such as "Thank you," as well as region and age information. The output data is the search request received by the server.

[1028] Step 3:

[1029] The server searches the sign language database based on the received search request. Specifically, it searches for entries in the database that match keywords such as "thank you," and also searches for associated regional and historical information. The input data is the search request, and the output data is the search results. The server processes this using a database search algorithm to obtain the corresponding sign language information and its additional information.

[1030] Step 4:

[1031] The server formats the acquired sign language information, prepares it in a format that can be sent, and sends it back to the user's device. The search results include detailed information such as the sign language gesture, region, era, and meaning. The input data are the search results, and the output data are the formatted search results. The server uses a data formatting algorithm to format the results and send them to the device.

[1032] Step 5:

[1033] The terminal analyzes the received search results and displays them visually. Specifically, it displays specific sign language gestures and information about their "cues" and "meanings" on the user interface. This makes it easier for users to understand the background and meaning of the sign language. The input data are the formatted search results, and the output data is the content displayed to the user. The terminal's analysis and display processing allows the delivery person to understand and use sign language accurately.

[1034] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1035] MODE FOR CARRYING OUT THE INVENTION

[1036] The present invention relates to a system for aiding in the understanding of sign language and fostering familiarity with sign language. In particular, the system provides a more personalized learning experience by incorporating an emotion engine that recognizes a user's emotions and dynamically adjusts the method and content of providing sign language information. Specific embodiments of the present invention are described below.

[1037] Overall system configuration

[1038] This system mainly consists of the following four main components:

[1039] 1. A device that incorporates an emotion engine to receive user input data and recognize emotions.

[1040] 2. A server that manages the sign language database and stores the sign language and additional information such as its "trigger" and "meaning."

[1041] 3. Users who search for, learn, and use information about sign language.

[1042] 4. A means for providing sign language information based on the analysis results of the emotion engine.

[1043] Sign Language Database

[1044] server:

[1045] The server manages a sign language database and stores detailed information about sign languages, including information about the gestures and actions of the sign language, as well as additional information such as the background, region, and era in which the sign language originated. The server efficiently searches and provides this information.

[1046] Emotion Engine

[1047] Device:

[1048] The emotion engine is a software module for analyzing the user's emotions in real time. It recognizes the user's emotional state by analyzing the user's facial expressions and tone of voice through a camera and microphone. Based on this information, the system provides sign language information that matches the user's current emotion.

[1049] Handling User Input

[1050] User:

[1051] The user inputs information about a specific sign language through the device. For example, to find out about the sign for "thank you," the user enters "thank you" into the search bar. The emotion engine then detects the user's facial expressions and voice and analyzes their emotional state.

[1052] Submitting a search request

[1053] Device:

[1054] The device sends a request to the server containing the user's input data and the emotion information recognized by the emotion engine, including information about the region and age.

[1055] Database search and result formatting

[1056] server:

[1057] The server analyzes the received request and searches for the corresponding sign language information from the sign language database. Taking into account the results of the emotion engine, the server adjusts the priority of the sign language information appropriate to the user's emotion. For example, if the user is relaxed, it provides more detailed information, but if the user is feeling impatient, it prioritizes concise information.

[1058] Displaying search results

[1059] Device:

[1060] The device analyzes the search results received from the server and adjusts the display content based on the emotional information obtained from the emotion engine. For example, it displays messages emphasizing success to users who show positive emotions, and adds encouraging messages to users who show negative emotions.

[1061] Specific examples

[1062] Consider a specific use case. When a user searches for the sign language for "thank you," they enter "thank you" in the search bar and press the search button. At this time, the emotion engine analyzes the user's facial expressions and tone of voice and recognizes that the user is relaxed.

[1063] The device sends a request containing the user's input data, region information, age information, and emotion information to the server. The server searches for sign language information corresponding to "thank you" and, taking into account the results of the emotion engine, provides detailed sign language information that is appropriate for when the user is relaxed. It also adds encouraging and positive feedback as support messages as needed.

[1064] The device analyzes the received search results and displays them on the user interface, visually displaying specific sign gestures and their "cues" and "meanings." This helps users understand the background and meaning of sign language, and by providing emotional feedback during the learning process, it increases interest in and familiarity with sign language.

[1065] The above is a specific description of an embodiment of the present invention. This system provides the diversity of sign languages ​​and their background information, and also provides a personalized sign language learning experience that responds to the user's emotions.

[1066] The processing flow will be explained below.

[1067] MODE FOR CARRYING OUT THE INVENTION

[1068] The processing flow of the system based on the present invention will be specifically explained by dividing it into the following steps.

[1069] Step 1: Receiving User Input

[1070] User:

[1071] If you want to find information about sign language, enter a keyword in the search bar, for example, "thank you."

[1072] Step 2: Emotion analysis using the emotion engine

[1073] Device:

[1074] The emotion engine is activated and analyzes the user's facial expressions and voice via the camera and microphone, recognizing their emotional state in real time, such as whether they are relaxed or anxious.

[1075] Step 3: Collect user information

[1076] Device:

[1077] Collects user input data (keywords), emotional information, regional information (e.g., identifying "Tokyo" using GPS), and age information (e.g., identifying "30-40 years old" from the profile).

[1078] Step 4: Create and submit a request

[1079] Device:

[1080] The request data is created based on the collected information and sent to the server. The request contains the following data: { keyword: 'Thank you', region: 'Tokyo', age_group: '30-40', emotion: 'Relax'}.

[1081] Step 5: Receiving and Parsing the Request

[1082] server:

[1083] The received request data is analyzed to identify the sign language the user wants to search for and related information. Emotional information is also used in the analysis.

[1084] Step 6: Retrieving information from the database

[1085] server:

[1086] Searches for sign language information and additional information that matches the entered keywords from a sign language database. For example, sign language variations related to "thank you" and their "trigger" and "meaning."

[1087] Step 7: Formatting the search results

[1088] server:

[1089] It shapes search results, adjusting the level of detail and content of the information provided based on the user's emotional state, for example prioritizing detailed information for a relaxed user and concise information for a rushed user.

[1090] Step 8: Submit search results

[1091] server:

[1092] The formatted search results are sent back to the device. The returned data can be in the following format: "{ sign: 'Thank you', variations: [{ region: 'Kansai', gesture: 'Circle your thumb and index finger', meaning: 'Expresses gratitude'}, { region: 'Tokyo', gesture: 'Put your palms together', meaning: 'Means thank you'}], message: 'You are relaxed. Show more information.'}".

[1093] Step 9: Parse and display search results

[1094] Device:

[1095] The search results received from the server are analyzed and displayed on the user interface. At this time, appropriate encouragement and feedback messages are also displayed based on the analysis results of the emotion engine. For example, a message such as "You are relaxed. Please check for more information" may be displayed.

[1096] Step 10: Verify the user and request additional information

[1097] User:

[1098] View the displayed sign language information, its supplementary information, and emotion-based feedback. If necessary, you can request more information. For example, if you want to learn more about the historical background of the sign language, you can click the button for more information.

[1099] The above are the specific program processing steps of an embodiment of the present invention, which combines an emotion engine that recognizes the user's emotions. This system provides the diversity of sign languages ​​and their background information, enabling a personalized sign language learning experience that responds to the user's emotions.

[1100] Example 2

[1101] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1102] Conventional systems for supporting sign language learning rely on user input data and basic search functions, and are unable to provide a personalized learning experience that takes into account the user's emotional state. This makes it difficult to elicit the user's interest and affinity and promote effective learning. Furthermore, optimization of search and provided information that takes into account the user's region and age is also insufficient. To solve this problem, a system is needed that analyzes the user's emotional state in real time and provides optimal sign language information based on that analysis.

[1103] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving input data and emotional state of a user, means for constructing a sign language database and storing additional information related to each sign language, means for searching for corresponding sign language information from the sign language database based on the user's input data and emotional information, and means for displaying the search results and the emotional information in a visually and emotionally adjusted form to the user. This makes it possible to provide optimal sign language information according to the user's emotional state, thereby realizing a personalized learning experience. Furthermore, by taking into account regional and era information, it is possible to provide more accurate information.

[1104] "User-entered data" refers to information entered by a user using a terminal, such as keywords and phrases related to sign language searches.

[1105] "Emotional state" refers to a psychological state analyzed from the user's facial expression, tone of voice, etc., and is classified into categories such as relaxation, impatience, and joy.

[1106] A "sign language database" refers to a database that constructs and stores data including sign language gestures, meanings, background information, etc.

[1107] "Additional information" is detailed information related to a sign language, including additional information such as the background, region, and era in which the sign language originated.

[1108] The "search means" refers to a function that searches for corresponding sign language information from a sign language database based on the user's input data and emotion information.

[1109] "Visual display" refers to a method of visually presenting sign language information to a user on a screen, including sign language gestures and additional information.

[1110] "Emotional adjustment" refers to a method of adjusting the priority of information based on the user's emotional state and providing the information to the user in an optimal manner.

[1111] "Region information" is information about the user's current location, and is used for filtering when searching for sign language information.

[1112] "Age information" is information about the user's age and birth decade, and is used for filtering when searching for sign language.

[1113] This invention is a system for providing a personalized experience for users learning sign language. The system has the ability to analyze the user's emotional state in real time and dynamically provide sign language information based on that analysis.

[1114] Overall system configuration

[1115] 1. Receiving user input data

[1116] User: The user enters the sign language keyword they want to search for in the search bar of their device. For example, if they want to learn the sign language for "thank you," they enter "thank you" in the search bar and click the search button.

[1117] 2. Acquiring emotional information

[1118] Device: The device uses a built-in camera and microphone to capture the user's facial expressions and tone of voice, which are then analyzed by an emotion engine. This data is then classified as the user's emotional state (e.g., relaxed, anxious, happy, etc.).

[1119] 3. Generating a search request

[1120] Device: The device generates a search request that includes the keywords and emotion information entered by the user, as well as location and age information. The generated request is sent to the server in the next step.

[1121] 4. Sending a request to the server

[1122] Device: The device generates a search request and sends it to the server, using an internet connection to transfer data using HTTP or HTTPS protocols.

[1123] 5. Database Search

[1124] Server: The server analyzes the received request and searches for the corresponding sign language information from a sign language database, which contains information about the gestures, meanings, and context of the sign language. The search results are prioritized based on the emotional information.

[1125] 6. Formatting Sign Language Information

[1126] Server: The server formats the search results and generates a response in a user-friendly format. If the user is relaxed, the response will contain more information.

[1127] 7. Receiving and Displaying Search Results

[1128] Device: The device analyzes the search results received from the server and displays them in a visually and emotionally tailored way to the user's emotions. For users who show positive emotions, it displays messages that highlight success, and for users who show negative emotions, it adds encouraging messages.

[1129] Specific use cases

[1130] As a specific example, consider the case where a user wants to search for the sign language for "thank you." The user types "thank you" into the search bar and presses the search button. At this time, the device's emotion engine analyzes the user's facial expressions and voice and recognizes that the user is relaxed.

[1131] The device generates a request including the input data "Thank you," region information, age information, and emotional information "Relax," and sends it to the server. The server searches for the corresponding sign language information in a sign language database, prioritizes providing detailed information, and also generates a response including positive feedback.

[1132] The device displays the received search results in a user interface. For example, not only are the sign gestures, background information, and meanings visually displayed, but detailed information such as "This sign means thank you and is used in these situations" is also provided. Furthermore, because the user is relaxed, a positive feedback message is also displayed.

[1133] Example of input prompt for generative AI model

[1134] "Suppose a user wants to know the sign for 'thank you'. You have emotional information that the user is relaxed. In this case, provide detailed information about the sign, including the gesture, background information, and geographic and historical information."

[1135] In this way, a sign language learning support system that reflects emotional information can provide users with a personalized learning experience.

[1136] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1137] Step 1: Receiving User Input

[1138] User:

[1139] The user enters the sign language keyword they want to search for in the search bar of their device and presses the search button. For example, if they want to search for the sign language for "thank you," they enter "thank you" in the search bar.

[1140] Input: Keyboard input ("Thank you")

[1141] Output: Keyword data ("Thank you")

[1142] Specific behavior:

[1143] The user enters keywords into the device's search bar and clicks the "Search" button.

[1144] Step 2: Acquiring emotional information

[1145] Device:

[1146] The device uses a built-in camera and microphone to capture the user's facial expressions and tone of voice, which are then analyzed by an emotion engine.

[1147] Input: Camera video data, microphone audio data

[1148] Output: Emotion data (e.g. "Relaxed")

[1149] Specific behavior:

[1150] The camera captures the user's face and the microphone records the user's voice.

[1151] An emotion engine analyzes this data to determine the user's emotional state.

[1152] Step 3: Generate a search request

[1153] Device:

[1154] The terminal generates a search request by combining the keywords, emotional information, and regional and age information entered by the user.

[1155] Input: Keyword data ("Thank you"), emotion data ("Relax"), region information, age information

[1156] Output: Search request data

[1157] Specific behavior:

[1158] Based on the input data, the data is formatted into a search request format.

[1159] Create request data including region and age information.

[1160] Step 4: Sending a request to the server

[1161] Device:

[1162] The terminal transmits the generated search request to the server.

[1163] Input: Search request data

[1164] Output: HTTP request to the server

[1165] Specific behavior:

[1166] Using the HTTP protocol, a request is sent to the specified URL on the server using the POST method.

[1167] Step 5: Database Search

[1168] server:

[1169] The server analyzes the received request and searches for the corresponding sign language information from the sign language database.

[1170] Input: Search request data

[1171] Output: Sign language information data

[1172] Specific behavior:

[1173] A database query is generated and searched against a sign language database.

[1174] Take search results and adjust priorities based on sentiment information.

[1175] Step 6: Formatting the sign language information

[1176] server:

[1177] The server formats the search results and prepares the sign language information in a user-friendly format.

[1178] Input: Sign language information data

[1179] Output: Formatted display data

[1180] Specific behavior:

[1181] Format sign language gestures, meanings and related information.

[1182] Add positive feedback messages based on emotions.

[1183] Step 7: Receiving and displaying search results

[1184] Device:

[1185] The terminal analyzes the search results received from the server and generates content to be displayed to the user.

[1186] Input: Formatted display data

[1187] Output: User interface display

[1188] Specific behavior:

[1189] The data received from the server is analyzed and sign language gestures and background information are displayed on the user interface.

[1190] It also displays emotion-based feedback messages.

[1191] (Application example 2)

[1192] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1193] There is a need to improve and streamline communication within factories, but there are challenges such as difficulty in communicating smoothly with hearing-impaired employees and responding to employees' emotions in stressful environments. In particular, when communicating using sign language, it is difficult to provide information that corresponds to the emotional state of employees, and an effective system for this purpose is needed.

[1194] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1195] In this invention, the server includes means for receiving user input data, means for building a sign language database and storing additional information related to each sign language, emotion recognition means using an emotion engine that analyzes the user's emotions in real time, means for searching the sign language database for corresponding sign language information based on the user's input data and emotion information, and means for visually displaying the search results in a form that corresponds to the user's emotional state. This enables smooth communication using sign language in factory communications and personalized support that corresponds to the emotions of employees.

[1196] The "means for receiving user input data" is a hardware or software mechanism for obtaining information provided by a user.

[1197] "Means for building a sign language database and storing additional information related to each sign language" refers to a method for comprehensively organizing and storing information about sign languages, as well as their background and meanings.

[1198] The "emotion recognition means" is a mechanism for analyzing the user's emotional state in real time and obtaining the results.

[1199] The "means for searching for corresponding sign language information from a sign language database" is a method for searching for appropriate sign language information from a sign language database based on the user's input data and emotion information.

[1200] "Visual display means" refers to the method of presenting search results to the user through a visual interface such as a display or screen.

[1201] "Means for collecting regional and generational information" refers to methods for acquiring and accumulating data relating to the geographical location and generation of users.

[1202] The "means for dynamically personalizing and providing additional information" is a mechanism for individually optimizing and providing additional information about sign language based on the user's emotional state and other factors.

[1203] The present invention provides a sign language compatible robot assistant system for realizing efficient communication within a factory. Specific embodiments are described below.

[1204] Overall system configuration

[1205] The system mainly consists of the following main components:

[1206] 1. Hardware and software for receiving user input data

[1207] 2. A server that builds a sign language database and stores additional information related to each sign language.

[1208] 3. A device that uses an emotion engine to analyze user emotions in real time

[1209] 4. A device that visually displays search results in a format that corresponds to the user's emotional state

[1210] Receiving user input data

[1211] User input data is collected by a device equipped with a camera and microphone, with the Python OpenCV library installed to capture video frames and collect audio data.

[1212] Sign Language Database

[1213] The server manages a sign language database and stores detailed information about sign languages. This includes information about the gestures and actions of the sign language, as well as additional information such as the background, region, and era in which the sign language originated. The server uses the Python sign_language_database module to efficiently search and provide this information.

[1214] Emotion Engine

[1215] The emotion engine is a software module for analyzing the user's emotions in real time. This module uses a library called EmotionRecognition to analyze the user's facial expressions and tone of voice. The level of detail of the information provided changes depending on whether the user is feeling stressed or not.

[1216] Processing user input data and emotional information

[1217] The device sends the user's input data and the emotional information recognized by the emotion engine together to the server. The server analyzes this request and searches the database for corresponding sign language information. The search results are returned by the server with either a detailed explanation or a concise explanation selected based on the user's emotional state.

[1218] Displaying search results

[1219] The device analyzes the search results received from the server and adjusts the display content based on the emotional information obtained from the emotion engine. For example, if the device detects that the user is feeling stressed, it will prioritize displaying a concise explanation.

[1220] Specific examples

[1221] If a user signs "OK," the emotion engine recognizes that the user is feeling stressed. The device sends this information to the server, which searches for the sign information corresponding to "OK" and returns a brief explanation. It also adds positive feedback as a support message. The device visually displays this to the user, helping them to reduce their stress.

[1222] Prompt Sentence Examples

[1223] Examples of input prompts for a generative AI model might include:

[1224] "Recognize sign language used in factories and provide dynamic support according to employees' emotions."

[1225] This will enable smooth communication within the factory using sign language and personalized support that responds to employees' emotions.

[1226] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1227] Step 1:

[1228] The user signs.

[1229] Input: User's sign language actions, facial expressions, and voice

[1230] Output: Video frames and audio data captured by the camera and microphone

[1231] Specific operation: When a user signs, the device's camera records the sign and the microphone collects the audio.

[1232] Step 2:

[1233] The device processes the input data to analyze the user's emotions in real time.

[1234] Input: Video frames and audio data

[1235] Output: User's emotional state (e.g., relaxed, stressed, etc.)

[1236] How it works: The device uses the EmotionRecognition library to analyze facial expressions from video frames and tone of voice from audio data, and then determines the user's emotional state based on the results of this analysis.

[1237] Step 3:

[1238] The device recognizes sign language.

[1239] Input: Video frame

[1240] Output: Sign language commands (e.g. OK, Thank you, etc.)

[1241] Specific actions: The device uses a sign language recognition algorithm to analyze the sign language actions in the video frames and extract specific sign language commands.

[1242] Step 4:

[1243] The terminal transmits the user's input data and emotion information to the server.

[1244] Input: sign language commands, emotional states, video frames, audio data

[1245] Output: Request data to the server

[1246] Specific operation: The terminal assembles the user's input data (sign language commands, video frames, and voice data) and emotional state into packets and sends them to the server.

[1247] Step 5:

[1248] The server searches for the corresponding sign language information from the sign language database.

[1249] Input: Request data

[1250] Output: Search results (sign meaning, trigger, additional information, etc.)

[1251] Specific operation: The server receives the request data and queries the sign language database, searching for sign language information corresponding to the sign language command and adjusting the level of detail of the information according to the emotional state.

[1252] Step 6:

[1253] The server sends the search results to the terminal.

[1254] Input: Search results

[1255] Output: Response data to the terminal

[1256] Specific operation: The server organizes the search results, converts them into an appropriate format, and sends them to the terminal.

[1257] Step 7:

[1258] The terminal visually displays the search results to the user.

[1259] Input: Response data

[1260] Output: What is displayed on the user interface

[1261] Specific operation: The device receives the response data and adjusts the display content according to the user's emotional state, for example, displaying a brief explanation to a stressed user and providing detailed information to a relaxed user.

[1262] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1263] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1264] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1265] [Fourth embodiment]

[1266] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1267] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1268] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1269] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1270] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1271] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1272] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1273] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1274] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1275] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1276] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1277] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1278] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1279] MODE FOR CARRYING OUT THE INVENTION

[1280] The present invention relates to a system that provides "cues" and "meanings" related to sign language to help users understand sign language. This system provides a means for efficiently learning sign languages ​​that vary by region and generation. Specific embodiments of the present invention are described below.

[1281] Overall system configuration

[1282] This system mainly consists of the following three main components:

[1283] 1. A server that manages the sign language database and stores additional information on sign languages ​​and their "cues" and "meanings."

[1284] 2. A terminal that accepts user input, sends requests to a server, and displays the results.

[1285] 3. Users who search for, learn, and use information about sign language.

[1286] Sign Language Database

[1287] server:

[1288] A sign language database systematically stores multiple pieces of information about sign languages. This includes not only information about the gestures and actions of sign languages, but also additional information such as the background, region, and era in which the sign language originated. The server manages this information and returns appropriate data in response to user search requests.

[1289] Handling User Input

[1290] User:

[1291] Users input information about a particular sign language through their device. For example, to find out about the sign for "thank you," they type "thank you" into the search bar. Information about the user's region and age is also collected. This information is important for identifying regional and age-specific variations in sign language.

[1292] Submitting a search request

[1293] Device:

[1294] The device receives the user's input data and sends a request for sign language information to the server based on that data. The request includes not only the keywords entered by the user, but also regional and historical information. This data is used to improve the accuracy of the search.

[1295] Database search and result formatting

[1296] server:

[1297] The server receives a user request and searches for the corresponding sign language information from a sign language database. Specifically, it retrieves each sign language variation and its additional information based on the input keywords. The search results are then formatted appropriately before being returned to the user. This formatted data includes details such as the sign gesture, the region where it originated, the era, and its meaning.

[1298] Displaying search results

[1299] Device:

[1300] The device analyzes the search results received from the server and displays them on the user interface. The display includes information about specific sign language gestures and their "cues" and "meanings," making it easier for users to understand the background and meaning of the sign language.

[1301] Specific examples

[1302] As a specific example, consider the case where a user searches for the sign language for "thank you." The user enters "thank you" in the search bar and presses the search button. At this time, the device sends the user's input data, along with region information (e.g., Tokyo) and age information (e.g., 30-40 years old) to the server.

[1303] The server searches the database for sign language information related to "thank you," searching for different variations such as "in the Kansai region, the thumb and index finger form a circle" and "in Tokyo, the palms are placed together," and obtaining the meaning and background information for each.

[1304] The search results are formatted and sent back to the device, where they are displayed on the user interface. Users can visually check the sign language and its meanings specific to each region and era. This helps users understand the diversity of sign language and stimulates their interest in sign language.

[1305] The embodiment of the present invention has been specifically described above. This system makes it easier to understand the diversity of sign languages ​​and makes people feel closer to sign languages.

[1306] The processing flow will be explained below.

[1307] Program processing steps

[1308] Step 1: Receiving User Input

[1309] User:

[1310] If you want to find information about sign language, enter a keyword in the search bar, for example, "thank you."

[1311] Step 2: Collect user information

[1312] Device:

[1313] It receives user input data (keywords) and also collects the user's location information (e.g., identifying "Tokyo" using GPS) and age information (e.g., identifying "30-40 years old" from the profile).

[1314] Step 3: Create and submit a request

[1315] Device:

[1316] The collected user input data, region information, and age information are sent to the server as a request. The request format is as follows: { keyword: 'Thank you', region: 'Tokyo', age_group: '30-40'}.

[1317] Step 4: Receiving and Parsing the Request

[1318] server:

[1319] The received request data is analyzed to identify the sign language the user wants to search for and its related information.

[1320] Step 5: Retrieving information from the database

[1321] server:

[1322] The system searches the sign language database for sign language information and additional information that matches the keywords in the request. Specifically, it searches for sign language variations related to "thank you" and their respective "trigger" and "meaning."

[1323] Step 6: Formatting the search results

[1324] server:

[1325] The search results are formatted and converted into a format to be returned to the user. For example, in JSON format, they are formatted as "{ sign: 'Thank you', variations: [{ region: 'Kansai', gesture: 'Make a circle with your thumb and index finger', meaning: 'Convey gratitude'}, { region: 'Tokyo', gesture: 'Put your palms together', meaning: 'Meaning of gratitude'}]}".

[1326] Step 7: Submit search results

[1327] server:

[1328] The formatted search results are returned to the terminal. The returned data is in a format that has already been formatted.

[1329] Step 8: Parse and display search results

[1330] Device:

[1331] The search results received from the server are analyzed and displayed on the user interface, visually displaying specific sign language gestures and information about their "cues" and "meanings."

[1332] Step 9: Verify the user and request additional information

[1333] User:

[1334] You can review the displayed sign language information and request more detailed information if necessary. For example, if you want to learn more about the development and historical background of the sign language, click the button for more information.

[1335] The above are the specific program processing steps in the embodiment of the present invention, which provide users with information about the diversity of sign languages ​​and their background information, thereby increasing their understanding and interest in sign languages.

[1336] Example 1

[1337] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1338] While there are many systems to promote the learning and understanding of sign language, few systems efficiently and accurately provide the different sign language variations by region and age. Furthermore, conventional systems have difficulty providing detailed background information and meanings of sign language. Therefore, there is a need for a system that allows users to easily obtain sign language information according to specific conditions and understand its meaning and background.

[1339] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1340] In this invention, the server includes means for receiving a user's search keywords and regional and era information, means for managing a database that systematically stores various information related to sign language, means for searching the database for information such as gestures, background, region, era, and meaning of the relevant sign language based on data entered by the user, means for formatting the search results in an appropriate format such as JSON and returning the results, and means for analyzing the search results received from the server and displaying them on a user interface. This enables a user to efficiently and accurately obtain sign language information tailored to a specific region or era and understand its background and meaning.

[1341] 1. "Means" refers to methods or devices used to achieve a particular purpose.

[1342] 2. "Search Keywords" refers to words or phrases that users enter to obtain specific sign language information.

[1343] 3. "Region information" refers to geographic information where a user lives or is associated, and is used to identify sign language variations.

[1344] 4. "Date information" refers to the age of a user or temporal information necessary to identify a sign language variation of a particular era.

[1345] 5. "Database" refers to a collection of information for the systematic storage and management of information related to sign languages.

[1346] 6. "Gesture" refers to the specific movements of the hands and body when signing.

[1347] 7. "Context" refers to information about how a particular sign language originated and what its cultural and historical background is.

[1348] 8. "Meaning" refers to the content or intent that a particular sign language is intended to convey to the user.

[1349] 9. "JSON format" refers to a lightweight data exchange format for representing data in a structured manner.

[1350] 10. "User interface" refers to the display and input devices that allow a user to directly interact with a system.

[1351] 11. "Parsing" refers to the process of understanding data received from the server and converting it into a user-friendly format.

[1352] MODE FOR CARRYING OUT THE INVENTION

[1353] The present invention relates to a system that allows users to learn sign language efficiently and accurately. To implement this system, a server that manages a sign language database and provides information based on user search requests, a terminal for users to perform searches, and users who actually learn sign language are required.

[1354] Server Configuration

[1355] The server has a database that systematically stores and manages various information related to sign language. For example, a relational database such as MySQL can be used. The database includes information such as sign language gestures, background, region, era, and meaning. Based on a search request received from a user, the server searches for the relevant sign language information, formats it in an appropriate format (e.g., JSON format), and returns it.

[1356] Device configuration

[1357] The terminal provides an interface for the user to input search keywords, region information, and era information. For example, a PC, smartphone, tablet, or other device can be used. The terminal receives the data entered by the user and sends it to the server in the form of an HTTP POST request, for example. The terminal analyzes the data returned from the server and displays it on the user interface.

[1358] User operations

[1359] Users enter keywords related to a specific sign language into the device's search bar. For example, if they want to find the sign for "thank you," they enter this keyword. In addition, the system also collects information about the user's region and age. For example, "Tokyo" or "30-40 years old." This information is important for improving search accuracy.

[1360] Specific example explanation

[1361] As a specific example of use, consider the case where a user searches for the sign for "thank you."

[1362] The user enters "Thank you" in the search bar of the device and presses the search button. The device sends the user's input data, region information (e.g., Tokyo), and age information (e.g., 30-40 years old) to the server.

[1363] The server searches the sign language database for sign language information related to "thank you." For example, different variations such as "in the Kansai region, the thumb and index finger form a circle" and "in Tokyo, the palms are placed together" are searched for, and the meanings and background information of each are also obtained. These search results are then formatted and sent back to the device.

[1364] The device analyzes the search results and displays them on the user interface. Users can visually check the sign language and its meanings specific to each region and era, which helps them understand the diversity of sign language and stimulates their interest in sign language.

[1365] Prompt Sentence Examples

[1366] Below are some examples of specific prompt sentences.

[1367] If a user wants to find out about the sign language for "thank you," they type "thank you" into the search bar and press the search button. The device sends the user's input data, along with region information (e.g., Tokyo) and age information (e.g., 30-40 years old), to the server. The server searches the sign language database for information related to "thank you," and obtains different variations, such as a circular thumb and index finger in the Kansai region and a palm-to-palm gesture in Tokyo. The server also provides information about the meaning and background of each variation. The formatted search results are sent back to the device and displayed on the user interface.

[1368] The above describes a specific embodiment of the present invention. This system allows users to efficiently learn the diversity of sign languages ​​and gain a deeper understanding of their backgrounds and meanings.

[1369] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1370] Specific flow of program processing

[1371] Step 1:

[1372] The user enters search keywords into the device

[1373] The user enters a specific sign language keyword into the device's search bar, for example, "thank you," along with region information (e.g., Tokyo) and age information (e.g., 30-40 years old).

[1374] Input: Search keyword "Thank you", region information "Tokyo", age information "30-40 years old"

[1375] Output: User input data

[1376] Step 2:

[1377] The device receives user input data and sends it to the server

[1378] The terminal creates an HTTP POST request based on the received user input data and sends it to a specific API endpoint on the server.

[1379] Input: User-entered data

[1380] Data transformation: Converting user-entered data into an HTTP POST request

[1381] Output: Request sent to server

[1382] Step 3:

[1383] The server receives the request and performs a database lookup.

[1384] The server receives the request from the device and queries the sign language database based on the keyword "thank you" to retrieve relevant information. The query searches for information such as the sign gesture, background, region, era, and meaning.

[1385] Input: HTTP POST request (search keyword "Thank you", region information "Tokyo", age information "30-40 years old")

[1386] Data manipulation: performing database queries

[1387] Output: Search result data (sign language information)

[1388] Step 4:

[1389] The server formats the search results and returns them to the device

[1390] The server formats the search results into JSON format, converts them into the appropriate format, and returns them to the terminal.

[1391] Input: Search result data

[1392] Data processing: Conversion to JSON format

[1393] Output: Formatted search results data

[1394] Step 5:

[1395] The device receives the search results and displays them on the user interface.

[1396] The device receives the formatted data returned from the server, analyzes it, and displays it on the user interface, allowing the user to visually check the various variations of sign language and their meanings and backgrounds.

[1397] Input: Formatted search result data

[1398] Data Calculation: Data Analysis

[1399] Output: User interface display of search results

[1400] The processing steps of the system have been described above in detail.

[1401] (Application example 1)

[1402] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1403] Learning sign language and using it efficiently in specific tasks is difficult because the background and meaning of sign language vary by region and age. Additionally, certain tasks, such as food delivery, require the accurate use of sign language when communicating with hearing-impaired customers, but there are a lack of ways for employees to quickly learn that sign language. To solve this issue, a system is needed that allows employees to efficiently learn and use sign language related to specific tasks in a timely manner.

[1404] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1405] In this invention, the server includes means for receiving user input data, means for constructing a sign language database and storing additional information related to each sign language, means for searching the sign language database for corresponding sign language information based on the user input data, means for visually displaying the search results to the user, and means for enabling efficient learning and use of sign languages ​​in specific tasks, thereby enabling accurate learning and rapid use of sign languages.

[1406] A "means for receiving user input" is the interface a user uses to provide information about a particular sign language, including speech, keyboard input, touch, etc.

[1407] "Means for building a database of sign languages ​​and storing additional information related to each sign language" refers to a system or server for systematically storing and managing detailed information such as sign language gestures, regional information, historical information, and their background and meanings.

[1408] "Means for searching for corresponding sign language information from a sign language database based on user input data" refers to algorithms or processes for efficiently searching and extracting relevant sign language information from a sign language database based on keywords, regional information, and era information provided by the user.

[1409] "Means for visually displaying search results to the user" refers to a graphical user interface or display device that displays the searched sign language information in a format that is easy for the user to understand.

[1410] "Means to enable efficient learning and use of sign language in specific jobs" refers to training modules and guidelines to support employees in specific jobs, such as food delivery, in quickly learning the necessary sign language and using it effectively in their work.

[1411] "When an employee performing a specific task uses sign language, collect regional and age information and use it to search for sign language information" refers to the process of inputting regional and age information and selecting the most appropriate sign language based on that information in order to provide the appropriate sign language according to the context when the employee uses sign language.

[1412] "Additional information related to the meaning or trigger of sign language" is supplementary information for understanding sign language, such as the situations in which a particular sign language is used, its background and history, etc.

[1413] This invention provides a system that enables employees of food delivery services to easily learn and use sign language. This system mainly consists of the following components:

[1414] Overall system configuration

[1415] This system mainly consists of the following five main components:

[1416] 1. A means of receiving user input data

[1417] 2. A means of building a database of sign languages ​​and storing additional information associated with each sign language.

[1418] 3. A means to search for corresponding sign language information from a sign language database based on user input data

[1419] 4. A means of visually displaying search results to the user

[1420] 5. Means of enabling people to learn and use sign language efficiently in specific jobs

[1421] Handling User Input

[1422] Device:

[1423] Users use their smartphones to input information about a specific sign language. For example, if a delivery person wants to know the sign for "thank you," they can enter "thank you" into the application's search bar. At the same time, the user's region and age information are also collected. This information is important for identifying regional and age-specific variations in sign language.

[1424] Submitting a search request

[1425] Device:

[1426] The device sends the user's input data, along with regional and historical information, to the server, which is used to improve search accuracy and identify sign language variations.

[1427] Database search and result formatting

[1428] server:

[1429] The server receives a request from a user and searches for the corresponding sign language information from a sign language database. Specifically, it retrieves each sign language variation and its additional information based on the input keyword. For example, for the sign expression for "thank you," it searches for sign language variations for the Tokyo area and the 30-40 age group, and also retrieves the meaning and background information of each.

[1430] Displaying search results

[1431] Device:

[1432] The terminal analyzes the search results received from the server and displays them on the user interface. The display includes information about the specific sign language gestures and their "cues" and "meanings," making it easier for employees to understand the background and meaning of the sign language.

[1433] Specific examples

[1434] As a specific use case, consider a food delivery person who wants to learn the sign language for "thank you." The delivery person enters "thank you" into the application's search bar and presses the search button. At this time, the device sends the user's input data, along with region information (e.g., Tokyo) and age information (e.g., 30-40 years old) to the server. The server then searches the database for sign language information related to "thank you" and obtains detailed variations in sign language for different regions and age groups. The search results are formatted and sent back to the device, where the delivery person can view this information on the user interface. This allows the delivery person to understand the sign language and its meaning specific to their region and age group, and use sign language appropriately in their actual delivery work.

[1435] Prompt Sentence Examples

[1436] An example of a prompt to input to a generative AI model is as follows:

[1437] "We will create a food delivery application program for the hearing impaired. It will run on smartphones and have a function that allows users to learn sign language efficiently. When a user enters keywords for the sign language for "thank you" and provides region and era information, the corresponding sign language instructions will be displayed. Specifically, search results will be displayed that include detailed information such as sign language gestures, region, era, and meaning. Please use the following data processing flow. Input data: Keywords, region, era. Output data: Sign language gestures, region, era, meaning."

[1438] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1439] Step 1:

[1440] The user operates the delivery person app and inputs information about sign language. For example, if the delivery person wants to know the sign language for "thank you," they enter "thank you" in the application's search bar and press the search button. At this point, region information (e.g., Tokyo) and age information (e.g., 30-40 years old) are also entered at the same time. The input data is the keyword, region, and age. The output data is the information entered by the user, and the device retains this as input.

[1441] Step 2:

[1442] The device collects the user's input data and sends it to the server. The input data includes keywords, region, and age information. Based on this, the device sends a request to the server. This request includes the keywords entered by the user, such as "Thank you," as well as region and age information. The output data is the search request received by the server.

[1443] Step 3:

[1444] The server searches the sign language database based on the received search request. Specifically, it searches for entries in the database that match keywords such as "thank you," and also searches for associated regional and historical information. The input data is the search request, and the output data is the search results. The server processes this using a database search algorithm to obtain the corresponding sign language information and its additional information.

[1445] Step 4:

[1446] The server formats the acquired sign language information, prepares it in a format that can be sent, and sends it back to the user's device. The search results include detailed information such as the sign language gesture, region, era, and meaning. The input data are the search results, and the output data are the formatted search results. The server uses a data formatting algorithm to format the results and send them to the device.

[1447] Step 5:

[1448] The terminal analyzes the received search results and displays them visually. Specifically, it displays specific sign language gestures and information about their "cues" and "meanings" on the user interface. This makes it easier for users to understand the background and meaning of the sign language. The input data are the formatted search results, and the output data is the content displayed to the user. The terminal's analysis and display processing allows the delivery person to understand and use sign language accurately.

[1449] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1450] MODE FOR CARRYING OUT THE INVENTION

[1451] The present invention relates to a system for aiding in the understanding of sign language and fostering familiarity with sign language. In particular, the system provides a more personalized learning experience by incorporating an emotion engine that recognizes a user's emotions and dynamically adjusts the method and content of providing sign language information. Specific embodiments of the present invention are described below.

[1452] Overall system configuration

[1453] This system mainly consists of the following four main components:

[1454] 1. A device that incorporates an emotion engine to receive user input data and recognize emotions.

[1455] 2. A server that manages the sign language database and stores the sign language and additional information such as its "trigger" and "meaning."

[1456] 3. Users who search for, learn, and use information about sign language.

[1457] 4. A means for providing sign language information based on the analysis results of the emotion engine.

[1458] Sign Language Database

[1459] server:

[1460] The server manages a sign language database and stores detailed information about sign languages, including information about the gestures and actions of the sign language, as well as additional information such as the background, region, and era in which the sign language originated. The server efficiently searches and provides this information.

[1461] Emotion Engine

[1462] Device:

[1463] The emotion engine is a software module for analyzing the user's emotions in real time. It recognizes the user's emotional state by analyzing the user's facial expressions and tone of voice through a camera and microphone. Based on this information, the system provides sign language information that matches the user's current emotion.

[1464] Handling User Input

[1465] User:

[1466] The user inputs information about a specific sign language through the device. For example, to find out about the sign for "thank you," the user enters "thank you" into the search bar. The emotion engine then detects the user's facial expressions and voice and analyzes their emotional state.

[1467] Submitting a search request

[1468] Device:

[1469] The device sends a request to the server containing the user's input data and the emotion information recognized by the emotion engine, including information about the region and age.

[1470] Database search and result formatting

[1471] server:

[1472] The server analyzes the received request and searches for the corresponding sign language information from the sign language database. Taking into account the results of the emotion engine, the server adjusts the priority of the sign language information appropriate to the user's emotion. For example, if the user is relaxed, it provides more detailed information, but if the user is feeling impatient, it prioritizes concise information.

[1473] Displaying search results

[1474] Device:

[1475] The device analyzes the search results received from the server and adjusts the display content based on the emotional information obtained from the emotion engine. For example, it displays messages emphasizing success to users who show positive emotions, and adds encouraging messages to users who show negative emotions.

[1476] Specific examples

[1477] Consider a specific use case. When a user searches for the sign language for "thank you," they enter "thank you" in the search bar and press the search button. At this time, the emotion engine analyzes the user's facial expressions and tone of voice and recognizes that the user is relaxed.

[1478] The device sends a request containing the user's input data, region information, age information, and emotion information to the server. The server searches for sign language information corresponding to "thank you" and, taking into account the results of the emotion engine, provides detailed sign language information that is appropriate for when the user is relaxed. It also adds encouraging and positive feedback as support messages as needed.

[1479] The device analyzes the received search results and displays them on the user interface, visually displaying specific sign gestures and their "cues" and "meanings." This helps users understand the background and meaning of sign language, and by providing emotional feedback during the learning process, it increases interest in and familiarity with sign language.

[1480] The above is a specific description of an embodiment of the present invention. This system provides the diversity of sign languages ​​and their background information, and also provides a personalized sign language learning experience that responds to the user's emotions.

[1481] The processing flow will be explained below.

[1482] MODE FOR CARRYING OUT THE INVENTION

[1483] The processing flow of the system based on the present invention will be specifically explained by dividing it into the following steps.

[1484] Step 1: Receiving User Input

[1485] User:

[1486] If you want to find information about sign language, enter a keyword in the search bar, for example, "thank you."

[1487] Step 2: Emotion analysis using the emotion engine

[1488] Device:

[1489] The emotion engine is activated and analyzes the user's facial expressions and voice via the camera and microphone, recognizing their emotional state in real time, such as whether they are relaxed or anxious.

[1490] Step 3: Collect user information

[1491] Device:

[1492] Collects user input data (keywords), emotional information, regional information (e.g., identifying "Tokyo" using GPS), and age information (e.g., identifying "30-40 years old" from the profile).

[1493] Step 4: Create and submit a request

[1494] Device:

[1495] The request data is created based on the collected information and sent to the server. The request contains the following data: { keyword: 'Thank you', region: 'Tokyo', age_group: '30-40', emotion: 'Relax'}.

[1496] Step 5: Receiving and Parsing the Request

[1497] server:

[1498] The received request data is analyzed to identify the sign language the user wants to search for and related information. Emotional information is also used in the analysis.

[1499] Step 6: Retrieving information from the database

[1500] server:

[1501] Searches for sign language information and additional information that matches the entered keywords from a sign language database. For example, sign language variations related to "thank you" and their "trigger" and "meaning."

[1502] Step 7: Formatting the search results

[1503] server:

[1504] It shapes search results, adjusting the level of detail and content of the information provided based on the user's emotional state, for example prioritizing detailed information for a relaxed user and concise information for a rushed user.

[1505] Step 8: Submit search results

[1506] server:

[1507] The formatted search results are sent back to the device. The returned data can be in the following format: "{ sign: 'Thank you', variations: [{ region: 'Kansai', gesture: 'Circle your thumb and index finger', meaning: 'Expresses gratitude'}, { region: 'Tokyo', gesture: 'Put your palms together', meaning: 'Means thank you'}], message: 'You are relaxed. Show more information.'}".

[1508] Step 9: Parse and display search results

[1509] Device:

[1510] The search results received from the server are analyzed and displayed on the user interface. At this time, appropriate encouragement and feedback messages are also displayed based on the analysis results of the emotion engine. For example, a message such as "You are relaxed. Please check for more information" may be displayed.

[1511] Step 10: Verify the user and request additional information

[1512] User:

[1513] View the displayed sign language information, its supplementary information, and emotion-based feedback. If necessary, you can request more information. For example, if you want to learn more about the historical background of the sign language, you can click the button for more information.

[1514] The above are the specific program processing steps of an embodiment of the present invention, which combines an emotion engine that recognizes the user's emotions. This system provides the diversity of sign languages ​​and their background information, enabling a personalized sign language learning experience that responds to the user's emotions.

[1515] Example 2

[1516] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1517] Conventional systems for supporting sign language learning rely on user input data and basic search functions, and are unable to provide a personalized learning experience that takes into account the user's emotional state. This makes it difficult to elicit the user's interest and affinity and promote effective learning. Furthermore, optimization of search and provided information that takes into account the user's region and age is also insufficient. To solve this problem, a system is needed that analyzes the user's emotional state in real time and provides optimal sign language information based on that analysis.

[1518] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for receiving input data and emotional state of a user, means for constructing a sign language database and storing additional information related to each sign language, means for searching for corresponding sign language information from the sign language database based on the user's input data and emotional information, and means for displaying the search results and the emotional information in a visually and emotionally adjusted form to the user. This makes it possible to provide optimal sign language information according to the user's emotional state, thereby realizing a personalized learning experience. Furthermore, by taking into account regional and era information, it is possible to provide more accurate information.

[1519] "User-entered data" refers to information entered by a user using a terminal, such as keywords and phrases related to sign language searches.

[1520] "Emotional state" refers to a psychological state analyzed from the user's facial expression, tone of voice, etc., and is classified into categories such as relaxation, impatience, and joy.

[1521] A "sign language database" refers to a database that constructs and stores data including sign language gestures, meanings, background information, etc.

[1522] "Additional information" is detailed information related to a sign language, including additional information such as the background, region, and era in which the sign language originated.

[1523] The "search means" refers to a function that searches for corresponding sign language information from a sign language database based on the user's input data and emotion information.

[1524] "Visual display" refers to a method of visually presenting sign language information to a user on a screen, including sign language gestures and additional information.

[1525] "Emotional adjustment" refers to a method of adjusting the priority of information based on the user's emotional state and providing the information to the user in an optimal manner.

[1526] "Region information" is information about the user's current location, and is used for filtering when searching for sign language information.

[1527] "Age information" is information about the user's age and birth decade, and is used for filtering when searching for sign language.

[1528] This invention is a system for providing a personalized experience for users learning sign language. The system has the ability to analyze the user's emotional state in real time and dynamically provide sign language information based on that analysis.

[1529] Overall system configuration

[1530] 1. Receiving user input data

[1531] User: The user enters the sign language keyword they want to search for in the search bar of their device. For example, if they want to learn the sign language for "thank you," they enter "thank you" in the search bar and click the search button.

[1532] 2. Acquiring emotional information

[1533] Device: The device uses a built-in camera and microphone to capture the user's facial expressions and tone of voice, which are then analyzed by an emotion engine. This data is then classified as the user's emotional state (e.g., relaxed, anxious, happy, etc.).

[1534] 3. Generating a search request

[1535] Device: The device generates a search request that includes the keywords and emotion information entered by the user, as well as location and age information. The generated request is sent to the server in the next step.

[1536] 4. Sending a request to the server

[1537] Device: The device generates a search request and sends it to the server, using an internet connection to transfer data using HTTP or HTTPS protocols.

[1538] 5. Database Search

[1539] Server: The server analyzes the received request and searches for the corresponding sign language information from a sign language database, which contains information about the gestures, meanings, and context of the sign language. The search results are prioritized based on the emotional information.

[1540] 6. Formatting Sign Language Information

[1541] Server: The server formats the search results and generates a response in a user-friendly format. If the user is relaxed, the response will contain more information.

[1542] 7. Receiving and Displaying Search Results

[1543] Device: The device analyzes the search results received from the server and displays them in a visually and emotionally tailored way to the user's emotions. For users who show positive emotions, it displays messages that highlight success, and for users who show negative emotions, it adds encouraging messages.

[1544] Specific use cases

[1545] As a specific example, consider the case where a user wants to search for the sign language for "thank you." The user types "thank you" into the search bar and presses the search button. At this time, the device's emotion engine analyzes the user's facial expressions and voice and recognizes that the user is relaxed.

[1546] The device generates a request including the input data "Thank you," region information, age information, and emotional information "Relax," and sends it to the server. The server searches for the corresponding sign language information in a sign language database, prioritizes providing detailed information, and also generates a response including positive feedback.

[1547] The device displays the received search results in a user interface. For example, not only are the sign gestures, background information, and meanings visually displayed, but detailed information such as "This sign means thank you and is used in these situations" is also provided. Furthermore, because the user is relaxed, a positive feedback message is also displayed.

[1548] Example of input prompt for generative AI model

[1549] "Suppose a user wants to know the sign for 'thank you'. You have emotional information that the user is relaxed. In this case, provide detailed information about the sign, including the gesture, background information, and geographic and historical information."

[1550] In this way, a sign language learning support system that reflects emotional information can provide users with a personalized learning experience.

[1551] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1552] Step 1: Receiving User Input

[1553] User:

[1554] The user enters the sign language keyword they want to search for in the search bar of their device and presses the search button. For example, if they want to search for the sign language for "thank you," they enter "thank you" in the search bar.

[1555] Input: Keyboard input ("Thank you")

[1556] Output: Keyword data ("Thank you")

[1557] Specific behavior:

[1558] The user enters keywords into the device's search bar and clicks the "Search" button.

[1559] Step 2: Acquiring emotional information

[1560] Device:

[1561] The device uses a built-in camera and microphone to capture the user's facial expressions and tone of voice, which are then analyzed by an emotion engine.

[1562] Input: Camera video data, microphone audio data

[1563] Output: Emotion data (e.g. "Relaxed")

[1564] Specific behavior:

[1565] The camera captures the user's face and the microphone records the user's voice.

[1566] An emotion engine analyzes this data to determine the user's emotional state.

[1567] Step 3: Generate a search request

[1568] Device:

[1569] The terminal generates a search request by combining the keywords, emotional information, and regional and age information entered by the user.

[1570] Input: Keyword data ("Thank you"), emotion data ("Relax"), region information, age information

[1571] Output: Search request data

[1572] Specific behavior:

[1573] Based on the input data, the data is formatted into a search request format.

[1574] Create request data including region and age information.

[1575] Step 4: Sending a request to the server

[1576] Device:

[1577] The terminal transmits the generated search request to the server.

[1578] Input: Search request data

[1579] Output: HTTP request to the server

[1580] Specific behavior:

[1581] Using the HTTP protocol, a request is sent to the specified URL on the server using the POST method.

[1582] Step 5: Database Search

[1583] server:

[1584] The server analyzes the received request and searches for the corresponding sign language information from the sign language database.

[1585] Input: Search request data

[1586] Output: Sign language information data

[1587] Specific behavior:

[1588] A database query is generated and searched against a sign language database.

[1589] Take search results and adjust priorities based on sentiment information.

[1590] Step 6: Formatting the sign language information

[1591] server:

[1592] The server formats the search results and prepares the sign language information in a user-friendly format.

[1593] Input: Sign language information data

[1594] Output: Formatted display data

[1595] Specific behavior:

[1596] Format sign language gestures, meanings and related information.

[1597] Add positive feedback messages based on emotions.

[1598] Step 7: Receiving and displaying search results

[1599] Device:

[1600] The terminal analyzes the search results received from the server and generates content to be displayed to the user.

[1601] Input: Formatted display data

[1602] Output: User interface display

[1603] Specific behavior:

[1604] The data received from the server is analyzed and sign language gestures and background information are displayed on the user interface.

[1605] It also displays emotion-based feedback messages.

[1606] (Application example 2)

[1607] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1608] There is a need to improve and streamline communication within factories, but there are challenges such as difficulty in communicating smoothly with hearing-impaired employees and responding to employees' emotions in stressful environments. In particular, when communicating using sign language, it is difficult to provide information that corresponds to the emotional state of employees, and an effective system for this purpose is needed.

[1609] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1610] In this invention, the server includes means for receiving user input data, means for building a sign language database and storing additional information related to each sign language, emotion recognition means using an emotion engine that analyzes the user's emotions in real time, means for searching the sign language database for corresponding sign language information based on the user's input data and emotion information, and means for visually displaying the search results in a form that corresponds to the user's emotional state. This enables smooth communication using sign language in factory communications and personalized support that corresponds to the emotions of employees.

[1611] The "means for receiving user input data" is a hardware or software mechanism for obtaining information provided by a user.

[1612] "Means for building a sign language database and storing additional information related to each sign language" refers to a method for comprehensively organizing and storing information about sign languages, as well as their background and meanings.

[1613] The "emotion recognition means" is a mechanism for analyzing the user's emotional state in real time and obtaining the results.

[1614] The "means for searching for corresponding sign language information from a sign language database" is a method for searching for appropriate sign language information from a sign language database based on the user's input data and emotion information.

[1615] "Visual display means" refers to the method of presenting search results to the user through a visual interface such as a display or screen.

[1616] "Means for collecting regional and generational information" refers to methods for acquiring and accumulating data relating to the geographical location and generation of users.

[1617] The "means for dynamically personalizing and providing additional information" is a mechanism for individually optimizing and providing additional information about sign language based on the user's emotional state and other factors.

[1618] The present invention provides a sign language compatible robot assistant system for realizing efficient communication within a factory. Specific embodiments are described below.

[1619] Overall system configuration

[1620] The system mainly consists of the following main components:

[1621] 1. Hardware and software for receiving user input data

[1622] 2. A server that builds a sign language database and stores additional information related to each sign language.

[1623] 3. A device that uses an emotion engine to analyze user emotions in real time

[1624] 4. A device that visually displays search results in a format that corresponds to the user's emotional state

[1625] Receiving user input data

[1626] User input data is collected by a device equipped with a camera and microphone, with the Python OpenCV library installed to capture video frames and collect audio data.

[1627] Sign Language Database

[1628] The server manages a sign language database and stores detailed information about sign languages. This includes information about the gestures and actions of the sign language, as well as additional information such as the background, region, and era in which the sign language originated. The server uses the Python sign_language_database module to efficiently search and provide this information.

[1629] Emotion Engine

[1630] The emotion engine is a software module for analyzing the user's emotions in real time. This module uses a library called EmotionRecognition to analyze the user's facial expressions and tone of voice. The level of detail of the information provided changes depending on whether the user is feeling stressed or not.

[1631] Processing user input data and emotional information

[1632] The device sends the user's input data and the emotional information recognized by the emotion engine together to the server. The server analyzes this request and searches the database for corresponding sign language information. The search results are returned by the server with either a detailed explanation or a concise explanation selected based on the user's emotional state.

[1633] Displaying search results

[1634] The device analyzes the search results received from the server and adjusts the display content based on the emotional information obtained from the emotion engine. For example, if the device detects that the user is feeling stressed, it will prioritize displaying a concise explanation.

[1635] Specific examples

[1636] If a user signs "OK," the emotion engine recognizes that the user is feeling stressed. The device sends this information to the server, which searches for the sign information corresponding to "OK" and returns a brief explanation. It also adds positive feedback as a support message. The device visually displays this to the user, helping them to reduce their stress.

[1637] Prompt Sentence Examples

[1638] Examples of input prompts for a generative AI model might include:

[1639] "Recognize sign language used in factories and provide dynamic support according to employees' emotions."

[1640] This will enable smooth communication within the factory using sign language and personalized support that responds to employees' emotions.

[1641] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1642] Step 1:

[1643] The user signs.

[1644] Input: User's sign language actions, facial expressions, and voice

[1645] Output: Video frames and audio data captured by the camera and microphone

[1646] Specific operation: When a user signs, the device's camera records the sign and the microphone collects the audio.

[1647] Step 2:

[1648] The device processes the input data to analyze the user's emotions in real time.

[1649] Input: Video frames and audio data

[1650] Output: User's emotional state (e.g., relaxed, stressed, etc.)

[1651] How it works: The device uses the EmotionRecognition library to analyze facial expressions from video frames and tone of voice from audio data, and then determines the user's emotional state based on the results of this analysis.

[1652] Step 3:

[1653] The device recognizes sign language.

[1654] Input: Video frame

[1655] Output: Sign language commands (e.g. OK, Thank you, etc.)

[1656] Specific actions: The device uses a sign language recognition algorithm to analyze the sign language actions in the video frames and extract specific sign language commands.

[1657] Step 4:

[1658] The terminal transmits the user's input data and emotion information to the server.

[1659] Input: sign language commands, emotional states, video frames, audio data

[1660] Output: Request data to the server

[1661] Specific operation: The terminal assembles the user's input data (sign language commands, video frames, and voice data) and emotional state into packets and sends them to the server.

[1662] Step 5:

[1663] The server searches for the corresponding sign language information from the sign language database.

[1664] Input: Request data

[1665] Output: Search results (sign meaning, trigger, additional information, etc.)

[1666] Specific operation: The server receives the request data and queries the sign language database, searching for sign language information corresponding to the sign language command and adjusting the level of detail of the information according to the emotional state.

[1667] Step 6:

[1668] The server sends the search results to the terminal.

[1669] Input: Search results

[1670] Output: Response data to the terminal

[1671] Specific operation: The server organizes the search results, converts them into an appropriate format, and sends them to the terminal.

[1672] Step 7:

[1673] The terminal visually displays the search results to the user.

[1674] Input: Response data

[1675] Output: What is displayed on the user interface

[1676] Specific operation: The device receives the response data and adjusts the display content according to the user's emotional state, for example, displaying a brief explanation to a stressed user and providing detailed information to a relaxed user.

[1677] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1678] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1679] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1680] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1681] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1682] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1683] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1684] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1685] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1686] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1687] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1688] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1689] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1690] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1691] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1692] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1693] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1694] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1695] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1696] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1697] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1698] The following is further disclosed regarding the above embodiment.

[1699] (Claim 1)

[1700] means for receiving user input data;

[1701] a means for constructing a sign language database and storing additional information associated with each sign language;

[1702] means for searching for corresponding sign language information from a sign language database based on user input data;

[1703] means for visually displaying search results to a user;

[1704] A system including:

[1705] (Claim 2)

[1706] The system according to claim 1, which collects regional and age information of users and uses it for sign language information searches.

[1707] (Claim 3)

[1708] 10. The system of claim 1, wherein the system provides the user with additional information related to the meaning or cue of the sign language.

[1709] "Example 1"

[1710] (Claim 1)

[1711] A means for receiving a user's search keywords, region information, and age information;

[1712] A means of managing a database that systematically stores various information related to sign language;

[1713] A means for searching a database for information such as gestures, background, region, era, and meaning of the corresponding sign language based on the user's input data;

[1714] A means to return search results in an appropriate format, such as JSON, and

[1715] means for analyzing the search results received from the server and displaying them on a user interface;

[1716] A system including:

[1717] (Claim 2)

[1718] 2. The system according to claim 1, wherein the system utilizes the user's regional information and age information to improve the accuracy of sign language information searches.

[1719] (Claim 3)

[1720] 10. The system of claim 1, further providing additional information to the user, such as sign language gestures, background information, and meaning.

[1721] "Application Example 1"

[1722] (Claim 1)

[1723] means for receiving user input data;

[1724] a means for constructing a database of sign languages ​​and storing additional information associated with each sign language;

[1725] means for searching for corresponding sign language information from a sign language database based on user input data;

[1726] means for visually displaying search results to a user;

[1727] means to enable the efficient learning and use of sign language in specific tasks;

[1728] A system including:

[1729] (Claim 2)

[1730] The system according to claim 1, wherein when an employee performing a specific task uses sign language, regional information and age information are collected and used to search for sign language information.

[1731] (Claim 3)

[1732] 10. The system of claim 1, wherein additional information related to the meaning or cue of sign language is provided to an employee performing a specific task.

[1733] "Example 2: Combining Emotion Engines"

[1734] (Claim 1)

[1735] means for receiving user input data and emotional state;

[1736] a means for constructing a database of sign languages ​​and storing additional information associated with each sign language;

[1737] means for searching for corresponding sign language information from a sign language database based on the user's input data and emotion information;

[1738] means for displaying the search results and the emotional information to the user in a visually and emotionally tailored manner;

[1739] A system including:

[1740] (Claim 2)

[1741] The system according to claim 1, which collects regional and age information of users and uses it for sign language information searches.

[1742] (Claim 3)

[1743] 10. The system of claim 1, wherein the system provides the user with additional information related to the meaning and background of the sign language.

[1744] "Application example 2 when combining emotion engines"

[1745] (Claim 1)

[1746] means for receiving user input data;

[1747] a means for constructing a sign language database and storing additional information associated with each sign language;

[1748] an emotion recognition means using an emotion engine that analyzes the user's emotions in real time;

[1749] means for searching for corresponding sign language information from a sign language database based on the user's input data and emotion information;

[1750] means for visually displaying search results in a manner that corresponds to the emotional state of the user;

[1751] A system including:

[1752] (Claim 2)

[1753] 2. The system according to claim 1, which collects regional and age information of users and uses it to search for and provide sign language information.

[1754] (Claim 3)

[1755] 10. The system of claim 1, wherein the system dynamically personalizes and provides additional information related to the meaning or cue of the sign language based on the user's emotional state. [Explanation of symbols]

[1756] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving user input data; a means for constructing a sign language database and storing additional information associated with each sign language; means for searching for corresponding sign language information from a sign language database based on user input data; means for visually displaying search results to a user; A system including:

2. 2. The system according to claim 1, wherein regional information and age information of the user are collected and used for searching for sign language information.

3. 10. The system of claim 1, further comprising: providing a user with additional information related to the meaning or cue of the sign language.

Images