system

Abstract

We provide the system. [Solution] A means of converting handwritten original language characters into digital data using optical character recognition technology, A method using generative technology that extracts and learns handwriting styles from digital data, A means of translating text from one language to several other languages ​​and applying a learned handwriting style to the translated text, A means for outputting the generated multilingual handwritten style font as an information providing device, A means of generating a two-dimensional code as a means of accessing the generated multilingual handwritten style font, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, with the increase in foreign tourists, many restaurants have been forced to provide multilingual menus. However, it is difficult to efficiently multilingualize while maintaining the design and individuality unique to handwritten menus. Furthermore, conventional printed menus cannot flexibly respond to frequent updates and language additions, which may impair the convenience of tourists. Thus, there is a need to provide a multilingual digital menu while maintaining a handwritten style, and further to provide a means for connecting it to an efficient ordering system.

Means for Solving the Problems

[0005] This invention provides a system that automatically translates handwritten menus from one language to another and generates multilingual handwritten style fonts by using a generation technology that converts handwritten menus into digital data and learns the handwriting style. This system outputs the generated multilingual handwritten fonts as an information provision device and includes a means for customers to easily access them via a QR code (registered trademark). Furthermore, by linking the information provision device using the generated fonts with a digital ordering system, it becomes possible to efficiently transmit order information electronically. As a result, restaurants can achieve efficient multilingual menu provision and order processing while maintaining the design of handwritten menus.

[0006] Optical character recognition (OCR) is a technology that identifies handwritten or printed text from an image and converts it into digital text data.

[0007] "Generative technology" refers to the technology of training computer models to automatically create digital content with specific design styles and characteristics.

[0008] An "information providing device" is a device for displaying or transmitting specific information to a user, and in this invention, it is a device for displaying a generated multilingual handwritten font.

[0009] A "digital ordering system" is a system that electronically transmits the menu items selected by customers to the store system, thereby streamlining order processing.

[0010] A "multilingual handwritten style font" refers to a font that displays text in a specific language using a character style that retains the characteristics of learned handwriting.

[0011] A "QR code" is a type of two-dimensional barcode that contains visually embedded data to provide quick access to information. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0013] An example of an embodiment of the system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0018] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] This invention is a system that enables multilingual support for handwritten menus in restaurants, streamlining the generation of multilingual menus and order processing while preserving the individuality of handwritten menus. This system operates through the cooperation of a server, terminals, and users.

[0034] First, the user scans or photographs a handwritten Japanese menu and uploads the image data to the server. The server applies optical character recognition (OCR) technology to the received image data, converting the handwritten text into digital text data. During this process, the shape of each character, the characteristics of the handwriting, and other details are saved as digital data.

[0035] Next, the server uses generation technology to learn the features of the handwriting style and performs multilingual translation while preserving the style of the digitized Japanese text. Translation is performed for major tourist languages ​​(e.g., English, French, Chinese, etc.). This generates multiple foreign language versions of the handwritten menu.

[0036] The server then outputs the generated handwritten fonts for each language to the information provider, which generates QR codes that serve as access links to the menus in each language. Users distribute these QR codes within the store, allowing customers to easily access the menus on their smartphones or tablets.

[0037] The terminal can receive customer access and display a handwritten menu in the selected language. Customers can select items from the menu and place their orders through the digital ordering system. This process allows orders to be instantly transmitted from the server to the kitchen and service terminals.

[0038] In this way, it becomes possible to maintain the warmth of handwritten menus while providing multilingual support for foreign tourists visiting Japan and processing orders efficiently. This system allows restaurants to offer multilingual menus with their own unique handwritten style while ensuring smooth store operations.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] Users scan or photograph handwritten menus and upload the image files to the server.

[0042] Step 2:

[0043] The server applies optical character recognition (OCR) technology to the received image data, converting handwritten characters into digital text data. During this process, it extracts handwritten style features such as character shape and line thickness.

[0044] Step 3:

[0045] The server uses generative techniques to learn the features of the extracted handwriting styles and generates those styles as models for digital text data.

[0046] Step 4:

[0047] The server uses a translation system to translate digital text from Japanese into other specified languages. During this process, the system applies the handwritten style to the text in each language to ensure it is preserved.

[0048] Step 5:

[0049] The server outputs the generated handwritten style fonts for each language to the information provider and generates a QR code containing an access link to the menu for each language.

[0050] Step 6:

[0051] Users print or digitally display QR codes received from the server and distribute them within the store for easy access by customers.

[0052] Step 7:

[0053] The terminal displays a handwritten menu in the selected language on the device when the customer scans a QR code.

[0054] Step 8:

[0055] The user (as a customer) refers to a multilingual handwritten menu displayed on the terminal, selects the desired products via the digital ordering system, and completes the order.

[0056] Step 9:

[0057] The server receives the order details sent by the user and immediately transfers the information to the kitchen and service terminals within the system, ensuring efficient order processing.

[0058] (Example 1)

[0059] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0060] In restaurants, when providing multilingual options for handwritten menus, it's necessary to efficiently offer menus in multiple languages ​​while preserving the unique character of handwritten menus and speeding up customer order processing. This challenge requires technology that can simultaneously reproduce the handwritten style and provide multilingual translation.

[0061] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0062] In this invention, the server includes means for converting handwritten characters in the original language into digital data using optical character recognition technology; means for using generation technology to extract and learn the handwriting style of characters from the digital data; and means for translating text from a specific language to multiple other languages ​​and applying the learned handwriting style to the translated text. This enables multilingual support while preserving the warmth of handwriting, and allows for fast and efficient order processing.

[0063] Optical character recognition technology is a technology that extracts handwritten or printed character information from image data and converts it into digital text data.

[0064] "Digital data" refers to information that has been converted from analog information, such as handwritten text or images, into a digital format.

[0065] "Generative technology" is a technique that generates new data using algorithms that extract and learn features from specific data.

[0066] "Handwritten style" refers to the shape and characteristics of handwritten characters, and is a typeface that is expressed as a unique style.

[0067] Translation is the process of converting text written in one language into another language.

[0068] An "information display device" is a device that displays digital data and provides information to users visually.

[0069] An "identification code" is a code generated to quickly access specific information; a QR code is an example of this.

[0070] A "digital ordering system" is a system that receives and processes customer orders electronically.

[0071] This invention is a system for making handwritten menus in restaurants multilingual. The system functions through the cooperation of a server, terminals, and users.

[0072] The user first creates a handwritten Japanese menu and then uses a smartphone or scanner to capture it as a digital image. This image is then uploaded to the server using a dedicated application or web interface. This process digitizes the handwritten menu, and the system begins processing it.

[0073] The server utilizes optical character recognition technologies such as Google® Cloud Vision API and Microsoft® Azure® OCR service to convert handwritten characters from received images into digital text. During this conversion process, the shape of the characters and the characteristics of the handwriting are also extracted, and the generating AI model learns from this information. The learned handwriting style is then applied to text in a specific language.

[0074] Next, the server uses a generative AI model to translate the digital text into major tourist languages. The Google Translate API and DeepL Translator API are used for this translation process. The translated text is then given a learned handwriting style to create a newly generated multilingual handwriting menu. This handwriting style preserves the individuality of the handwriting while reflecting its characteristics in the multilingual menu.

[0075] The server then outputs the menus, generated in multiple languages, to the information provider and creates a QR code as an identification code. This QR code is provided to users for distribution within the store, allowing customers to easily access the menu via information devices such as smartphones and tablets.

[0076] When the customer scans the QR code, the terminal can display a handwritten menu in the selected language. Through this process, the customer can select products and place orders via the digital ordering system. This order information is immediately transmitted to the kitchen equipment and service terminals via the server, enabling efficient order processing.

[0077] As a specific example, when offering "Matcha Latte" in a café, if the user handwrites "Matcha Latte" in Japanese, the system translates it into "Matcha Latte" in English, "Thé Matcha Latte" in French, "抹茶拿铁" in Chinese, etc., and displays it while maintaining the handwritten style.

[0078] Examples of prompt sentences:

[0079] "Perform OCR processing on the image of the handwritten menu, digitize the Japanese text for multilingual translation. Maintain the handwritten style after translation and use generative AI to create menus in various languages."

[0080] The flow of the specific process in Example 1 will be described using FIG. 11.

[0081] Step 1:

[0082] The user obtains a digital image of the handwritten menu using a smartphone or scanner and uploads the image to the server via a dedicated application or web interface. The input is the image of the handwritten menu, and the output is the digital image data stored on the server. As a specific operation, the user takes a clear photo of the image using a high-resolution camera, performs preprocessing such as rotation and trimming, and then sends it to the server.

[0083] Step 2:

[0084] The server uses Optical Character Recognition (OCR) technology to read character information from the received image data. The input is the uploaded digital image, and the output is the identified text data. Specifically, OCR extracts the outlines of characters and matches them with existing character patterns to convert them into digital text.

[0085] Step 3:

[0086] The server uses a generative AI model to extract and learn handwriting style features from the text obtained by OCR. The input is text data obtained through OCR processing, and the output is digital data that retains the handwriting style. In this step, the server analyzes the shape data of the characters and models the tendencies of the handwriting.

[0087] Step 4:

[0088] The server translates text from a specific language into multiple languages ​​targeting major tourists and applies learned handwriting styles. The input is text data that retains handwriting styles, and the output is translated text written in the handwriting style fonts of each language. Specifically, it uses a translation API to perform language conversion and then applies style data to the result.

[0089] Step 5:

[0090] The server outputs the generated multilingual handwritten-style fonts as an information provider and generates identification codes (QR codes) for accessing each language. The input is the translated text of the handwritten-style font, and the output is font data and QR codes available on the information provider.

[0091] Step 6:

[0092] The terminal displays a handwritten-style menu in the selected language when the customer scans a QR code. Input is the menu information obtained via the QR code, and output is the handwritten-style menu displayed on the terminal. Specifically, the terminal retrieves data from a server via a URL and displays it at the appropriate display resolution.

[0093] Step 7:

[0094] Customers select items from the displayed menu and place their orders through the digital ordering system. Input is the product information selected by the customer on their terminal, and output is the order data sent to the server. The server transmits this order information in real time to kitchen equipment and service terminals, supporting rapid processing.

[0095] (Application Example 1)

[0096] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0097] In today's world, with the increasing number of global tourists, the food and beverage industry faces the need to provide multilingual menus. However, traditional printed menus struggle to maintain the individuality of handwritten text while also being able to handle orders efficiently. Against this backdrop, there is a need for a system that can handle multiple languages ​​while preserving the warmth of handwritten text, and that can also process orders smoothly.

[0098] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0099] In this invention, the server includes means for converting handwritten characters in the original language into digital data using optical character recognition technology; means for using generation technology to extract and learn the handwriting style of characters from the digital data; and means for translating text from a specific language to multiple other languages ​​and applying the learned handwriting style to the translated text. This enables the efficient generation of multilingual handwritten-style menus that maintain the handwriting style, allowing users to easily access and order from them.

[0100] Optical character recognition (OCR) technology is a technology that reads handwritten or printed characters from an image and converts them into digital text.

[0101] "Digital data" refers to information that has been converted into a format that can be read by a computer, and includes data such as text and images.

[0102] "Handwritten style" refers to the shape and characteristics of handwriting as written by a human being.

[0103] "Generative technology" is a technique that uses machine learning to learn the characteristics of specific data and generate new data.

[0104] A "two-dimensional code" is a code that encodes information into a grid pattern of vertical and horizontal elements, allowing it to be read by a scanner.

[0105] A "terminal device" is an electronic device used by a user to display, input, and manipulate information.

[0106] "Data communication means" refers to methods and technologies for electronically transmitting or receiving information to or from other devices.

[0107] The present invention provides a system for digitizing handwritten menus in stores in a multilingual format and efficiently delivering them to users. This system operates through the coordinated efforts of a server, terminals, and users.

[0108] The user first scans or photographs the handwritten menu as an image and uploads the image data to the server. The server then applies optical character recognition (OCR) technology to this image data to convert the handwritten characters into digital text. This process uses the Python Imaging Library (PIL) and character recognition is performed by pytesseract.

[0109] Next, the server uses generation technology to learn handwriting styles and perform multilingual translation while maintaining the digitized text. This process utilizes the translate library to translate into major tourist languages ​​(e.g., English, French, Chinese, etc.). Then, the handwriting style is applied to the translated text to generate handwritten-style fonts for the foreign languages.

[0110] Furthermore, a QR code containing an access link to view the handwritten menus in each generated language is generated and provided on the terminal. At this stage, a QR code library is used to generate the QR code, making it easy for customers to access it on devices such as smartphones and tablets. Upon receiving the customer's access, the terminal displays the handwritten menu in the desired language and enables immediate ordering through the digital ordering system.

[0111] As a concrete example, one cafe creates handwritten menus in Japanese and uses this system to generate English, French, and Chinese versions, allowing tourists from various countries to view and order in their own languages. By using this system, they can achieve multilingualism without losing the charm of handwritten menus and streamline store operations.

[0112] Examples of prompts for a generative AI model:

[0113] "Please use OCR technology to convert the following handwritten image into text and translate it into Chinese, English, and French. Maintain the handwritten style while translating as naturally as possible in each language. Finally, generate a QR code corresponding to each language and output it with an encrypted access link."

[0114] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0115] Step 1:

[0116] The user obtains an image file of a handwritten menu by taking a photograph or scanning it, and uploads that data to the server. The input is an image of the handwritten menu, which becomes a digital image file. The output is a status message indicating that the data upload is complete.

[0117] Step 2:

[0118] The server applies optical character recognition (OCR) technology to uploaded image data. The input is an image of a handwritten menu received from the user. The server uses Pytesseract to convert the handwritten characters in the image into text and outputs the result as character data. In this process, the outlines and shapes of the characters are converted into digital data, and the extracted string is generated.

[0119] Step 3:

[0120] The server uses generative technology to learn the features of handwritten styles. The input consists of character data and its shape features obtained by OCR. Based on this, a generative AI model is used to recognize the handwriting and style of each character and output characteristic data. This output includes style information that will be applied in the subsequent translation process.

[0121] Step 4:

[0122] The server translates character data into other languages ​​and applies the initial handwriting style to the new language text. The input is the characteristic data and character data obtained in step 3. The server uses the Translate library to translate into major foreign languages ​​and applies the learned handwriting style to the translation results. The output is handwriting-style font style data for each language.

[0123] Step 5:

[0124] The server generates a two-dimensional code for output to the information provider device based on the handwritten style fonts generated for each language. The input is handwritten style font data for each language. The server uses the QRCode library to create and output a two-dimensional code that contains a link for the user to access the menu in each language.

[0125] Step 6:

[0126] The terminal uses a generated QR code to display a handwritten menu in the language selected by the customer upon access. Input is scanned data from the customer's smartphone or tablet. The terminal downloads the menu in the corresponding language and displays it on the customer's device. Output is a displayed handwritten-style multilingual menu.

[0127] Step 7:

[0128] Customers select an order from the displayed menu and send it to the server via the digital ordering system. The input is the customer's order selection data. The server receives this information, immediately processes it to transmit it to the kitchen and service staff, and generates an output indicating that the order has been processed.

[0129] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0130] This invention is a system that provides interactive menus based on user emotions by incorporating an emotion engine. It provides multilingual menus that retain a handwritten style, recognizes customer emotions, and makes menu suggestions accordingly.

[0131] First, the user provides facial and voice data to the emotion engine through the camera and microphone of a terminal installed in the store. The server analyzes this data and recognizes the emotions the user is feeling. In this process, it determines whether the customer's reaction to the displayed menu items is positive or negative.

[0132] Next, the server suggests a menu tailored to the user based on their perceived emotions. For example, if the user is happy, recommended dishes can be highlighted. Conversely, if the user appears dissatisfied, other options may be presented to draw their attention, providing information that matches the user's state.

[0133] Furthermore, the terminal dynamically updates the menu content in conjunction with an emotion engine. This allows users to always see a menu tailored to their preferences, ensuring a comfortable ordering experience. For example, if a family is visiting and everyone seems satisfied, the terminal can present a set menu and ask if they would like to order it.

[0134] Furthermore, once a user confirms an order, the information is immediately transmitted via the server to the kitchen and related service locations, enabling efficient order processing. The system also stores a history of emotionally-based suggestions, allowing for more personalized service on subsequent visits.

[0135] In this way, the present invention provides not only handwritten-style multilingual menus but also interactive food and beverage services utilizing emotion recognition, thereby constructing an efficient and flexible system that can meet diverse user needs.

[0136] The following describes the processing flow.

[0137] Step 1:

[0138] Users provide facial expressions and voice recordings via cameras and microphones installed on the store's information terminals. This allows the necessary data for the emotion engine to be acquired.

[0139] Step 2:

[0140] The server analyzes the received facial expression and voice data to determine the user's current emotional state. This analysis uses an emotion engine that employs facial expression recognition algorithms and voice analysis algorithms.

[0141] Step 3:

[0142] Based on the recognized emotions, the server suggests dishes likely to suit the user's preferences from a handwritten-style multilingual menu. For example, if a relaxed emotion is detected, information on new menu items and special dishes will be displayed preferentially.

[0143] Step 4:

[0144] The terminal visually highlights suggested menu items for the user. It dynamically updates menu content to support flexible menu selections that respond to the user's emotions.

[0145] Step 5:

[0146] The user selects a dish based on the displayed suggestions and enters their order via an information terminal.

[0147] Step 6:

[0148] The server receives confirmed order information and immediately transmits it to the terminals of the relevant kitchen and service staff. This enables smooth order processing and food delivery.

[0149] Step 7:

[0150] After processing an order, the server saves the user's sentiment data and order history, enabling more personalized menu suggestions for future visits.

[0151] Through this series of processes, improved user experience and efficient store operations are achieved by utilizing the emotion engine.

[0152] (Example 2)

[0153] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0154] Amidst the increasing diversification and globalization of information, there is a growing demand for flexible information delivery tailored to individual needs and emotions. However, conventional systems have struggled to dynamically adjust information based on users' emotional states. Furthermore, there has been a lack of efficient means to deliver personalized information in multilingual environments. This has resulted in a limited user experience and decreased satisfaction.

[0155] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0156] In this invention, the server includes means for digitizing handwritten characters, means for extracting handwriting characteristics from the digital data, and means for acquiring the user's emotional state and dynamically adjusting the information. This enables interactive and personalized information provision that responds to the user's emotions.

[0157] "Handwritten text" refers to text created by a person using a writing instrument on paper or a device.

[0158] "Optical character recognition technology" is a technology that analyzes characters and symbols acquired as images and converts them into digital character data.

[0159] "Digital data" refers to binary data that is a digital representation of analog information.

[0160] "Handwriting characteristics" refer to the unique shape and style of handwritten characters.

[0161] "Generative technology" refers to techniques that create new data using certain types of learning models.

[0162] "Emotional state" refers to the psychological state inferred from the user's facial expression and voice.

[0163] A "generative AI model" is an artificial intelligence model that learns patterns from data based on machine learning algorithms and generates new information.

[0164] "Interactive" refers to a characteristic that dynamically changes in response to user input and possesses two-way operability.

[0165] "Personalization" refers to customizing information and services according to an individual's characteristics and needs.

[0166] An "information device" is an electronic device used for receiving, processing, or transmitting information.

[0167] This invention aims to realize a system that generates multilingual handwritten-style fonts and provides personalized information based on the user's emotional state.

[0168] Users access the system through a GUI displayed on an information terminal. Upon arrival, the terminal's built-in camera and microphone capture the user's facial expressions and voice. This information is processed through an emotion recognition algorithm to analyze the user's emotional state in real time. This analysis utilizes image processing tools such as OpenCV and speech analysis libraries.

[0169] The server runs a generative AI model based on the analyzed sentiment data to generate a personalized menu of suggestions. This AI model uses algorithms learned from a large dataset, enabling it to generate information tailored to the user's individual preferences and circumstances.

[0170] One concrete example is generating a prompt that highlights special desserts or promotions if the user is smiling. An example of a generated prompt might be, "The user is happy, please show us our recommended special menu items."

[0171] Based on the generated prompt, the terminal displays the content using a multilingual handwritten-style font. This allows for information to be presented in a visually appealing way to the user. This handwritten-style font is applied to text translated from the original language into various languages, ensuring style preservation.

[0172] Furthermore, once a user confirms their order, the information is promptly transmitted via the server to the kitchen and related services. This enables smooth, real-time service delivery, improving the user experience.

[0173] Through this system, the present invention integrates a handwritten-style multilingual font with an interactive service utilizing emotion recognition, achieving advanced personalization that meets the diverse needs of users.

[0174] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0175] Step 1:

[0176] The user accesses an information terminal and provides facial expressions and voice to the system via camera and microphone. Input data includes image data and audio data. The terminal uses facial recognition and voice analysis algorithms to capture and preprocess this data in real time. The output is organized facial image data and voice feature data. Specifically, the system captures the user's facial expressions, such as smiling while looking at a menu, and the tone of their voice.

[0177] Step 2:

[0178] The server inputs the organized data received from the terminal into an emotion recognition algorithm. This algorithm uses OpenCV and speech analysis libraries to analyze the user's emotions from facial muscle movements and voice tone. The output at this stage is an emotion label (e.g., joy, surprise, dissatisfaction) that the user is experiencing. Specifically, the server performs a process to identify the user's smile as "joy."

[0179] Step 3:

[0180] The server generates prompts using a generative AI model based on recognized emotion labels. The input includes emotion labels and associated historical data. The AI ​​model analyzes this data to generate context-appropriate suggestions. The output is a user-specific prompt. For example, a prompt such as "Suggest a dessert for a happy user" might be generated.

[0181] Step 4:

[0182] The terminal receives the generated prompt and displays information on the screen using a multilingual handwritten-style font based on it. Visual information corresponding to the prompt is incorporated into the menu. Specifically, recommended desserts are highlighted and displayed to the user in an attractive handwritten font.

[0183] Step 5:

[0184] Once a user confirms an order, the information is automatically transmitted to the relevant facilities via the server. The input is the menu items selected by the user, and the output is the order data communicated to the kitchen and relevant service teams. This ensures that order processing is expedited. Specifically, the selected menu is immediately sent to the internal terminal, and cooking preparation begins.

[0185] (Application Example 2)

[0186] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0187] Conventional menu information systems have the challenge of not being able to respond immediately and flexibly to users' emotions and diverse language needs. In particular, personalized menu suggestions based on users' emotions are difficult, and there is a need for more efficient and individualized ordering processes.

[0188] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0189] In this invention, the server includes means for converting handwritten original language characters into digital data using optical character recognition technology, means for using generation technology to extract and learn the handwriting style of the characters from the digital data, and means for recognizing the user's emotions and dynamically updating and displaying menu information according to those emotions. As a result, users can always receive multilingual menus that are tailored to their emotions and preferences, enabling an efficient and personalized ordering experience.

[0190] "Handwritten style" refers to fonts and characters that imitate the texture and typeface of handwriting.

[0191] "Optical character recognition technology" is a technology that scans images or handwritten characters and converts them into corresponding character codes.

[0192] "Digital data" refers to information that has been converted into a format that can be processed by computers and electronic devices.

[0193] "Generative technology" refers to techniques that use algorithms to create new data and information.

[0194] An "information provision device" refers to a device or hardware used to display information to a user.

[0195] An "information code" refers to an identifier used as a structured form of information, which is necessary for users to access it.

[0196] An "emotion engine" refers to software that analyzes a user's emotions from their facial expressions, voice, etc., and processes those emotions accordingly.

[0197] "Communication means" refers to the technologies and methods used to transmit information to other devices or systems.

[0198] A "commercial facility" refers to a place or space that provides goods or services to consumers.

[0199] To implement this invention, a program is required that ensures the entire system operates in a coordinated manner. The server converts handwritten original language characters into digital data using optical character recognition technology, and then generates a multi-language digital font using a generation technology that extracts handwritten style features from the digital data. This digital font is output to a terminal acting as an information provider.

[0200] The device analyzes the user's facial expressions and voice using an emotion engine through interaction with the user. This allows the server to recognize the emotions the user is feeling towards the displayed menu. Based on these recognized emotions, the device dynamically updates the menu in real time and re-presents it in a visually appealing way.

[0201] In this process, dynamic menu suggestions based on emotion recognition are crucial, and this role is fulfilled by using a generative AI model. Finally, the user's selected order information is immediately transmitted via communication to the relevant cooking departments, etc.

[0202] For example, if a family is enjoying themselves at a restaurant, the device can highlight special family set menus and add recommendations for desserts and side dishes. By providing a prompt such as, "This family seems to be enjoying themselves. Please suggest a set menu," to the AI ​​model, attractive suggestions can be automatically generated.

[0203] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0204] Step 1:

[0205] The server captures handwritten characters with a camera and converts them into digital data using optical character recognition (OCR) technology. The input is image data of the handwritten characters, and the output is a digital character code. This data conversion makes the character information processable in digital format.

[0206] Step 2:

[0207] The server extracts features from the converted digital data to identify handwritten character styles and uses generative techniques for learning. In this process, digital character data is provided as input, and learned handwritten style parameters are generated as output. The generative AI model is then used to reproduce natural-looking handwriting.

[0208] Step 3:

[0209] The device collects the user's facial expressions and voice through its camera and microphone, and analyzes them using an emotion engine. The input to the emotion recognition process is facial image and audio data, and the output is the analyzed emotion (e.g., joy, anger). This output is used as basic data for menu suggestions.

[0210] Step 4:

[0211] The server generates the most suitable menu for the user based on the emotion recognition results. The input includes the user's emotion data and menu information, and the output is an optimized menu tailored to the emotion. This process involves inputting prompt sentences into a generating AI model to provide effective menu suggestions.

[0212] Step 5:

[0213] The terminal displays an optimization menu received from the server to the user. The user can review the menu displayed on the screen and make a selection freely. The output is visually presented menu information, reflecting the user's reactions as they occur during the selection process.

[0214] Step 6:

[0215] Once a user confirms an order, the terminal sends the order details to the server and forwards them to the relevant department. The input is the user's order selection, and the output is the order data. The server immediately transmits this information to the cooking department and other relevant departments via communication means, supporting efficient order processing.

[0216] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0217] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0218] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0219] [Second Embodiment]

[0220] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0221] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0222] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0223] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0224] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0225] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0226] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0227] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0228] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0229] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0230] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0231] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0232] This invention is a system that enables multilingual support for handwritten menus in restaurants, streamlining the generation of multilingual menus and order processing while preserving the individuality of handwritten menus. This system operates through the cooperation of a server, terminals, and users.

[0233] First, the user scans or photographs a handwritten Japanese menu and uploads the image data to the server. The server applies optical character recognition (OCR) technology to the received image data, converting the handwritten text into digital text data. During this process, the shape of each character, the characteristics of the handwriting, and other details are saved as digital data.

[0234] Next, the server uses generation technology to learn the features of the handwriting style and performs multilingual translation while preserving the style of the digitized Japanese text. Translation is performed for major tourist languages ​​(e.g., English, French, Chinese, etc.). This generates multiple foreign language versions of the handwritten menu.

[0235] The server then outputs the generated handwritten fonts for each language to the information provider, which generates QR codes that serve as access links to the menus in each language. Users distribute these QR codes within the store, allowing customers to easily access the menus on their smartphones or tablets.

[0236] The terminal can receive customer access and display a handwritten menu in the selected language. Customers can select items from the menu and place their orders through the digital ordering system. This process allows orders to be instantly transmitted from the server to the kitchen and service terminals.

[0237] In this way, it becomes possible to maintain the warmth of handwritten menus while providing multilingual support for foreign tourists visiting Japan and processing orders efficiently. This system allows restaurants to offer multilingual menus with their own unique handwritten style while ensuring smooth store operations.

[0238] The following describes the processing flow.

[0239] Step 1:

[0240] Users scan or photograph handwritten menus and upload the image files to the server.

[0241] Step 2:

[0242] The server applies optical character recognition (OCR) technology to the received image data, converting handwritten characters into digital text data. During this process, it extracts handwritten style features such as character shape and line thickness.

[0243] Step 3:

[0244] The server uses generative techniques to learn the features of the extracted handwriting styles and generates those styles as models for digital text data.

[0245] Step 4:

[0246] The server uses a translation system to translate digital text from Japanese into other specified languages. During this process, the system applies the handwritten style to the text in each language to ensure it is preserved.

[0247] Step 5:

[0248] The server outputs the generated handwritten style fonts for each language to the information provider and generates a QR code containing an access link to the menu for each language.

[0249] Step 6:

[0250] Users print or digitally display QR codes received from the server and distribute them within the store for easy access by customers.

[0251] Step 7:

[0252] The terminal displays a handwritten menu in the selected language on the device when the customer scans a QR code.

[0253] Step 8:

[0254] The user (as a customer) refers to a multilingual handwritten menu displayed on the terminal, selects the desired products via the digital ordering system, and completes the order.

[0255] Step 9:

[0256] The server receives the order details sent by the user and immediately transfers the information to the kitchen and service terminals within the system, ensuring efficient order processing.

[0257] (Example 1)

[0258] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0259] In restaurants, when providing multilingual options for handwritten menus, it's necessary to efficiently offer menus in multiple languages ​​while preserving the unique character of handwritten menus and speeding up customer order processing. This challenge requires technology that can simultaneously reproduce the handwritten style and provide multilingual translation.

[0260] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0261] In this invention, the server includes means for converting handwritten characters in the original language into digital data using optical character recognition technology; means for using generation technology to extract and learn the handwriting style of characters from the digital data; and means for translating text from a specific language to multiple other languages ​​and applying the learned handwriting style to the translated text. This enables multilingual support while preserving the warmth of handwriting, and allows for fast and efficient order processing.

[0262] Optical character recognition technology is a technology that extracts handwritten or printed character information from image data and converts it into digital text data.

[0263] "Digital data" refers to information that has been converted from analog information, such as handwritten text or images, into a digital format.

[0264] "Generative technology" is a technique that generates new data using algorithms that extract and learn features from specific data.

[0265] "Handwritten style" refers to the shape and characteristics of handwritten characters, and is a typeface that is expressed as a unique style.

[0266] Translation is the process of converting text written in one language into another language.

[0267] An "information display device" is a device that displays digital data and provides information to users visually.

[0268] An "identification code" is a code generated to quickly access specific information; a QR code is an example of this.

[0269] A "digital ordering system" is a system that receives and processes customer orders electronically.

[0270] This invention is a system for making handwritten menus in restaurants multilingual. The system functions through the cooperation of a server, terminals, and users.

[0271] The user first creates a handwritten Japanese menu and then uses a smartphone or scanner to capture it as a digital image. This image is then uploaded to the server using a dedicated application or web interface. This process digitizes the handwritten menu, and the system begins processing it.

[0272] The server utilizes optical character recognition technologies such as Google Cloud Vision API and Microsoft Azure OCR service to convert handwritten characters from received images into digital text. During this conversion process, the shape of the characters and the characteristics of the handwriting are also extracted, and the generating AI model learns from this information. The learned handwriting style is then applied to text in a specific language.

[0273] Subsequently, the server uses the generative AI model to translate digital text into languages for major tourists. Google Translate API and DeepL Translator API are used in this translation process. The learned handwriting style is applied to the translated text, and a newly generated multilingual handwritten menu is created. This handwriting style does not damage the individuality of the handwriting and also reflects its characteristics in the multilingual menu.

[0274] After that, the server outputs the menu generated in multiple languages to the information providing device and generates a QR code as an identification code. This QR code is provided for the user to distribute in the store, enabling customers to easily access the menu via an information terminal such as a smartphone or tablet.

[0275] When the customer reads the QR code, the terminal can display the handwritten menu in the selected language. Through this process, customers can select products and place orders through the digital ordering system. This order information is transmitted immediately to the kitchen equipment and service terminals via the server, enabling efficient order processing.

[0276] As a specific example, when offering "Matcha Latte" in a café, if the user handwrites "抹茶ラテ" in Japanese, the system translates it into English as "Matcha Latte", French as "Thé Matcha Latte", Chinese as "抹茶拿铁", etc., and displays it while maintaining the handwritten style.

[0277] Example of prompt text:

[0278] "Perform OCR processing on the image of the handwritten menu, digitize the Japanese text, and translate it into multiple languages. Maintain the handwritten style after translation and create menus in various languages using generative AI."

[0279] The flow of the specific process in Example 1 will be described using FIG. 11.

[0280] Step 1:

[0281] The user obtains the handwritten menu as a digital image using a smartphone or scanner, and uploads the image to the server via a dedicated application or web interface. The input is the image of the handwritten menu, and the output is the digital image data stored on the server. Specifically, the user takes a clear picture of the image using a high-resolution camera, performs preprocessing such as rotation and trimming, and then sends it to the server.

[0282] Step 2:

[0283] The server uses optical character recognition (OCR) technology to read the character information in the received image data. The input is the uploaded digital image, and the output is the identified text data. As a specific operation of OCR, the outline of the characters is extracted and compared with existing character patterns to be converted into digital text.

[0284] Step 3:

[0285] The server extracts the features of the handwritten style contained in the text obtained by OCR using the generative AI model and performs learning. The input is the text data obtained by OCR processing, and the output is the digital data retaining the handwritten style. In this step, the server analyzes the shape data of the characters and models the tendency of the handwriting.

[0286] Step 4:

[0287] The server translates the text from a specific language into multiple languages for major tourists and applies the learned handwritten style. The input is the text data retaining the handwritten style, and the output is the translated text written in the handwritten style font of each language. Specifically, language conversion is performed using a translation API, and style data is applied to the result.

[0288] Step 5:

[0289] The server outputs the generated multilingual handwritten-style fonts as an information provider and generates identification codes (QR codes) for accessing each language. The input is the translated text of the handwritten-style font, and the output is font data and QR codes available on the information provider.

[0290] Step 6:

[0291] The terminal displays a handwritten-style menu in the selected language when the customer scans a QR code. Input is the menu information obtained via the QR code, and output is the handwritten-style menu displayed on the terminal. Specifically, the terminal retrieves data from a server via a URL and displays it at the appropriate display resolution.

[0292] Step 7:

[0293] Customers select items from the displayed menu and place their orders through the digital ordering system. Input is the product information selected by the customer on their terminal, and output is the order data sent to the server. The server transmits this order information in real time to kitchen equipment and service terminals, supporting rapid processing.

[0294] (Application Example 1)

[0295] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0296] In today's world, with the increasing number of global tourists, the food and beverage industry faces the need to provide multilingual menus. However, traditional printed menus struggle to maintain the individuality of handwritten text while also being able to handle orders efficiently. Against this backdrop, there is a need for a system that can handle multiple languages ​​while preserving the warmth of handwritten text, and that can also process orders smoothly.

[0297] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0298] In this invention, the server includes means for converting handwritten characters in the original language into digital data using optical character recognition technology; means for using generation technology to extract and learn the handwriting style of characters from the digital data; and means for translating text from a specific language to multiple other languages ​​and applying the learned handwriting style to the translated text. This enables the efficient generation of multilingual handwritten-style menus that maintain the handwriting style, allowing users to easily access and order from them.

[0299] Optical character recognition (OCR) technology is a technology that reads handwritten or printed characters from an image and converts them into digital text.

[0300] "Digital data" refers to information that has been converted into a format that can be read by a computer, and includes data such as text and images.

[0301] "Handwritten style" refers to the shape and characteristics of handwriting as written by a human being.

[0302] "Generative technology" is a technique that uses machine learning to learn the characteristics of specific data and generate new data.

[0303] A "two-dimensional code" is a code that encodes information into a grid pattern of vertical and horizontal elements, allowing it to be read by a scanner.

[0304] A "terminal device" is an electronic device used by a user to display, input, and manipulate information.

[0305] "Data communication means" refers to methods and technologies for electronically transmitting or receiving information to or from other devices.

[0306] The system of the present invention digitizes handwritten menus in stores in a multilingual manner and efficiently provides them to users. This system operates in cooperation with a server, a terminal, and a user.

[0307] First, the user scans or takes a photo of the handwritten menu as an image and uploads the image data to the server. The server applies optical character recognition technology to this image data and converts the handwritten characters into digital text. In this process, the Python Imaging Library (PIL) is used, and character recognition is performed by pytesseract.

[0308] Next, the server uses a generation technology to learn the handwriting style and perform multilingual translation while maintaining the digitized text. In this operation, the translate library is utilized to translate into major tourist languages (e.g., English, French, Chinese, etc.). Then, the handwriting style is applied to the translated text to generate a foreign language handwritten font.

[0309] Furthermore, a two-dimensional code including an access link for viewing the generated handwritten menu in each language is generated and information is provided on the terminal. At this stage, the qrcode library is used to generate a two-dimensional code so that customers can easily access it on terminals such as smartphones and tablets. The terminal receives the customer's access, displays the handwritten menu in the desired language, and enables immediate ordering through a digital ordering system.

[0310] As a specific example, in a certain café, a Japanese menu is created by hand, and by using this system to generate English, French, and Chinese menus, tourists from various countries can view and order the menu in their own languages. By using this system, multilingualization can be achieved without sacrificing the charm of the handwritten menu, and store operations can be made more efficient.

[0311] Example of a prompt sentence for the generation AI model:

[0312] "Please use OCR technology to convert the following handwritten image into text and translate it into Chinese, English, and French. Maintain the handwritten style while translating as naturally as possible in each language. Finally, generate a QR code corresponding to each language and output it with an encrypted access link."

[0313] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0314] Step 1:

[0315] The user obtains an image file of a handwritten menu by taking a photograph or scanning it, and uploads that data to the server. The input is an image of the handwritten menu, which becomes a digital image file. The output is a status message indicating that the data upload is complete.

[0316] Step 2:

[0317] The server applies optical character recognition (OCR) technology to uploaded image data. The input is an image of a handwritten menu received from the user. The server uses Pytesseract to convert the handwritten characters in the image into text and outputs the result as character data. In this process, the outlines and shapes of the characters are converted into digital data, and the extracted string is generated.

[0318] Step 3:

[0319] The server uses generative technology to learn the features of handwritten styles. The input consists of character data and its shape features obtained by OCR. Based on this, a generative AI model is used to recognize the handwriting and style of each character and output characteristic data. This output includes style information that will be applied in the subsequent translation process.

[0320] Step 4:

[0321] The server translates character data into other languages ​​and applies the initial handwriting style to the new language text. The input is the characteristic data and character data obtained in step 3. The server uses the Translate library to translate into major foreign languages ​​and applies the learned handwriting style to the translation results. The output is handwriting-style font style data for each language.

[0322] Step 5:

[0323] The server generates a two-dimensional code for output to the information provider device based on the handwritten style fonts generated for each language. The input is handwritten style font data for each language. The server uses the QRCode library to create and output a two-dimensional code that contains a link for the user to access the menu in each language.

[0324] Step 6:

[0325] The terminal uses a generated QR code to display a handwritten menu in the language selected by the customer upon access. Input is scanned data from the customer's smartphone or tablet. The terminal downloads the menu in the corresponding language and displays it on the customer's device. Output is a displayed handwritten-style multilingual menu.

[0326] Step 7:

[0327] Customers select an order from the displayed menu and send it to the server via the digital ordering system. The input is the customer's order selection data. The server receives this information, immediately processes it to transmit it to the kitchen and service staff, and generates an output indicating that the order has been processed.

[0328] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0329] This invention is a system that provides interactive menus based on user emotions by incorporating an emotion engine. It provides multilingual menus that retain a handwritten style, recognizes customer emotions, and makes menu suggestions accordingly.

[0330] First, the user provides facial and voice data to the emotion engine through the camera and microphone of a terminal installed in the store. The server analyzes this data and recognizes the emotions the user is feeling. In this process, it determines whether the customer's reaction to the displayed menu items is positive or negative.

[0331] Next, the server suggests a menu tailored to the user based on their perceived emotions. For example, if the user is happy, recommended dishes can be highlighted. Conversely, if the user appears dissatisfied, other options may be presented to draw their attention, providing information that matches the user's state.

[0332] Furthermore, the terminal dynamically updates the menu content in conjunction with an emotion engine. This allows users to always see a menu tailored to their preferences, ensuring a comfortable ordering experience. For example, if a family is visiting and everyone seems satisfied, the terminal can present a set menu and ask if they would like to order it.

[0333] Furthermore, once a user confirms an order, the information is immediately transmitted via the server to the kitchen and related service locations, enabling efficient order processing. The system also stores a history of emotionally-based suggestions, allowing for more personalized service on subsequent visits.

[0334] In this way, the present invention provides not only handwritten-style multilingual menus but also interactive food and beverage services utilizing emotion recognition, thereby constructing an efficient and flexible system that can meet diverse user needs.

[0335] The following describes the processing flow.

[0336] Step 1:

[0337] Users provide facial expressions and voice recordings via cameras and microphones installed on the store's information terminals. This allows the necessary data for the emotion engine to be acquired.

[0338] Step 2:

[0339] The server analyzes the received facial expression and voice data to determine the user's current emotional state. This analysis uses an emotion engine that employs facial expression recognition algorithms and voice analysis algorithms.

[0340] Step 3:

[0341] Based on the recognized emotions, the server suggests dishes likely to suit the user's preferences from a handwritten-style multilingual menu. For example, if a relaxed emotion is detected, information on new menu items and special dishes will be displayed preferentially.

[0342] Step 4:

[0343] The terminal visually highlights suggested menu items for the user. It dynamically updates menu content to support flexible menu selections that respond to the user's emotions.

[0344] Step 5:

[0345] The user selects a dish based on the displayed suggestions and enters their order via an information terminal.

[0346] Step 6:

[0347] The server receives confirmed order information and immediately transmits it to the terminals of the relevant kitchen and service staff. This enables smooth order processing and food delivery.

[0348] Step 7:

[0349] After processing an order, the server saves the user's sentiment data and order history, enabling more personalized menu suggestions for future visits.

[0350] Through this series of processes, improved user experience and efficient store operations are achieved by utilizing the emotion engine.

[0351] (Example 2)

[0352] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0353] Amidst the increasing diversification and globalization of information, there is a growing demand for flexible information delivery tailored to individual needs and emotions. However, conventional systems have struggled to dynamically adjust information based on users' emotional states. Furthermore, there has been a lack of efficient means to deliver personalized information in multilingual environments. This has resulted in a limited user experience and decreased satisfaction.

[0354] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0355] In this invention, the server includes means for digitizing handwritten characters, means for extracting handwriting characteristics from the digital data, and means for acquiring the user's emotional state and dynamically adjusting the information. This enables interactive and personalized information provision that responds to the user's emotions.

[0356] "Handwritten text" refers to text created by a person using a writing instrument on paper or a device.

[0357] "Optical character recognition technology" is a technology that analyzes characters and symbols acquired as images and converts them into digital character data.

[0358] "Digital data" refers to binary data that is a digital representation of analog information.

[0359] "Handwriting characteristics" refer to the unique shape and style of handwritten characters.

[0360] "Generative technology" refers to techniques that create new data using certain types of learning models.

[0361] "Emotional state" refers to the psychological state inferred from the user's facial expression and voice.

[0362] A "generative AI model" is an artificial intelligence model that learns patterns from data based on machine learning algorithms and generates new information.

[0363] "Interactive" refers to a characteristic that dynamically changes in response to user input and possesses two-way operability.

[0364] "Personalization" refers to customizing information and services according to an individual's characteristics and needs.

[0365] An "information device" is an electronic device used for receiving, processing, or transmitting information.

[0366] This invention aims to realize a system that generates multilingual handwritten-style fonts and provides personalized information based on the user's emotional state.

[0367] Users access the system through a GUI displayed on an information terminal. Upon arrival, the terminal's built-in camera and microphone capture the user's facial expressions and voice. This information is processed through an emotion recognition algorithm to analyze the user's emotional state in real time. This analysis utilizes image processing tools such as OpenCV and speech analysis libraries.

[0368] The server runs a generative AI model based on the analyzed sentiment data to generate a personalized menu of suggestions. This AI model uses algorithms learned from a large dataset, enabling it to generate information tailored to the user's individual preferences and circumstances.

[0369] One concrete example is generating a prompt that highlights special desserts or promotions if the user is smiling. An example of a generated prompt might be, "The user is happy, please show us our recommended special menu items."

[0370] Based on the generated prompt, the terminal displays the content using a multilingual handwritten-style font. This allows for information to be presented in a visually appealing way to the user. This handwritten-style font is applied to text translated from the original language into various languages, ensuring style preservation.

[0371] Furthermore, once a user confirms their order, the information is promptly transmitted via the server to the kitchen and related services. This enables smooth, real-time service delivery, improving the user experience.

[0372] Through this system, the present invention integrates a handwritten-style multilingual font with an interactive service utilizing emotion recognition, achieving advanced personalization that meets the diverse needs of users.

[0373] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0374] Step 1:

[0375] The user accesses an information terminal and provides facial expressions and voice to the system via camera and microphone. Input data includes image data and audio data. The terminal uses facial recognition and voice analysis algorithms to capture and preprocess this data in real time. The output is organized facial image data and voice feature data. Specifically, the system captures the user's facial expressions, such as smiling while looking at a menu, and the tone of their voice.

[0376] Step 2:

[0377] The server inputs the organized data received from the terminal into an emotion recognition algorithm. This algorithm uses OpenCV and speech analysis libraries to analyze the user's emotions from facial muscle movements and voice tone. The output at this stage is an emotion label (e.g., joy, surprise, dissatisfaction) that the user is experiencing. Specifically, the server performs a process to identify the user's smile as "joy."

[0378] Step 3:

[0379] The server generates prompts using a generative AI model based on recognized emotion labels. The input includes emotion labels and associated historical data. The AI ​​model analyzes this data to generate context-appropriate suggestions. The output is a user-specific prompt. For example, a prompt such as "Suggest a dessert for a happy user" might be generated.

[0380] Step 4:

[0381] The terminal receives the generated prompt and displays information on the screen using a multilingual handwritten-style font based on it. Visual information corresponding to the prompt is incorporated into the menu. Specifically, recommended desserts are highlighted and displayed to the user in an attractive handwritten font.

[0382] Step 5:

[0383] Once a user confirms an order, the information is automatically transmitted to the relevant facilities via the server. The input is the menu items selected by the user, and the output is the order data communicated to the kitchen and relevant service teams. This ensures that order processing is expedited. Specifically, the selected menu is immediately sent to the internal terminal, and cooking preparation begins.

[0384] (Application Example 2)

[0385] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0386] Conventional menu information systems have the challenge of not being able to respond immediately and flexibly to users' emotions and diverse language needs. In particular, personalized menu suggestions based on users' emotions are difficult, and there is a need for more efficient and individualized ordering processes.

[0387] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0388] In this invention, the server includes means for converting handwritten original language characters into digital data using optical character recognition technology, means for using generation technology to extract and learn the handwriting style of the characters from the digital data, and means for recognizing the user's emotions and dynamically updating and displaying menu information according to those emotions. As a result, users can always receive multilingual menus that are tailored to their emotions and preferences, enabling an efficient and personalized ordering experience.

[0389] "Handwritten style" refers to fonts and characters that imitate the texture and typeface of handwriting.

[0390] "Optical character recognition technology" is a technology that scans images or handwritten characters and converts them into corresponding character codes.

[0391] "Digital data" refers to information that has been converted into a format that can be processed by computers and electronic devices.

[0392] "Generative technology" refers to techniques that use algorithms to create new data and information.

[0393] An "information provision device" refers to a device or hardware used to display information to a user.

[0394] An "information code" refers to an identifier used as a structured form of information, which is necessary for users to access it.

[0395] An "emotion engine" refers to software that analyzes a user's emotions from their facial expressions, voice, etc., and processes those emotions accordingly.

[0396] "Communication means" refers to the technologies and methods used to transmit information to other devices or systems.

[0397] A "commercial facility" refers to a place or space that provides goods or services to consumers.

[0398] To implement this invention, a program is required that ensures the entire system operates in a coordinated manner. The server converts handwritten original language characters into digital data using optical character recognition technology, and then generates a multi-language digital font using a generation technology that extracts handwritten style features from the digital data. This digital font is output to a terminal acting as an information provider.

[0399] The device analyzes the user's facial expressions and voice using an emotion engine through interaction with the user. This allows the server to recognize the emotions the user is feeling towards the displayed menu. Based on these recognized emotions, the device dynamically updates the menu in real time and re-presents it in a visually appealing way.

[0400] In this process, dynamic menu suggestions based on emotion recognition are crucial, and this role is fulfilled by using a generative AI model. Finally, the user's selected order information is immediately transmitted via communication to the relevant cooking departments, etc.

[0401] For example, if a family is enjoying themselves at a restaurant, the device can highlight special family set menus and add recommendations for desserts and side dishes. By providing a prompt such as, "This family seems to be enjoying themselves. Please suggest a set menu," to the AI ​​model, attractive suggestions can be automatically generated.

[0402] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0403] Step 1:

[0404] The server captures handwritten characters with a camera and converts them into digital data using optical character recognition (OCR) technology. The input is image data of the handwritten characters, and the output is a digital character code. This data conversion makes the character information processable in digital format.

[0405] Step 2:

[0406] The server extracts features from the converted digital data to identify handwritten character styles and uses generative techniques for learning. In this process, digital character data is provided as input, and learned handwritten style parameters are generated as output. The generative AI model is then used to reproduce natural-looking handwriting.

[0407] Step 3:

[0408] The device collects the user's facial expressions and voice through its camera and microphone, and analyzes them using an emotion engine. The input to the emotion recognition process is facial image and audio data, and the output is the analyzed emotion (e.g., joy, anger). This output is used as basic data for menu suggestions.

[0409] Step 4:

[0410] The server generates the most suitable menu for the user based on the emotion recognition results. The input includes the user's emotion data and menu information, and the output is an optimized menu tailored to the emotion. This process involves inputting prompt sentences into a generating AI model to provide effective menu suggestions.

[0411] Step 5:

[0412] The terminal displays an optimization menu received from the server to the user. The user can review the menu displayed on the screen and make a selection freely. The output is visually presented menu information, reflecting the user's reactions as they occur during the selection process.

[0413] Step 6:

[0414] Once a user confirms an order, the terminal sends the order details to the server and forwards them to the relevant department. The input is the user's order selection, and the output is the order data. The server immediately transmits this information to the cooking department and other relevant departments via communication means, supporting efficient order processing.

[0415] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0416] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0417] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0418] [Third Embodiment]

[0419] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0420] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0421] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0422] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0423] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0424] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0425] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0426] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0427] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0428] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0429] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0430] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0431] This invention is a system that enables multilingual support for handwritten menus in restaurants, streamlining the generation of multilingual menus and order processing while preserving the individuality of handwritten menus. This system operates through the cooperation of a server, terminals, and users.

[0432] First, the user scans or photographs a handwritten Japanese menu and uploads the image data to the server. The server applies optical character recognition (OCR) technology to the received image data, converting the handwritten text into digital text data. During this process, the shape of each character, the characteristics of the handwriting, and other details are saved as digital data.

[0433] Next, the server uses generation technology to learn the features of the handwriting style and performs multilingual translation while preserving the style of the digitized Japanese text. Translation is performed for major tourist languages ​​(e.g., English, French, Chinese, etc.). This generates multiple foreign language versions of the handwritten menu.

[0434] The server then outputs the generated handwritten fonts for each language to the information provider, which generates QR codes that serve as access links to the menus in each language. Users distribute these QR codes within the store, allowing customers to easily access the menus on their smartphones or tablets.

[0435] The terminal can receive customer access and display a handwritten menu in the selected language. Customers can select items from the menu and place their orders through the digital ordering system. This process allows orders to be instantly transmitted from the server to the kitchen and service terminals.

[0436] In this way, it becomes possible to maintain the warmth of handwritten menus while providing multilingual support for foreign tourists visiting Japan and processing orders efficiently. This system allows restaurants to offer multilingual menus with their own unique handwritten style while ensuring smooth store operations.

[0437] The following describes the processing flow.

[0438] Step 1:

[0439] Users scan or photograph handwritten menus and upload the image files to the server.

[0440] Step 2:

[0441] The server applies optical character recognition (OCR) technology to the received image data, converting handwritten characters into digital text data. During this process, it extracts handwritten style features such as character shape and line thickness.

[0442] Step 3:

[0443] The server uses generative techniques to learn the features of the extracted handwriting styles and generates those styles as models for digital text data.

[0444] Step 4:

[0445] The server uses a translation system to translate digital text from Japanese into other specified languages. During this process, the system applies the handwritten style to the text in each language to ensure it is preserved.

[0446] Step 5:

[0447] The server outputs the generated handwritten style fonts for each language to the information provider and generates a QR code containing an access link to the menu for each language.

[0448] Step 6:

[0449] Users print or digitally display QR codes received from the server and distribute them within the store for easy access by customers.

[0450] Step 7:

[0451] The terminal displays a handwritten menu in the selected language on the device when the customer scans a QR code.

[0452] Step 8:

[0453] The user (as a customer) refers to a multilingual handwritten menu displayed on the terminal, selects the desired products via the digital ordering system, and completes the order.

[0454] Step 9:

[0455] The server receives the order details sent by the user and immediately transfers the information to the kitchen and service terminals within the system, ensuring efficient order processing.

[0456] (Example 1)

[0457] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0458] In restaurants, when providing multilingual options for handwritten menus, it's necessary to efficiently offer menus in multiple languages ​​while preserving the unique character of handwritten menus and speeding up customer order processing. This challenge requires technology that can simultaneously reproduce the handwritten style and provide multilingual translation.

[0459] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0460] In this invention, the server includes means for converting handwritten characters in the original language into digital data using optical character recognition technology; means for using generation technology to extract and learn the handwriting style of characters from the digital data; and means for translating text from a specific language to multiple other languages ​​and applying the learned handwriting style to the translated text. This enables multilingual support while preserving the warmth of handwriting, and allows for fast and efficient order processing.

[0461] Optical character recognition technology is a technology that extracts handwritten or printed character information from image data and converts it into digital text data.

[0462] "Digital data" refers to information that has been converted from analog information, such as handwritten text or images, into a digital format.

[0463] "Generative technology" is a technique that generates new data using algorithms that extract and learn features from specific data.

[0464] "Handwritten style" refers to the shape and characteristics of handwritten characters, and is a typeface that is expressed as a unique style.

[0465] Translation is the process of converting text written in one language into another language.

[0466] An "information display device" is a device that displays digital data and provides information to users visually.

[0467] An "identification code" is a code generated to quickly access specific information; a QR code is an example of this.

[0468] A "digital ordering system" is a system that receives and processes customer orders electronically.

[0469] This invention is a system for making handwritten menus in restaurants multilingual. The system functions through the cooperation of a server, terminals, and users.

[0470] The user first creates a handwritten Japanese menu and then uses a smartphone or scanner to capture it as a digital image. This image is then uploaded to the server using a dedicated application or web interface. This process digitizes the handwritten menu, and the system begins processing it.

[0471] The server utilizes optical character recognition technologies such as Google Cloud Vision API and Microsoft Azure OCR service to convert handwritten characters from received images into digital text. During this conversion process, the shape of the characters and the characteristics of the handwriting are also extracted, and the generating AI model learns from this information. The learned handwriting style is then applied to text in a specific language.

[0472] Subsequently, the server uses the generative AI model to translate the digital text into the language for the main tourists. Google Translate API or DeepL Translator API is used in this translation process. The learned handwriting style is applied to the translated text, and a newly generated multilingual handwritten menu is created. This handwriting style does not damage the individuality of the handwriting and also reflects its characteristics in the multilingual menu.

[0473] After that, the server outputs the menu generated in multiple languages to the information providing device and generates a QR code as an identification code. This QR code is provided for the user to distribute within the store, enabling customers to easily access the menu via an information terminal such as a smartphone or tablet.

[0474] When the customer scans the QR code, the terminal can display the handwritten menu in the selected language. Through this process, the customer selects the product and places an order through the digital ordering system. This order information is immediately transmitted to the kitchen equipment and service terminals via the server, enabling efficient order processing.

[0475] As a specific example, when offering "Matcha Latte" in a café, if the user handwrites "抹茶ラテ" in Japanese, the system translates it into "Matcha Latte" in English, "Thé Matcha Latte" in French, "抹茶拿铁” in Chinese, etc., and displays it while maintaining the handwritten style.

[0476] Example of a prompt sentence:

[0477] "Perform OCR processing on the image of the handwritten menu, digitize the Japanese text, and translate it into multiple languages. Maintain the handwritten style after translation and create language versions of the menu using generative AI."

[0478] The flow of the specific process in Example 1 will be described using FIG. 11.

[0479] Step 1:

[0480] Users acquire handwritten menus as digital images using their smartphones or scanners and upload these images to the server via a dedicated application or web interface. The input is the image of the handwritten menu, and the output is the digital image data stored on the server. Specifically, users take clear photos of the images using a high-resolution camera, perform pre-processing such as rotation and cropping, and then send them to the server.

[0481] Step 2:

[0482] The server uses Optical Character Recognition (OCR) technology to read character information from the received image data. The input is the uploaded digital image, and the output is the identified text data. Specifically, OCR extracts the outlines of characters and matches them with existing character patterns to convert them into digital text.

[0483] Step 3:

[0484] The server uses a generative AI model to extract and learn handwriting style features from the text obtained by OCR. The input is text data obtained through OCR processing, and the output is digital data that retains the handwriting style. In this step, the server analyzes the shape data of the characters and models the tendencies of the handwriting.

[0485] Step 4:

[0486] The server translates text from a specific language into multiple languages ​​targeting major tourists and applies learned handwriting styles. The input is text data that retains handwriting styles, and the output is translated text written in the handwriting style fonts of each language. Specifically, it uses a translation API to perform language conversion and then applies style data to the result.

[0487] Step 5:

[0488] The server outputs the generated multilingual handwritten-style fonts as an information provider and generates identification codes (QR codes) for accessing each language. The input is the translated text of the handwritten-style font, and the output is font data and QR codes available on the information provider.

[0489] Step 6:

[0490] The terminal displays a handwritten-style menu in the selected language when the customer scans a QR code. Input is the menu information obtained via the QR code, and output is the handwritten-style menu displayed on the terminal. Specifically, the terminal retrieves data from a server via a URL and displays it at the appropriate display resolution.

[0491] Step 7:

[0492] Customers select items from the displayed menu and place their orders through the digital ordering system. Input is the product information selected by the customer on their terminal, and output is the order data sent to the server. The server transmits this order information in real time to kitchen equipment and service terminals, supporting rapid processing.

[0493] (Application Example 1)

[0494] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0495] In today's world, with the increasing number of global tourists, the food and beverage industry faces the need to provide multilingual menus. However, traditional printed menus struggle to maintain the individuality of handwritten text while also being able to handle orders efficiently. Against this backdrop, there is a need for a system that can handle multiple languages ​​while preserving the warmth of handwritten text, and that can also process orders smoothly.

[0496] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0497] In this invention, the server includes means for converting handwritten characters in the original language into digital data using optical character recognition technology; means for using generation technology to extract and learn the handwriting style of characters from the digital data; and means for translating text from a specific language to multiple other languages ​​and applying the learned handwriting style to the translated text. This enables the efficient generation of multilingual handwritten-style menus that maintain the handwriting style, allowing users to easily access and order from them.

[0498] Optical character recognition (OCR) technology is a technology that reads handwritten or printed characters from an image and converts them into digital text.

[0499] "Digital data" refers to information that has been converted into a format that can be read by a computer, and includes data such as text and images.

[0500] "Handwritten style" refers to the shape and characteristics of handwriting as written by a human being.

[0501] "Generative technology" is a technique that uses machine learning to learn the characteristics of specific data and generate new data.

[0502] A "two-dimensional code" is a code that encodes information into a grid pattern of vertical and horizontal elements, allowing it to be read by a scanner.

[0503] A "terminal device" is an electronic device used by a user to display, input, and manipulate information.

[0504] "Data communication means" refers to methods and technologies for electronically transmitting or receiving information to or from other devices.

[0505] The present invention provides a system for digitizing handwritten menus in stores in a multilingual format and efficiently delivering them to users. This system operates through the coordinated efforts of a server, terminals, and users.

[0506] The user first scans or photographs the handwritten menu as an image and uploads the image data to the server. The server then applies optical character recognition (OCR) technology to this image data to convert the handwritten characters into digital text. This process uses the Python Imaging Library (PIL) and character recognition is performed by pytesseract.

[0507] Next, the server uses generation technology to learn handwriting styles and perform multilingual translation while maintaining the digitized text. This process utilizes the translate library to translate into major tourist languages ​​(e.g., English, French, Chinese, etc.). Then, the handwriting style is applied to the translated text to generate handwritten-style fonts for the foreign languages.

[0508] Furthermore, a QR code containing an access link to view the handwritten menus in each generated language is generated and provided on the terminal. At this stage, a QR code library is used to generate the QR code, making it easy for customers to access it on devices such as smartphones and tablets. Upon receiving the customer's access, the terminal displays the handwritten menu in the desired language and enables immediate ordering through the digital ordering system.

[0509] As a concrete example, one cafe creates handwritten menus in Japanese and uses this system to generate English, French, and Chinese versions, allowing tourists from various countries to view and order in their own languages. By using this system, they can achieve multilingualism without losing the charm of handwritten menus and streamline store operations.

[0510] Examples of prompts for a generative AI model:

[0511] "Please use OCR technology to convert the following handwritten image into text and translate it into Chinese, English, and French. Maintain the handwritten style while translating as naturally as possible in each language. Finally, generate a QR code corresponding to each language and output it with an encrypted access link."

[0512] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0513] Step 1:

[0514] The user obtains an image file of a handwritten menu by taking a photograph or scanning it, and uploads that data to the server. The input is an image of the handwritten menu, which becomes a digital image file. The output is a status message indicating that the data upload is complete.

[0515] Step 2:

[0516] The server applies optical character recognition (OCR) technology to uploaded image data. The input is an image of a handwritten menu received from the user. The server uses Pytesseract to convert the handwritten characters in the image into text and outputs the result as character data. In this process, the outlines and shapes of the characters are converted into digital data, and the extracted string is generated.

[0517] Step 3:

[0518] The server uses generative technology to learn the features of handwritten styles. The input consists of character data and its shape features obtained by OCR. Based on this, a generative AI model is used to recognize the handwriting and style of each character and output characteristic data. This output includes style information that will be applied in the subsequent translation process.

[0519] Step 4:

[0520] The server translates character data into other languages ​​and applies the initial handwriting style to the new language text. The input is the characteristic data and character data obtained in step 3. The server uses the Translate library to translate into major foreign languages ​​and applies the learned handwriting style to the translation results. The output is handwriting-style font style data for each language.

[0521] Step 5:

[0522] The server generates a two-dimensional code for output to the information provider device based on the handwritten style fonts generated for each language. The input is handwritten style font data for each language. The server uses the QRCode library to create and output a two-dimensional code that contains a link for the user to access the menu in each language.

[0523] Step 6:

[0524] The terminal uses a generated QR code to display a handwritten menu in the language selected by the customer upon access. Input is scanned data from the customer's smartphone or tablet. The terminal downloads the menu in the corresponding language and displays it on the customer's device. Output is a displayed handwritten-style multilingual menu.

[0525] Step 7:

[0526] Customers select an order from the displayed menu and send it to the server via the digital ordering system. The input is the customer's order selection data. The server receives this information, immediately processes it to transmit it to the kitchen and service staff, and generates an output indicating that the order has been processed.

[0527] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0528] This invention is a system that provides interactive menus based on user emotions by incorporating an emotion engine. It provides multilingual menus that retain a handwritten style, recognizes customer emotions, and makes menu suggestions accordingly.

[0529] First, the user provides facial and voice data to the emotion engine through the camera and microphone of a terminal installed in the store. The server analyzes this data and recognizes the emotions the user is feeling. In this process, it determines whether the customer's reaction to the displayed menu items is positive or negative.

[0530] Next, the server suggests a menu tailored to the user based on their perceived emotions. For example, if the user is happy, recommended dishes can be highlighted. Conversely, if the user appears dissatisfied, other options may be presented to draw their attention, providing information that matches the user's state.

[0531] Furthermore, the terminal dynamically updates the menu content in conjunction with an emotion engine. This allows users to always see a menu tailored to their preferences, ensuring a comfortable ordering experience. For example, if a family is visiting and everyone seems satisfied, the terminal can present a set menu and ask if they would like to order it.

[0532] Furthermore, once a user confirms an order, the information is immediately transmitted via the server to the kitchen and related service locations, enabling efficient order processing. The system also stores a history of emotionally-based suggestions, allowing for more personalized service on subsequent visits.

[0533] In this way, the present invention provides not only handwritten-style multilingual menus but also interactive food and beverage services utilizing emotion recognition, thereby constructing an efficient and flexible system that can meet diverse user needs.

[0534] The following describes the processing flow.

[0535] Step 1:

[0536] Users provide facial expressions and voice recordings via cameras and microphones installed on the store's information terminals. This allows the necessary data for the emotion engine to be acquired.

[0537] Step 2:

[0538] The server analyzes the received facial expression and voice data to determine the user's current emotional state. This analysis uses an emotion engine that employs facial expression recognition algorithms and voice analysis algorithms.

[0539] Step 3:

[0540] Based on the recognized emotions, the server suggests dishes likely to suit the user's preferences from a handwritten-style multilingual menu. For example, if a relaxed emotion is detected, information on new menu items and special dishes will be displayed preferentially.

[0541] Step 4:

[0542] The terminal visually highlights suggested menu items for the user. It dynamically updates menu content to support flexible menu selections that respond to the user's emotions.

[0543] Step 5:

[0544] The user selects a dish based on the displayed suggestions and enters their order via an information terminal.

[0545] Step 6:

[0546] The server receives confirmed order information and immediately transmits it to the terminals of the relevant kitchen and service staff. This enables smooth order processing and food delivery.

[0547] Step 7:

[0548] After processing an order, the server saves the user's sentiment data and order history, enabling more personalized menu suggestions for future visits.

[0549] Through this series of processes, improved user experience and efficient store operations are achieved by utilizing the emotion engine.

[0550] (Example 2)

[0551] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0552] Amidst the increasing diversification and globalization of information, there is a growing demand for flexible information delivery tailored to individual needs and emotions. However, conventional systems have struggled to dynamically adjust information based on users' emotional states. Furthermore, there has been a lack of efficient means to deliver personalized information in multilingual environments. This has resulted in a limited user experience and decreased satisfaction.

[0553] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0554] In this invention, the server includes means for digitizing handwritten characters, means for extracting handwriting characteristics from the digital data, and means for acquiring the user's emotional state and dynamically adjusting the information. This enables interactive and personalized information provision that responds to the user's emotions.

[0555] "Handwritten text" refers to text created by a person using a writing instrument on paper or a device.

[0556] "Optical character recognition technology" is a technology that analyzes characters and symbols acquired as images and converts them into digital character data.

[0557] "Digital data" refers to binary data that is a digital representation of analog information.

[0558] "Handwriting characteristics" refer to the unique shape and style of handwritten characters.

[0559] "Generative technology" refers to techniques that create new data using certain types of learning models.

[0560] "Emotional state" refers to the psychological state inferred from the user's facial expression and voice.

[0561] A "generative AI model" is an artificial intelligence model that learns patterns from data based on machine learning algorithms and generates new information.

[0562] "Interactive" refers to a characteristic that dynamically changes in response to user input and possesses two-way operability.

[0563] "Personalization" refers to customizing information and services according to an individual's characteristics and needs.

[0564] An "information device" is an electronic device used for receiving, processing, or transmitting information.

[0565] This invention aims to realize a system that generates multilingual handwritten-style fonts and provides personalized information based on the user's emotional state.

[0566] Users access the system through a GUI displayed on an information terminal. Upon arrival, the terminal's built-in camera and microphone capture the user's facial expressions and voice. This information is processed through an emotion recognition algorithm to analyze the user's emotional state in real time. This analysis utilizes image processing tools such as OpenCV and speech analysis libraries.

[0567] The server runs a generative AI model based on the analyzed sentiment data to generate a personalized menu of suggestions. This AI model uses algorithms learned from a large dataset, enabling it to generate information tailored to the user's individual preferences and circumstances.

[0568] One concrete example is generating a prompt that highlights special desserts or promotions if the user is smiling. An example of a generated prompt might be, "The user is happy, please show us our recommended special menu items."

[0569] Based on the generated prompt, the terminal displays the content using a multilingual handwritten-style font. This allows for information to be presented in a visually appealing way to the user. This handwritten-style font is applied to text translated from the original language into various languages, ensuring style preservation.

[0570] Furthermore, once a user confirms their order, the information is promptly transmitted via the server to the kitchen and related services. This enables smooth, real-time service delivery, improving the user experience.

[0571] Through this system, the present invention integrates a handwritten-style multilingual font with an interactive service utilizing emotion recognition, achieving advanced personalization that meets the diverse needs of users.

[0572] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0573] Step 1:

[0574] The user accesses an information terminal and provides facial expressions and voice to the system via camera and microphone. Input data includes image data and audio data. The terminal uses facial recognition and voice analysis algorithms to capture and preprocess this data in real time. The output is organized facial image data and voice feature data. Specifically, the system captures the user's facial expressions, such as smiling while looking at a menu, and the tone of their voice.

[0575] Step 2:

[0576] The server inputs the organized data received from the terminal into an emotion recognition algorithm. This algorithm uses OpenCV and speech analysis libraries to analyze the user's emotions from facial muscle movements and voice tone. The output at this stage is an emotion label (e.g., joy, surprise, dissatisfaction) that the user is experiencing. Specifically, the server performs a process to identify the user's smile as "joy."

[0577] Step 3:

[0578] The server generates prompts using a generative AI model based on recognized emotion labels. The input includes emotion labels and associated historical data. The AI ​​model analyzes this data to generate context-appropriate suggestions. The output is a user-specific prompt. For example, a prompt such as "Suggest a dessert for a happy user" might be generated.

[0579] Step 4:

[0580] The terminal receives the generated prompt and displays information on the screen using a multilingual handwritten-style font based on it. Visual information corresponding to the prompt is incorporated into the menu. Specifically, recommended desserts are highlighted and displayed to the user in an attractive handwritten font.

[0581] Step 5:

[0582] Once a user confirms an order, the information is automatically transmitted to the relevant facilities via the server. The input is the menu items selected by the user, and the output is the order data communicated to the kitchen and relevant service teams. This ensures that order processing is expedited. Specifically, the selected menu is immediately sent to the internal terminal, and cooking preparation begins.

[0583] (Application Example 2)

[0584] Next, we will explain Application Example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0585] Conventional menu information systems have the challenge of not being able to respond immediately and flexibly to users' emotions and diverse language needs. In particular, personalized menu suggestions based on users' emotions are difficult, and there is a need for more efficient and individualized ordering processes.

[0586] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0587] In this invention, the server includes means for converting handwritten original language characters into digital data using optical character recognition technology, means for using generation technology to extract and learn the handwriting style of the characters from the digital data, and means for recognizing the user's emotions and dynamically updating and displaying menu information according to those emotions. As a result, users can always receive multilingual menus that are tailored to their emotions and preferences, enabling an efficient and personalized ordering experience.

[0588] "Handwritten style" refers to fonts and characters that imitate the texture and typeface of handwriting.

[0589] "Optical character recognition technology" is a technology that scans images or handwritten characters and converts them into corresponding character codes.

[0590] "Digital data" refers to information that has been converted into a format that can be processed by computers and electronic devices.

[0591] "Generative technology" refers to techniques that use algorithms to create new data and information.

[0592] An "information provision device" refers to a device or hardware used to display information to a user.

[0593] An "information code" refers to an identifier used as a structured form of information, which is necessary for users to access it.

[0594] An "emotion engine" refers to software that analyzes a user's emotions from their facial expressions, voice, etc., and processes those emotions accordingly.

[0595] "Communication means" refers to the technologies and methods used to transmit information to other devices or systems.

[0596] A "commercial facility" refers to a place or space that provides goods or services to consumers.

[0597] To implement this invention, a program is required that ensures the entire system operates in a coordinated manner. The server converts handwritten original language characters into digital data using optical character recognition technology, and then generates a multi-language digital font using a generation technology that extracts handwritten style features from the digital data. This digital font is output to a terminal acting as an information provider.

[0598] The device analyzes the user's facial expressions and voice using an emotion engine through interaction with the user. This allows the server to recognize the emotions the user is feeling towards the displayed menu. Based on these recognized emotions, the device dynamically updates the menu in real time and re-presents it in a visually appealing way.

[0599] In this process, dynamic menu suggestions based on emotion recognition are crucial, and this role is fulfilled by using a generative AI model. Finally, the user's selected order information is immediately transmitted via communication to the relevant cooking departments, etc.

[0600] For example, if a family is enjoying themselves at a restaurant, the device can highlight special family set menus and add recommendations for desserts and side dishes. By providing a prompt such as, "This family seems to be enjoying themselves. Please suggest a set menu," to the AI ​​model, attractive suggestions can be automatically generated.

[0601] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0602] Step 1:

[0603] The server captures handwritten characters with a camera and converts them into digital data using optical character recognition (OCR) technology. The input is image data of the handwritten characters, and the output is a digital character code. This data conversion makes the character information processable in digital format.

[0604] Step 2:

[0605] The server extracts features from the converted digital data to identify handwritten character styles and uses generative techniques for learning. In this process, digital character data is provided as input, and learned handwritten style parameters are generated as output. The generative AI model is then used to reproduce natural-looking handwriting.

[0606] Step 3:

[0607] The device collects the user's facial expressions and voice through its camera and microphone, and analyzes them using an emotion engine. The input to the emotion recognition process is facial image and audio data, and the output is the analyzed emotion (e.g., joy, anger). This output is used as basic data for menu suggestions.

[0608] Step 4:

[0609] The server generates the most suitable menu for the user based on the emotion recognition results. The input includes the user's emotion data and menu information, and the output is an optimized menu tailored to the emotion. This process involves inputting prompt sentences into a generating AI model to provide effective menu suggestions.

[0610] Step 5:

[0611] The terminal displays an optimization menu received from the server to the user. The user can review the menu displayed on the screen and make a selection freely. The output is visually presented menu information, reflecting the user's reactions as they occur during the selection process.

[0612] Step 6:

[0613] Once a user confirms an order, the terminal sends the order details to the server and forwards them to the relevant department. The input is the user's order selection, and the output is the order data. The server immediately transmits this information to the cooking department and other relevant departments via communication means, supporting efficient order processing.

[0614] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0615] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0616] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0617] [Fourth Embodiment]

[0618] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0619] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0620] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0621] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0622] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0623] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0624] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0625] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0626] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0627] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0628] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0629] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0630] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0631] This invention is a system that enables multilingual support for handwritten menus in restaurants, streamlining the generation of multilingual menus and order processing while preserving the individuality of handwritten menus. This system operates through the cooperation of a server, terminals, and users.

[0632] First, the user scans or photographs a handwritten Japanese menu and uploads the image data to the server. The server applies optical character recognition (OCR) technology to the received image data, converting the handwritten text into digital text data. During this process, the shape of each character, the characteristics of the handwriting, and other details are saved as digital data.

[0633] Next, the server uses generation technology to learn the features of the handwriting style and performs multilingual translation while preserving the style of the digitized Japanese text. Translation is performed for major tourist languages ​​(e.g., English, French, Chinese, etc.). This generates multiple foreign language versions of the handwritten menu.

[0634] The server then outputs the generated handwritten fonts for each language to the information provider, which generates QR codes that serve as access links to the menus in each language. Users distribute these QR codes within the store, allowing customers to easily access the menus on their smartphones or tablets.

[0635] The terminal can receive customer access and display a handwritten menu in the selected language. Customers can select items from the menu and place their orders through the digital ordering system. This process allows orders to be instantly transmitted from the server to the kitchen and service terminals.

[0636] In this way, it becomes possible to maintain the warmth of handwritten menus while providing multilingual support for foreign tourists visiting Japan and processing orders efficiently. This system allows restaurants to offer multilingual menus with their own unique handwritten style while ensuring smooth store operations.

[0637] The following describes the processing flow.

[0638] Step 1:

[0639] Users scan or photograph handwritten menus and upload the image files to the server.

[0640] Step 2:

[0641] The server applies optical character recognition (OCR) technology to the received image data, converting handwritten characters into digital text data. During this process, it extracts handwritten style features such as character shape and line thickness.

[0642] Step 3:

[0643] The server uses generative techniques to learn the features of the extracted handwriting styles and generates those styles as models for digital text data.

[0644] Step 4:

[0645] The server uses a translation system to translate digital text from Japanese into other specified languages. During this process, the system applies the handwritten style to the text in each language to ensure it is preserved.

[0646] Step 5:

[0647] The server outputs the generated handwritten style fonts for each language to the information provider and generates a QR code containing an access link to the menu for each language.

[0648] Step 6:

[0649] Users print or digitally display QR codes received from the server and distribute them within the store for easy access by customers.

[0650] Step 7:

[0651] The terminal displays a handwritten menu in the selected language on the device when the customer scans a QR code.

[0652] Step 8:

[0653] The user (as a customer) refers to a multilingual handwritten menu displayed on the terminal, selects the desired products via the digital ordering system, and completes the order.

[0654] Step 9:

[0655] The server receives the order details sent by the user and immediately transfers the information to the kitchen and service terminals within the system, ensuring efficient order processing.

[0656] (Example 1)

[0657] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0658] In restaurants, when providing multilingual options for handwritten menus, it's necessary to efficiently offer menus in multiple languages ​​while preserving the unique character of handwritten menus and speeding up customer order processing. This challenge requires technology that can simultaneously reproduce the handwritten style and provide multilingual translation.

[0659] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0660] In this invention, the server includes means for converting handwritten characters in the original language into digital data using optical character recognition technology; means for using generation technology to extract and learn the handwriting style of characters from the digital data; and means for translating text from a specific language to multiple other languages ​​and applying the learned handwriting style to the translated text. This enables multilingual support while preserving the warmth of handwriting, and allows for fast and efficient order processing.

[0661] Optical character recognition technology is a technology that extracts handwritten or printed character information from image data and converts it into digital text data.

[0662] "Digital data" refers to information that has been converted from analog information, such as handwritten text or images, into a digital format.

[0663] "Generative technology" is a technique that generates new data using algorithms that extract and learn features from specific data.

[0664] "Handwritten style" refers to the shape and characteristics of handwritten characters, and is a typeface that is expressed as a unique style.

[0665] Translation is the process of converting text written in one language into another language.

[0666] An "information display device" is a device that displays digital data and provides information to users visually.

[0667] An "identification code" is a code generated to quickly access specific information; a QR code is an example of this.

[0668] A "digital ordering system" is a system that receives and processes customer orders electronically.

[0669] This invention is a system for making handwritten menus in restaurants multilingual. The system functions through the cooperation of a server, terminals, and users.

[0670] The user first creates a handwritten Japanese menu and then uses a smartphone or scanner to capture it as a digital image. This image is then uploaded to the server using a dedicated application or web interface. This process digitizes the handwritten menu, and the system begins processing it.

[0671] The server utilizes optical character recognition technologies such as Google Cloud Vision API and Microsoft Azure OCR service to convert handwritten characters in the received image into digital text. In this conversion process, the shape of the characters and the characteristics of the handwriting are also extracted, and the generative AI model learns from them. The learned handwriting style is applied to the text in a specific language.

[0672] Subsequently, the server uses the generative AI model to translate the digital text into the main languages for tourists. Google Translate API and DeepL Translator API are used in this translation process. The learned handwriting style is applied to the translated text, and a newly generated multilingual handwritten menu is created. This handwriting style does not damage the individuality of the handwriting and reflects its characteristics in the multilingual menu.

[0673] After that, the server outputs the menus generated in multiple languages to the information providing device and generates a QR code as an identification code. This QR code is provided for the user to distribute within the store, enabling customers to easily access the menu via an information terminal such as a smartphone or tablet.

[0674] When the customer reads the QR code, the terminal can display the handwritten menu in the selected language. Through this process, the customer selects the product and places an order through the digital ordering system. This order information is immediately transmitted to the kitchen equipment and service terminals via the server, enabling efficient order processing.

[0675] As a specific example, when a café offers "Matcha Latte", if the user handwrites "抹茶ラテ" in Japanese, the system translates it into English as "Matcha Latte", French as "Thé Matcha Latte", Chinese as "抹茶拿铁", etc., and displays it while maintaining the handwritten style.

[0676] Examples of prompt sentences:

[0677] "We process handwritten menu images using OCR (optical character recognition) to digitize the Japanese text and translate it into multiple languages. The handwritten style is maintained after translation, and AI is used to create menus in various languages."

[0678] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0679] Step 1:

[0680] Users acquire handwritten menus as digital images using their smartphones or scanners and upload these images to the server via a dedicated application or web interface. The input is the image of the handwritten menu, and the output is the digital image data stored on the server. Specifically, users take clear photos of the images using a high-resolution camera, perform pre-processing such as rotation and cropping, and then send them to the server.

[0681] Step 2:

[0682] The server uses Optical Character Recognition (OCR) technology to read character information from the received image data. The input is the uploaded digital image, and the output is the identified text data. Specifically, OCR extracts the outlines of characters and matches them with existing character patterns to convert them into digital text.

[0683] Step 3:

[0684] The server uses a generative AI model to extract and learn handwriting style features from the text obtained by OCR. The input is text data obtained through OCR processing, and the output is digital data that retains the handwriting style. In this step, the server analyzes the shape data of the characters and models the tendencies of the handwriting.

[0685] Step 4:

[0686] The server translates text from a specific language into multiple languages ​​targeting major tourists and applies learned handwriting styles. The input is text data that retains handwriting styles, and the output is translated text written in the handwriting style fonts of each language. Specifically, it uses a translation API to perform language conversion and then applies style data to the result.

[0687] Step 5:

[0688] The server outputs the generated multilingual handwritten-style fonts as an information provider and generates identification codes (QR codes) for accessing each language. The input is the translated text of the handwritten-style font, and the output is font data and QR codes available on the information provider.

[0689] Step 6:

[0690] The terminal displays a handwritten-style menu in the selected language when the customer scans a QR code. Input is the menu information obtained via the QR code, and output is the handwritten-style menu displayed on the terminal. Specifically, the terminal retrieves data from a server via a URL and displays it at the appropriate display resolution.

[0691] Step 7:

[0692] Customers select items from the displayed menu and place their orders through the digital ordering system. Input is the product information selected by the customer on their terminal, and output is the order data sent to the server. The server transmits this order information in real time to kitchen equipment and service terminals, supporting rapid processing.

[0693] (Application Example 1)

[0694] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0695] In today's world, with the increasing number of global tourists, the food and beverage industry faces the need to provide multilingual menus. However, traditional printed menus struggle to maintain the individuality of handwritten text while also being able to handle orders efficiently. Against this backdrop, there is a need for a system that can handle multiple languages ​​while preserving the warmth of handwritten text, and that can also process orders smoothly.

[0696] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0697] In this invention, the server includes means for converting handwritten characters in the original language into digital data using optical character recognition technology; means for using generation technology to extract and learn the handwriting style of characters from the digital data; and means for translating text from a specific language to multiple other languages ​​and applying the learned handwriting style to the translated text. This enables the efficient generation of multilingual handwritten-style menus that maintain the handwriting style, allowing users to easily access and order from them.

[0698] Optical character recognition (OCR) technology is a technology that reads handwritten or printed characters from an image and converts them into digital text.

[0699] "Digital data" refers to information that has been converted into a format that can be read by a computer, and includes data such as text and images.

[0700] "Handwritten style" refers to the shape and characteristics of handwriting as written by a human being.

[0701] "Generative technology" is a technique that uses machine learning to learn the characteristics of specific data and generate new data.

[0702] A "two-dimensional code" is a code that encodes information into a grid pattern of vertical and horizontal elements, allowing it to be read by a scanner.

[0703] A "terminal device" is an electronic device used by a user to display, input, and manipulate information.

[0704] "Data communication means" refers to methods and technologies for electronically transmitting or receiving information to or from other devices.

[0705] The present invention provides a system for digitizing handwritten menus in stores in a multilingual format and efficiently delivering them to users. This system operates through the coordinated efforts of a server, terminals, and users.

[0706] The user first scans or photographs the handwritten menu as an image and uploads the image data to the server. The server then applies optical character recognition (OCR) technology to this image data to convert the handwritten characters into digital text. This process uses the Python Imaging Library (PIL) and character recognition is performed by pytesseract.

[0707] Next, the server uses generation technology to learn handwriting styles and perform multilingual translation while maintaining the digitized text. This process utilizes the translate library to translate into major tourist languages ​​(e.g., English, French, Chinese, etc.). Then, the handwriting style is applied to the translated text to generate handwritten-style fonts for the foreign languages.

[0708] Furthermore, a QR code containing an access link to view the handwritten menus in each generated language is generated and provided on the terminal. At this stage, a QR code library is used to generate the QR code, making it easy for customers to access it on devices such as smartphones and tablets. Upon receiving the customer's access, the terminal displays the handwritten menu in the desired language and enables immediate ordering through the digital ordering system.

[0709] As a concrete example, one cafe creates handwritten menus in Japanese and uses this system to generate English, French, and Chinese versions, allowing tourists from various countries to view and order in their own languages. By using this system, they can achieve multilingualism without losing the charm of handwritten menus and streamline store operations.

[0710] Examples of prompts for a generative AI model:

[0711] "Please use OCR technology to convert the following handwritten image into text and translate it into Chinese, English, and French. Maintain the handwritten style while translating as naturally as possible in each language. Finally, generate a QR code corresponding to each language and output it with an encrypted access link."

[0712] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0713] Step 1:

[0714] The user obtains an image file of a handwritten menu by taking a photograph or scanning it, and uploads that data to the server. The input is an image of the handwritten menu, which becomes a digital image file. The output is a status message indicating that the data upload is complete.

[0715] Step 2:

[0716] The server applies optical character recognition (OCR) technology to uploaded image data. The input is an image of a handwritten menu received from the user. The server uses Pytesseract to convert the handwritten characters in the image into text and outputs the result as character data. In this process, the outlines and shapes of the characters are converted into digital data, and the extracted string is generated.

[0717] Step 3:

[0718] The server uses generative technology to learn the features of handwritten styles. The input consists of character data and its shape features obtained by OCR. Based on this, a generative AI model is used to recognize the handwriting and style of each character and output characteristic data. This output includes style information that will be applied in the subsequent translation process.

[0719] Step 4:

[0720] The server translates character data into other languages ​​and applies the initial handwriting style to the new language text. The input is the characteristic data and character data obtained in step 3. The server uses the Translate library to translate into major foreign languages ​​and applies the learned handwriting style to the translation results. The output is handwriting-style font style data for each language.

[0721] Step 5:

[0722] The server generates a two-dimensional code for output to the information provider device based on the handwritten style fonts generated for each language. The input is handwritten style font data for each language. The server uses the QRCode library to create and output a two-dimensional code that contains a link for the user to access the menu in each language.

[0723] Step 6:

[0724] The terminal uses a generated QR code to display a handwritten menu in the language selected by the customer upon access. Input is scanned data from the customer's smartphone or tablet. The terminal downloads the menu in the corresponding language and displays it on the customer's device. Output is a displayed handwritten-style multilingual menu.

[0725] Step 7:

[0726] Customers select an order from the displayed menu and send it to the server via the digital ordering system. The input is the customer's order selection data. The server receives this information, immediately processes it to transmit it to the kitchen and service staff, and generates an output indicating that the order has been processed.

[0727] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0728] This invention is a system that provides interactive menus based on user emotions by incorporating an emotion engine. It provides multilingual menus that retain a handwritten style, recognizes customer emotions, and makes menu suggestions accordingly.

[0729] First, the user provides facial and voice data to the emotion engine through the camera and microphone of a terminal installed in the store. The server analyzes this data and recognizes the emotions the user is feeling. In this process, it determines whether the customer's reaction to the displayed menu items is positive or negative.

[0730] Next, the server suggests a menu tailored to the user based on their perceived emotions. For example, if the user is happy, recommended dishes can be highlighted. Conversely, if the user appears dissatisfied, other options may be presented to draw their attention, providing information that matches the user's state.

[0731] Furthermore, the terminal dynamically updates the menu content in conjunction with an emotion engine. This allows users to always see a menu tailored to their preferences, ensuring a comfortable ordering experience. For example, if a family is visiting and everyone seems satisfied, the terminal can present a set menu and ask if they would like to order it.

[0732] Furthermore, once a user confirms an order, the information is immediately transmitted via the server to the kitchen and related service locations, enabling efficient order processing. The system also stores a history of emotionally-based suggestions, allowing for more personalized service on subsequent visits.

[0733] In this way, the present invention provides not only handwritten-style multilingual menus but also interactive food and beverage services utilizing emotion recognition, thereby constructing an efficient and flexible system that can meet diverse user needs.

[0734] The following describes the processing flow.

[0735] Step 1:

[0736] Users provide facial expressions and voice recordings via cameras and microphones installed on the store's information terminals. This allows the necessary data for the emotion engine to be acquired.

[0737] Step 2:

[0738] The server analyzes the received facial expression and voice data to determine the user's current emotional state. This analysis uses an emotion engine that employs facial expression recognition algorithms and voice analysis algorithms.

[0739] Step 3:

[0740] Based on the recognized emotions, the server suggests dishes likely to suit the user's preferences from a handwritten-style multilingual menu. For example, if a relaxed emotion is detected, information on new menu items and special dishes will be displayed preferentially.

[0741] Step 4:

[0742] The terminal visually highlights suggested menu items for the user. It dynamically updates menu content to support flexible menu selections that respond to the user's emotions.

[0743] Step 5:

[0744] The user selects a dish based on the displayed suggestions and enters their order via an information terminal.

[0745] Step 6:

[0746] The server receives confirmed order information and immediately transmits it to the terminals of the relevant kitchen and service staff. This enables smooth order processing and food delivery.

[0747] Step 7:

[0748] After processing an order, the server saves the user's sentiment data and order history, enabling more personalized menu suggestions for future visits.

[0749] Through this series of processes, improved user experience and efficient store operations are achieved by utilizing the emotion engine.

[0750] (Example 2)

[0751] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0752] Amidst the increasing diversification and globalization of information, there is a growing demand for flexible information delivery tailored to individual needs and emotions. However, conventional systems have struggled to dynamically adjust information based on users' emotional states. Furthermore, there has been a lack of efficient means to deliver personalized information in multilingual environments. This has resulted in a limited user experience and decreased satisfaction.

[0753] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0754] In this invention, the server includes means for digitizing handwritten characters, means for extracting handwriting characteristics from the digital data, and means for acquiring the user's emotional state and dynamically adjusting the information. This enables interactive and personalized information provision that responds to the user's emotions.

[0755] "Handwritten text" refers to text created by a person using a writing instrument on paper or a device.

[0756] "Optical character recognition technology" is a technology that analyzes characters and symbols acquired as images and converts them into digital character data.

[0757] "Digital data" refers to binary data that is a digital representation of analog information.

[0758] "Handwriting characteristics" refer to the unique shape and style of handwritten characters.

[0759] "Generative technology" refers to techniques that create new data using certain types of learning models.

[0760] "Emotional state" refers to the psychological state inferred from the user's facial expression and voice.

[0761] A "generative AI model" is an artificial intelligence model that learns patterns from data based on machine learning algorithms and generates new information.

[0762] "Interactive" refers to a characteristic that dynamically changes in response to user input and possesses two-way operability.

[0763] "Personalization" refers to customizing information and services according to an individual's characteristics and needs.

[0764] An "information device" is an electronic device used for receiving, processing, or transmitting information.

[0765] This invention aims to realize a system that generates multilingual handwritten-style fonts and provides personalized information based on the user's emotional state.

[0766] Users access the system through a GUI displayed on an information terminal. Upon arrival, the terminal's built-in camera and microphone capture the user's facial expressions and voice. This information is processed through an emotion recognition algorithm to analyze the user's emotional state in real time. This analysis utilizes image processing tools such as OpenCV and speech analysis libraries.

[0767] The server runs a generative AI model based on the analyzed sentiment data to generate a personalized menu of suggestions. This AI model uses algorithms learned from a large dataset, enabling it to generate information tailored to the user's individual preferences and circumstances.

[0768] One concrete example is generating a prompt that highlights special desserts or promotions if the user is smiling. An example of a generated prompt might be, "The user is happy, please show us our recommended special menu items."

[0769] Based on the generated prompt, the terminal displays the content using a multilingual handwritten-style font. This allows for information to be presented in a visually appealing way to the user. This handwritten-style font is applied to text translated from the original language into various languages, ensuring style preservation.

[0770] Furthermore, once a user confirms their order, the information is promptly transmitted via the server to the kitchen and related services. This enables smooth, real-time service delivery, improving the user experience.

[0771] Through this system, the present invention integrates a handwritten-style multilingual font with an interactive service utilizing emotion recognition, achieving advanced personalization that meets the diverse needs of users.

[0772] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0773] Step 1:

[0774] The user accesses an information terminal and provides facial expressions and voice to the system via camera and microphone. Input data includes image data and audio data. The terminal uses facial recognition and voice analysis algorithms to capture and preprocess this data in real time. The output is organized facial image data and voice feature data. Specifically, the system captures the user's facial expressions, such as smiling while looking at a menu, and the tone of their voice.

[0775] Step 2:

[0776] The server inputs the organized data received from the terminal into an emotion recognition algorithm. This algorithm uses OpenCV and speech analysis libraries to analyze the user's emotions from facial muscle movements and voice tone. The output at this stage is an emotion label (e.g., joy, surprise, dissatisfaction) that the user is experiencing. Specifically, the server performs a process to identify the user's smile as "joy."

[0777] Step 3:

[0778] The server generates prompts using a generative AI model based on recognized emotion labels. The input includes emotion labels and associated historical data. The AI ​​model analyzes this data to generate context-appropriate suggestions. The output is a user-specific prompt. For example, a prompt such as "Suggest a dessert for a happy user" might be generated.

[0779] Step 4:

[0780] The terminal receives the generated prompt and displays information on the screen using a multilingual handwritten-style font based on it. Visual information corresponding to the prompt is incorporated into the menu. Specifically, recommended desserts are highlighted and displayed to the user in an attractive handwritten font.

[0781] Step 5:

[0782] Once a user confirms an order, the information is automatically transmitted to the relevant facilities via the server. The input is the menu items selected by the user, and the output is the order data communicated to the kitchen and relevant service teams. This ensures that order processing is expedited. Specifically, the selected menu is immediately sent to the internal terminal, and cooking preparation begins.

[0783] (Application Example 2)

[0784] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0785] Conventional menu information systems have the challenge of not being able to respond immediately and flexibly to users' emotions and diverse language needs. In particular, personalized menu suggestions based on users' emotions are difficult, and there is a need for more efficient and individualized ordering processes.

[0786] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0787] In this invention, the server includes means for converting handwritten original language characters into digital data using optical character recognition technology, means for using generation technology to extract and learn the handwriting style of the characters from the digital data, and means for recognizing the user's emotions and dynamically updating and displaying menu information according to those emotions. As a result, users can always receive multilingual menus that are tailored to their emotions and preferences, enabling an efficient and personalized ordering experience.

[0788] "Handwritten style" refers to fonts and characters that imitate the texture and typeface of handwriting.

[0789] "Optical character recognition technology" is a technology that scans images or handwritten characters and converts them into corresponding character codes.

[0790] "Digital data" refers to information that has been converted into a format that can be processed by computers and electronic devices.

[0791] "Generative technology" refers to techniques that use algorithms to create new data and information.

[0792] An "information provision device" refers to a device or hardware used to display information to a user.

[0793] An "information code" refers to an identifier used as a structured form of information, which is necessary for users to access it.

[0794] An "emotion engine" refers to software that analyzes a user's emotions from their facial expressions, voice, etc., and processes those emotions accordingly.

[0795] "Communication means" refers to the technologies and methods used to transmit information to other devices or systems.

[0796] A "commercial facility" refers to a place or space that provides goods or services to consumers.

[0797] To implement this invention, a program is required that ensures the entire system operates in a coordinated manner. The server converts handwritten original language characters into digital data using optical character recognition technology, and then generates a multi-language digital font using a generation technology that extracts handwritten style features from the digital data. This digital font is output to a terminal acting as an information provider.

[0798] The device analyzes the user's facial expressions and voice using an emotion engine through interaction with the user. This allows the server to recognize the emotions the user is feeling towards the displayed menu. Based on these recognized emotions, the device dynamically updates the menu in real time and re-presents it in a visually appealing way.

[0799] In this process, dynamic menu suggestions based on emotion recognition are crucial, and this role is fulfilled by using a generative AI model. Finally, the user's selected order information is immediately transmitted via communication to the relevant cooking departments, etc.

[0800] For example, if a family is enjoying themselves at a restaurant, the device can highlight special family set menus and add recommendations for desserts and side dishes. By providing a prompt such as, "This family seems to be enjoying themselves. Please suggest a set menu," to the AI ​​model, attractive suggestions can be automatically generated.

[0801] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0802] Step 1:

[0803] The server captures handwritten characters with a camera and converts them into digital data using optical character recognition (OCR) technology. The input is image data of the handwritten characters, and the output is a digital character code. This data conversion makes the character information processable in digital format.

[0804] Step 2:

[0805] The server extracts features from the converted digital data to identify handwritten character styles and uses generative techniques for learning. In this process, digital character data is provided as input, and learned handwritten style parameters are generated as output. The generative AI model is then used to reproduce natural-looking handwriting.

[0806] Step 3:

[0807] The device collects the user's facial expressions and voice through its camera and microphone, and analyzes them using an emotion engine. The input to the emotion recognition process is facial image and audio data, and the output is the analyzed emotion (e.g., joy, anger). This output is used as basic data for menu suggestions.

[0808] Step 4:

[0809] The server generates the most suitable menu for the user based on the emotion recognition results. The input includes the user's emotion data and menu information, and the output is an optimized menu tailored to the emotion. This process involves inputting prompt sentences into a generating AI model to provide effective menu suggestions.

[0810] Step 5:

[0811] The terminal displays an optimization menu received from the server to the user. The user can review the menu displayed on the screen and make a selection freely. The output is visually presented menu information, reflecting the user's reactions as they occur during the selection process.

[0812] Step 6:

[0813] Once a user confirms an order, the terminal sends the order details to the server and forwards them to the relevant department. The input is the user's order selection, and the output is the order data. The server immediately transmits this information to the cooking department and other relevant departments via communication means, supporting efficient order processing.

[0814] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0815] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0816] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0817] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0818] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0819] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0820] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0821] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0822] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0823] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0824] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0825] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0826] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0827] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0828] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0829] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0830] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0831] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0832] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0833] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0834] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0835] The following is further disclosed regarding the embodiments described above.

[0836] (Claim 1)

[0837] A means of converting handwritten original language characters into digital data using optical character recognition technology,

[0838] A method using generative technology that extracts and learns handwriting styles from digital data,

[0839] A means of translating text from one language to several other languages ​​and applying a learned handwriting style to the translated text,

[0840] A means for outputting the generated multilingual handwritten style font as an information providing device,

[0841] A means of generating a QR code as a means of accessing the generated multilingual handwritten style font,

[0842] A system that includes this.

[0843] (Claim 2)

[0844] The system according to claim 1, comprising means for distributing the generated multilingual handwritten style font and QR code within the store and enabling customers to access them using information terminals.

[0845] (Claim 3)

[0846] The system according to claim 1, comprising means for instantly transmitting order information electronically by linking an information provision device using a generated handwritten style font with a digital ordering system.

[0847] "Example 1"

[0848] (Claim 1)

[0849] A means of converting handwritten original language characters into digital data using optical character recognition technology,

[0850] A method using generative technology that extracts and learns handwriting styles from digital data,

[0851] A means of translating text from one language to several other languages ​​and applying a learned handwriting style to the translated text,

[0852] A means for outputting the generated multilingual handwritten style font as an information display device,

[0853] A means for generating an identification code as a means for accessing the generated multilingual handwritten style font,

[0854] A means of displaying a handwritten-style font of the translation language selected by the customer in conjunction with an information display device,

[0855] A means of instantly transmitting selections to kitchen equipment via a digital ordering system,

[0856] A system that includes this.

[0857] (Claim 2)

[0858] The system according to claim 1, comprising means for providing the generated multilingual handwritten style fonts and identification codes at the point of use and making them accessible to customers using an information terminal.

[0859] (Claim 3)

[0860] The system according to claim 1, comprising means for instantly transmitting selected information electronically by linking an information display device featuring a generated handwritten-style font with a digital ordering system.

[0861] "Application Example 1"

[0862] (Claim 1)

[0863] A means of converting handwritten original language characters into digital data using optical character recognition technology,

[0864] A method using generative technology that extracts and learns handwriting styles from digital data,

[0865] A means of translating text from one language to several other languages ​​and applying a learned handwriting style to the translated text,

[0866] A means for outputting the generated multilingual handwritten style font as an information providing device,

[0867] A means of generating a two-dimensional code as a means of accessing the generated multilingual handwritten style font,

[0868] A means comprising a terminal device that provides an interface for viewing and selecting from digitized menus,

[0869] A data communication means for instantly transmitting order information electronically based on a generated two-dimensional code,

[0870] A system that includes this.

[0871] (Claim 2)

[0872] The system according to claim 1, comprising means for distributing the generated multilingual handwritten style font and two-dimensional code within the business office and enabling users to access them using information terminals.

[0873] (Claim 3)

[0874] The system according to claim 1, comprising means for instantly transmitting instruction information electronically by linking an information provision device using a generated handwritten style font with a digital instruction system.

[0875] "Example 2 of combining an emotion engine"

[0876] (Claim 1)

[0877] A means of converting handwritten original language characters into digital data using optical character recognition technology,

[0878] A method using generative technology that extracts and learns the characteristics of handwriting from digital data,

[0879] A means for translating information from a specific symbol to many types of symbols, and applying learned handwriting characteristics to the translated information,

[0880] A means for outputting the generated multilingual handwriting style from a machine device,

[0881] A means for acquiring the user's emotional state and dynamically adjusting the information provided based on the acquired emotional state,

[0882] A method for selecting information that matches the emotional state using a generative AI model,

[0883] A means of instantly transmitting order information electronically,

[0884] A system that includes this.

[0885] (Claim 2)

[0886] The system according to claim 1, comprising means for providing generated multilingual handwriting styles and means for accessing them within a location, and for making them accessible to customers using an information device.

[0887] (Claim 3)

[0888] The system according to claim 1, comprising means for optimizing the user experience by generating personalized information in response to the user's emotions using a generative AI model and displaying it in an identifiable manner on the user interface.

[0889] "Application example 2 when combining with an emotional engine"

[0890] (Claim 1)

[0891] A means of converting handwritten original language characters into digital data using optical character recognition technology,

[0892] A method using generative technology that extracts and learns handwriting styles from digital data,

[0893] A means of translating text from one language to several other languages ​​and applying a learned handwriting style to the translated text,

[0894] A means for outputting the generated multilingual handwritten style font as an information providing device,

[0895] A means for generating an information code as a means of accessing the generated multilingual handwritten style font,

[0896] A means of recognizing the user's emotions using an emotion engine and dynamically updating and displaying menu information according to those emotions,

[0897] A means of transmitting user order confirmation information to the relevant departments via communication means,

[0898] A system that includes this.

[0899] (Claim 2)

[0900] The system according to claim 1, comprising means for distributing the generated multilingual handwritten style font and information code within a commercial facility and making them accessible to users using information terminals.

[0901] (Claim 3)

[0902] The system according to claim 1, comprising means for instantly transmitting order information electronically by linking an information provision device using a generated handwritten style font with an electronic ordering system. [Explanation of Symbols]

[0903] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of converting handwritten original language characters into digital data using optical character recognition technology, A method using generative technology that extracts and learns handwriting styles from digital data, A means of translating text from one language to several other languages ​​and applying a learned handwriting style to the translated text, A means for outputting the generated multilingual handwritten style font as an information providing device, A means of generating a two-dimensional code as a means of accessing the generated multilingual handwritten style font, A system that includes this.

2. The system according to claim 1, comprising means for distributing the generated multilingual handwritten style font and two-dimensional code within the store and making them accessible to customers using information terminals.

3. The system according to claim 1, comprising means for instantly transmitting order information electronically by linking an information provision device using a generated handwritten style font with a digital ordering system.

Images