System

Abstract

A system is provided.SOLUTION: This system is provided with a means for inputting a character to be written, a means for analyzing inputted character information and retrieving model data, a means for presenting the model data, a means for fetching a work created by a user as an image, a means for analyzing the fetched image and performing scoring and correction, and a means for presenting the scoring result and the correction contents.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional calligraphy education has had the problem that those who want to learn calligraphy lose the opportunity to do so because there are no classrooms nearby. Furthermore, the number of calligraphy instructors is decreasing due to the aging of calligraphy teachers, making it difficult to pass on high-quality calligraphy education to the next generation. Furthermore, for students living in remote areas and busy modern people, there is a need to provide a calligraphy learning environment that is not restricted by time or place. The goal is to solve these problems and spread the traditional Japanese culture of calligraphy both domestically and internationally. [Means for solving the problem]

[0005] The present invention provides a system including a means for inputting characters to be written, a means for analyzing the input character information and searching for model data, a means for presenting the model data, a means for capturing a work created by the user as an image, a means for analyzing the captured image, scoring and correcting the work, and a means for presenting the scoring results and corrections. The system also includes a means for notifying the user of the conditions for obtaining a rank or grade, a means for evaluating works that meet the conditions, and a means for determining the rank or grade. The system also includes a means for selecting an instructor who requests remote instruction, a means for notifying the selected instructor and arranging a lesson schedule, a means for providing instruction via video call, and a means for recording the content of the lesson. This system allows users to receive high-quality calligraphy education from the comfort of their own home by referring to models, practicing repeatedly, and receiving AI scoring and corrections. Furthermore, a more practical learning environment is provided through the acquisition of ranks and grades and remote instruction, which is expected to improve calligraphy skills.

[0006] "Means for inputting characters to be written" refers to an interface device that allows a user to input characters to be written for learning purposes. Examples include keyboards on tablets and smartphones, and handwriting recognition functions.

[0007] "Means for searching model data" refers to an algorithm or process that searches for corresponding calligraphy model data stored in advance based on the character information entered by the user.

[0008] "Means for presenting exemplar data" refers to a device or interface that visually displays retrieved exemplar data to a user. For example, it includes the ability to display exemplars on a display or screen.

[0009] "Means for capturing images of works created by users" refers to devices or functions for capturing image data of calligraphy works created by users as a result of their practice. Examples include cameras, scanners, and the camera function of smartphones.

[0010] "Means for analyzing the captured image" refers to software or algorithms that analyze the image data of the captured calligraphy work and evaluate it based on the evaluation criteria.

[0011] "Means for scoring and correcting" refers to a process or function for scoring a user's calligraphy work based on the analysis results and pointing out areas for improvement or points to note as necessary.

[0012] "Means for presenting the grading results and corrections" refers to an interface for displaying or notifying the user of the grading results and corrections, including, for example, the display of text or graphics on a display.

[0013] "Means for notifying users of the conditions for obtaining a rank or kyu" refers to a device or function for notifying users of the conditions for obtaining a rank or kyu that they must meet. Examples include in-app notification functions and email notifications.

[0014] "Means for evaluating works that meet the qualification requirements" refers to the algorithm or process used to evaluate works submitted by users to obtain ranks or grades.

[0015] "Means for determining rank and grade" refers to the process or system that determines a user's rank and grade based on the evaluation results.

[0016] "Means for selecting an instructor from whom you would like to receive remote instruction" refers to an interface or function that allows a user to select a calligraphy instructor from whom you would like to receive remote instruction. For example, this includes a function to select from a list of instructors within the app.

[0017] "Means for notifying the selected instructor and arranging the schedule for instruction" refers to a system or function for notifying the instructor selected by the user and arranging the schedule for instruction.

[0018] "Means for providing instruction via video calls" refers to functions and devices that enable a calligraphy instructor to provide remote instruction to a user via video calls. Examples include video call applications and platforms.

[0019] "Means for recording instruction content" refers to a system or function for recording and saving the instruction content that has been provided. For example, this includes a function for recording video calls and text recordings. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The present invention relates to a calligraphy learning support system that enables users who wish to learn calligraphy to receive high-quality calligraphy instruction and enables remote calligraphy instruction using artificial intelligence (AI).

[0042] System configuration

[0043] This system consists of the following main elements:

[0044] 1. User device: Tablet, smartphone, or PC. Users use these devices to study calligraphy.

[0045] 2. Server: The core part of the system, installed in the center, provides a database of calligraphy samples, image analysis using AI, and scoring and correction functions.

[0046] 3. Interface: The software or application that allows the user to interact with the system.

[0047] Program processing

[0048] Modeling process

[0049] 1. The user inputs the characters they want to write on the terminal. For example, input the character "美" into the input field of the application.

[0050] 2. The terminal sends the input character information to the server.

[0051] 3. The server receives the character information, analyzes it, and searches for the corresponding model data from the database.

[0052] 4. The server sends the search result to the terminal.

[0053] 5. The terminal displays the sent model data and presents it to the user. Thus, the user can view the model while at home.

[0054] Practice and Work Submission Process

[0055] 1. The user practices calligraphy based on the displayed model. For example, use paper and pen to write the character "美".

[0056] 2. Once the practice work is completed, the user takes a photo of the work as an image and imports it into the terminal. The camera function of a smartphone is often used.

[0057] 3. The terminal sends the captured image data to the server.

[0058] 4. The server receives the image and analyzes it using an AI algorithm. It grades based on multiple criteria such as the movement of the pen and the balance of the characters.

[0059] 5. The server generates the grading result and the content for revision and sends it to the terminal.

[0060] 6. The terminal displays the received result to the user. Thus, the user can receive feedback and use it for further practice.

[0061] Optional Function Process

[0062] 1. Obtaining Dan and Kyu ranks:

[0063] The user sends multiple practice works to the server and receives a certain evaluation.

[0064] The server accumulates the user's practice data and sends a notification when the conditions for obtaining Dan and Kyu ranks are met.

[0065] The user sends a work that meets the conditions, and the server conducts a final evaluation to determine the Dan and Kyu ranks.

[0066] 2. Remote guidance:

[0067] The user selects a calligraphy instructor who wishes to provide remote guidance within the app.

[0068] The server sends a notification to the selected instructor and adjusts the guidance schedule.

[0069] The user and the calligraphy instructor conduct remote guidance through a video call.

[0070] The server records the guidance content so that the user can review it later.

[0071] Specific example

[0072] For example, when the user wants to practice the character "美" (beautiful), first enter "美" into the application. The terminal sends this information to the server, and the server retrieves the corresponding model data from the database and sends it to the terminal. The user practices while looking at the presented model, takes a photo of the completed work with a smartphone, and uploads it to the terminal. The server analyzes the image, conducts an evaluation and correction, and sends the results to the user, who can use them for the next practice.

[0073] Users aiming to acquire ranks or grades can receive notifications from the server, submit works that meet the requirements, and finally receive their rank or grade after being evaluated. Furthermore, if users require external instruction, they can select their preferred instructor within the app and receive instruction remotely. This process makes it possible to receive high-quality calligraphy education from the comfort of their own home.

[0074] The processing flow will be explained below.

[0075] Modeling process

[0076] Step 1:

[0077] The user inputs the characters they want to write into the terminal.

[0078] Specifically, the user opens the application and enters the character they want to write, such as "beauty," in the text input field.

[0079] Step 2:

[0080] The terminal transmits the input character information to the server.

[0081] Specifically, the terminal issues a request to send the character data entered by the user to the server via the API.

[0082] Step 3:

[0083] The server receives the text information, analyzes it, and searches the database for corresponding example data.

[0084] Specifically, the server receives the request and uses a character recognition module to query the database and retrieve the corresponding example data.

[0085] Step 4:

[0086] The server sends the search results to the terminal.

[0087] Specifically, the server transmits the acquired sample data to the terminal as a response.

[0088] Step 5:

[0089] The terminal displays the transmitted example data and presents it to the user.

[0090] Specifically, the terminal receives the response and displays the example image within the application.

[0091] Practice and submission process

[0092] Step 1:

[0093] The user practices calligraphy based on the displayed example.

[0094] Specifically, the user uses paper and a brush to write characters according to the provided example.

[0095] Step 2:

[0096] Once the practice piece is complete, the user takes a picture of it and imports it into their device.

[0097] Specifically, users take a photo of the work using their smartphone camera and upload the image to the app.

[0098] Step 3:

[0099] The terminal transmits the captured image data to the server.

[0100] Specifically, the terminal issues a request to upload image data to the server via the API.

[0101] Step 4:

[0102] The server receives the images and analyzes them using AI algorithms.

[0103] Specifically, the server passes the image data to an analysis engine, which evaluates it based on multiple criteria, such as brush strokes and character balance.

[0104] Step 5:

[0105] The server generates the scoring results and corrections and sends them to the terminal.

[0106] Specifically, the server calculates a score based on the analysis results, generates feedback including points for improvement, and sends it to the device.

[0107] Step 6:

[0108] The terminal displays the received results to the user.

[0109] Specifically, the device receives a response from the server and displays the scoring results and corrections within the app.

[0110] Optional Feature Process

[0111] Obtaining Dan / Kyu rank

[0112] Step 1:

[0113] A user submits multiple practice pieces to the server and receives a certain evaluation.

[0114] Specifically, users continue to upload their work to the app and accumulate ratings.

[0115] Step 2:

[0116] The server accumulates the user's practice data and notifies them when they meet the requirements for obtaining a rank or grade.

[0117] Specifically, the server checks the user's evaluation log and determines whether the acquisition conditions are met. If the conditions are met, the server notifies the user.

[0118] Step 3:

[0119] Users submit works that meet the requirements, and the server performs a final evaluation to determine rank and grade.

[0120] Specifically, users submit their work to be evaluated through a specified procedure, and the server performs a final evaluation to determine the rank and grade.

[0121] Remote instruction

[0122] Step 1:

[0123] The user selects the calligraphy instructor from whom they would like to receive remote instruction within the app.

[0124] Specifically, the user selects the desired instructor from a list of instructors in the app.

[0125] Step 2:

[0126] The server notifies the selected instructor and schedules the instruction.

[0127] Specifically, the server sends requests to instructors and provides an interface for scheduling.

[0128] Step 3:

[0129] The user and the calligraphy instructor will conduct remote instruction via video call.

[0130] Specifically, the user and instructor will provide instruction via a video call app at a specified time.

[0131] Step 4:

[0132] The server records the instruction content so that the user can review it later.

[0133] Specifically, the server stores recorded data of video calls and makes it available for users to review if necessary.

[0134] Example 1

[0135] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0136] Conventional calligraphy education systems require face-to-face instruction and practice using paper and brush in order for users to receive high-quality calligraphy training, which can be difficult for users living in remote locations. There are also issues with the lack of prompt feedback on the work users have practiced and evaluation of their ranks and grades. Furthermore, there is a lack of a flexible system that allows for real-time remote instruction. To address these issues, an effective and efficient calligraphy learning support system is needed.

[0137] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0138] In this invention, the server includes means for a user to input characters that the user wants to write, means for analyzing the input character information and searching for model data, means for presenting the searched model data, means for capturing an image of a work created by the user, means for analyzing the captured image using an artificial intelligence algorithm and grading and correcting the image, means for presenting the grading results and corrections to the user, means for notifying the user of the conditions for obtaining a rank or grade, means for evaluating works that satisfy the conditions, means for determining the rank or grade, means for the user to select an instructor from whom they wish to receive remote instruction, means for notifying the selected instructor and arranging a schedule for instruction, means for providing instruction via video call, and means for recording the content of instruction. This enables users to receive high-quality calligraphy education from the comfort of their own homes, receive prompt feedback and evaluation, and receive real-time remote instruction.

[0139] "User" refers to a user of the system who intends to receive calligraphy education.

[0140] A "terminal" is a device that allows a user to use the system, and includes a tablet, smartphone, or PC.

[0141] The "server" is the central computer of the system, and is a device that provides functions for searching sample data, analyzing images, and scoring and correcting.

[0142] "Model data" is calligraphy sample data that is presented as a standard for the characters that the user wants to learn.

[0143] "Means for capturing images" refers to devices and functions necessary for inputting a user's work into the system as a digital image, and includes camera functions, etc.

[0144] An "artificial intelligence algorithm" is a program that runs on a server and analyzes, grades, and corrects works created by users.

[0145] "Scoring result" is the evaluation score given to the user's work by the artificial intelligence algorithm.

[0146] "Correction content" is feedback information indicating corrections and improvements provided to the user's work.

[0147] "Rank / Grade" refers to the qualification grade that indicates the level of the user's calligraphy skills.

[0148] "Acquisition conditions" are the requirements that a user must meet in order to acquire a rank or grade.

[0149] The "instructor who wishes to receive remote instruction" is a calligraphy instructor that the user selects within the system to receive calligraphy instruction via video call.

[0150] "Video call" refers to a communication means that enables real-time exchange of video and audio over the Internet.

[0151] "Means for recording guidance content" refers to a function that saves the guidance during a video call in digital format.

[0152] This invention is a calligraphy learning support system for users to receive high-quality calligraphy education. The system mainly consists of a user terminal, a server, and a user interface.

[0153] System Configuration

[0154] 1. User Terminal: Devices such as tablets, smartphones, and personal computers, which users use to conduct calligraphy learning.

[0155] 2. Server: The core device of the system, which provides a database of calligraphy models, image analysis by AI, and grading and correction functions.

[0156] 3. User Interface: Software or applications for users and the system to interact with each other.

[0157] Model Presentation

[0158] 1. The user enters the character they want to write in the input field of the application. For example, enter the character "beautiful".

[0159] 2. The terminal sends the entered character information to the server. Specifically, it uses an HTTP request to send the character information to the server.

[0160] 3. The server receives the character information, analyzes it, and searches for the corresponding model data from the database. The server executes an analysis script written in Python and uses MySQL (registered trademark) for the database.

[0161] 4. The server sends the search results to the terminal, and the terminal displays the received example data on the application.

[0162] Practice and submission of work

[0163] The user practices calligraphy using a model. Using paper and a brush, for example, they write the character "beauty." Once the practice piece is complete, the user takes a photo of the piece using the smartphone's camera function and imports it into the device as an image. The device then sends the captured image data to the server using an HTTP POST request.

[0164] The server receives the image data and analyzes it using an AI algorithm using TENSORFLOW (registered trademark). It evaluates factors such as brush strokes and character balance, and generates a score and corrections. The generated data is in JSON format, and the server sends it to the device. The device then displays the results received on the application. The score and corrections are visually displayed using HTML and CSS.

[0165] Obtaining Dan / Kyu rank

[0166] The user sends multiple practice pieces to the server and receives evaluation. The server stores the user's practice data in a database and notifies the user when the conditions for obtaining a rank or grade are met. The user then sends another piece that meets the conditions for final evaluation, and the server determines the final rank or grade and sends the evaluation results to the device.

[0167] Remote instruction

[0168] When a user requests remote instruction within the application, they select an instructor using the system's interface. The server then notifies the selected instructor and schedules the instruction. WebRTC technology is used for video calls, enabling real-time instruction. The instruction content is stored in cloud storage and recorded using Amazon S3.

[0169] Specific examples

[0170] For example, if a user wants to practice writing the character "美", the user inputs "美" into the application. The terminal sends this information to the server, and the server retrieves the corresponding model data from the database and sends it to the terminal. The user practices while looking at the presented model, takes a photo of the completed work with a smartphone, and uploads it to the terminal. The server analyzes the image, conducts evaluation and correction, and sends the results to the user, which can be used for the next practice. Also, when obtaining a rank or grade, the user can submit multiple works for evaluation, receive a notification when the criteria are met, and obtain a rank or grade.

[0171] Examples of prompt sentences

[0172] Please display a model for writing the character "美".

[0173] Search for and display the model of the input character "美" from the database. Also, please take a photo of the user's practice work and evaluate it.

[0174] The flow of the specific process in Example 1 will be described with reference to FIG. 11.

[0175] Step 1:

[0176] The user inputs the character to be written into the input field of the application. The data to be input is a Chinese character such as "美". The user inputs a character into the input field and performs the operation of pressing the send button.

[0177] Step 2: [[ID=三十二]]

[0178] The terminal sends the input character information to the server. The input is the character information input by the user, and the output is an HTTP request to the server. The data is sent in JSON format. At this time, the terminal performs the operation of sending a request to the server using network communication.

[0179] Step 3:

[0180] The server receives character information and analyzes the characters. The input is the character information sent from the terminal, and the output is the character code as the analysis result. The server executes a Python script and performs the operation of analyzing the specified characters.

[0181] Step 4:

[0182] The server searches the database for corresponding example data based on the character code. The input is the parsed character code, and the output is the example data. The server executes an SQL query against MySQL to retrieve the relevant example data.

[0183] Step 5:

[0184] The server sends the search results to the terminal in JSON format. The input is the sample data retrieved from the database, and the output is an HTTP response to the terminal. The server converts the obtained data into JSON format and sends it as an HTTP response.

[0185] Step 6:

[0186] The terminal displays the sample data received on the application. The input is the sample data sent from the server, and the output is the sample displayed on the user interface. The terminal uses JavaScript (registered trademark) to draw the sample data.

[0187] Step 7:

[0188] The user practices calligraphy based on the displayed example. Using paper and a brush, the user writes the character "beauty."

[0189] Step 8:

[0190] Once the practice piece is complete, the user takes a photo of the piece using the smartphone's camera function and imports the image into the device. The input is the image of the completed piece, and the output is a digital image saved on the device. The user uses the camera function to acquire the image.

[0191] Step 9:

[0192] The device sends the captured image data to the server using an HTTP POST request. The input is the image data captured by the user, and the output is a request to the server. The device uses network communication to send the image data to the server.

[0193] Step 10:

[0194] The server receives the image data and analyzes it using an AI algorithm. The input is the transmitted image data, and the output is the evaluation data as the analysis result. The server uses TensorFlow to analyze the image and evaluate the brush strokes and balance of the characters.

[0195] Step 11:

[0196] The server generates the scoring results and corrections and sends them to the device. The input is the analysis result, and the output is the evaluation results and corrections sent to the device. The server converts the evaluation data into JSON format and sends it as an HTTP response.

[0197] Step 12:

[0198] The terminal displays the results it receives on the application. The input is the evaluation results and corrections sent from the server, and the output is the feedback displayed on the user interface. The terminal uses HTML and CSS to visually display the results.

[0199] Step 13:

[0200] The user performs further practice based on the provided feedback. The input is the displayed feedback, and the output is an improved practice piece. The user then uses the feedback to perform the writing action again.

[0201] (Application example 1)

[0202] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0203] Conventional calligraphy learning systems provide paper-based examples and feedback, making real-time instruction and evaluation difficult. Furthermore, due to limited learning support in remote locations, students must attend a calligraphy class in order to receive high-quality calligraphy education. Furthermore, there are currently insufficient methods for improving the efficiency of calligraphy learning using smart devices.

[0204] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0205] In this invention, the server includes means for inputting characters to be written, means for analyzing the input character information and searching for model data, means for presenting the model data, means for capturing a work created by the user as an image, means for analyzing the captured image and performing scoring and corrections, means for presenting the scoring results and corrections, means for displaying the model data using smart glasses, means for photographing the created work using the camera function of the smart glasses, means for transmitting the image data to a cloud server and analyzing it using an AI algorithm, and means for displaying the analysis results and feedback on the smart glasses. This enables real-time feedback, enables high-quality calligraphy education even from remote locations, and realizes efficient calligraphy learning by utilizing smart devices.

[0206] The "means for inputting characters to be written" is an interface for inputting characters to be written into a device when a user is learning calligraphy.

[0207] The "means for analyzing input character information and searching for model data" is a function that analyzes character information input by the user and searches for corresponding calligraphy model data from a database.

[0208] The "means for presenting example data" is a function for displaying the retrieved example data on the display of the terminal so that the user can confirm it.

[0209] The "means for capturing a work created by a user as an image" is a function for capturing a calligraphy work that a user has written on paper or the like as a digital image using a camera or the like.

[0210] The "means for analyzing the imported image and scoring and correcting it" refers to an AI algorithm that analyzes the imported image of the calligraphy work and automatically scores and corrects it.

[0211] The "means for presenting the scoring results and corrections" is a function for displaying the analysis results and corrections in an easy-to-understand manner to the user.

[0212] "Means for displaying example data using smart glasses" refers to a function that uses the display of the smart glasses to virtually display example data within the user's field of view.

[0213] "Means for photographing a work created using the camera function of the smart glasses" refers to a function for photographing a work written by a user using the camera built into the smart glasses.

[0214] "Means for sending image data to a cloud server and analyzing it using an AI algorithm" refers to a function that uploads image data of calligraphy works sent from smart glasses or other devices to a server on the cloud and uses an AI algorithm to analyze the data.

[0215] "Means for displaying analysis results and feedback on the smart glasses" is a function for displaying the analysis results and feedback returned from the cloud server on the display of the smart glasses.

[0216] This invention is a system that utilizes smart glasses to provide real-time calligraphy learning support. This system allows users who want to learn calligraphy to receive high-quality calligraphy education using smart glasses and a cloud server.

[0217] System Configuration

[0218] 1. User Device

[0219] Smart glasses (e.g., Google® Glass®) are used as the user terminal. This device has the function of displaying the example data, taking a photo of the written characters, and sending the photo to a cloud server.

[0220] 2. Cloud Server

[0221] The cloud server provides a database of calligraphy samples, AI image analysis, and scoring and correction functions, and serves as the infrastructure for processing and analyzing data in real time.

[0222] 3. Interface

[0223] An interface is provided for users to interact with the system through the smart glasses' display, operation panel, and voice recognition function.

[0224] Program Implementation Overview

[0225] Modeling process

[0226] The server provides a means for the user to input the characters they want to write through voice or touchpad operations via smart glasses. For example, when the user inputs the character "美", the server analyzes the character information and searches for the corresponding model data from the database. The model data is displayed on the smart glasses, and the user can practice calligraphy while checking in real time.

[0227] Practice and Work Submission Process

[0228] The user practices calligraphy while looking at the model displayed through the smart glasses. After practice, the user uses the camera function of the smart glasses to take a picture of the work and send the image data to the cloud server. The cloud server analyzes the image with an AI algorithm and evaluates aspects such as the stroke movement and character balance. The analysis results are displayed on the smart glasses together with the calligraphy correction content, and the user can obtain real-time feedback.

[0229] Hardware and Software Used

[0230] This system operates with smart glasses, a cloud server, and an AI analysis model (using, for example, TensorFlow) as the main components. The smart glasses function as a user interface, and the cloud server provides data processing and storage solutions. An image analysis model is used for AI analysis to conduct a detailed evaluation of the user's calligraphy works.

[0231] Specific Examples and Example Prompt Texts

[0232] For example, when the user wants to practice the character "美", first, the user inputs the command "Display Model" through voice or touchpad operations on the smart glasses. The following are example prompt texts.

[0233] Prompt Text: "Please input the model character."

[0234] When the user finishes practicing calligraphy and the character is completed, the camera is then activated using the gesture function of the smart glasses to take a photo of the work. The following is an example of a prompt sentence.

[0235] Prompt sentence: "Please activate the camera and take a photo when the character is completed."

[0236] The captured image data is sent to the cloud server, the AI model performs analysis, and the results and feedback are displayed on the display of the smart glasses. When the analysis is completed, the following prompt sentence is displayed.

[0237] Prompt sentence: "The analysis is complete. Please check the feedback."

[0238] In this way, a system is provided that allows the user to practice calligraphy and receive feedback in real time.

[0239] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0240] Step 1:

[0241] The user wears the smart glasses and inputs the character to be written. The input method is voice recognition or touchpad operation. The user inputs "beautiful" in response to the prompt "Please input the model character". The input character information is sent by the smart glasses to the cloud server. The server analyzes the received character information and searches for the corresponding model data from the model database.

[0242] Step 2:

[0243] The server sends the model data to the smart glasses, and the smart glasses display the data in the user's field of vision. As a result, the user can start practicing calligraphy while checking the model in real time. The display of the model data is projected onto the display of the smart glasses, and the user writes characters on the paper while looking at the model.

[0244] Step 3:

[0245] Once the user has finished writing the character using the model, they can use the smart glasses' camera to take a photo of their creation. They can then use gestures or the touchpad to follow the prompt, "When you've finished writing the character, launch the camera and take a photo." The captured image data is then sent to a cloud server by the smart glasses.

[0246] Step 4:

[0247] The server analyzes the received image data using an AI algorithm. Data processing involves pre-processing the image and extracting features, and analyzing the brush strokes, balance of the characters, line thickness, and other evaluation criteria. A generative AI model such as TensorFlow is used for this analysis. The analysis results in a score and detailed corrections.

[0248] Step 5:

[0249] The server sends the generated analysis results and feedback to the smart glasses, which then display the received results in the user's field of view. Following the prompt "Analysis completed. Please check your feedback," feedback is presented to the user, allowing the user to use this information to improve their practice next time.

[0250] Step 6:

[0251] When a user practices multiple times and meets certain conditions, the conditions for obtaining a rank or grade are notified to the smart glasses. The server accumulates the user's practice data and notifies the user when the conditions are met. The user then photographs and submits works that meet the conditions, and the server performs a final evaluation to determine the rank or grade.

[0252] Step 7:

[0253] When a user requests remote instruction, they select an instructor through the smart glasses interface. The server then notifies the instructor and schedules the instruction. The instructor provides real-time instruction to the user through a video call function. The instruction is then recorded on the server for later review by the user.

[0254] Through this series of steps, the calligraphy learning support system, which utilizes smart glasses and a cloud server, provides users with an efficient and high-quality learning environment.

[0255] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0256] The present invention relates to a calligraphy learning support system. This system allows users who want to learn calligraphy to receive high-quality calligraphy education, and enables remote calligraphy instruction using artificial intelligence (AI). In addition, by combining it with an emotion engine that analyzes the user's emotions, the user's learning experience can be further improved.

[0257] System configuration

[0258] This system consists of the following main elements:

[0259] 1. User device: Tablet, smartphone, or PC. Users use these devices to study calligraphy.

[0260] 2. Server: The core part of the system, installed in the center, provides a database of calligraphy samples, image analysis using AI, and scoring and correction functions.

[0261] 3. Interface: The software or application that allows the user to interact with the system.

[0262] 4. Emotional Engine: A device or software that analyzes the user's facial expressions and vocal tones to recognize their emotional state.

[0263] Program processing

[0264] Model presentation process

[0265] 1. The user inputs the characters they want to write on the terminal. For example, the user inputs the character "美" into the input field of the application.

[0266] 2. The terminal sends the input character information to the server.

[0267] 3. The server receives the character information, analyzes it, and searches for the corresponding model data from the database.

[0268] 4. The server sends the search results to the terminal.

[0269] 5. The terminal displays and presents the sent model data to the user. In this way, the user can view the model while at home.

[0270] Practice and work submission process

[0271] 1. The user practices calligraphy based on the displayed model. For example, using paper and pen, the user writes the character "美".

[0272] 2. When the practice work is completed, the user takes a photo of the work as an image and imports it into the terminal. Usually, the camera function of a smartphone is used.

[0273] 3. The terminal sends the captured image data to the server.

[0274] 4. The server receives the image, analyzes it using an AI algorithm, and grades it based on multiple criteria such as the stroke movement and the balance of the characters.

[0275] 5. The server generates the scoring results and corrections and sends them to the terminal.

[0276] 6. The device displays the received results to the user, allowing the user to receive feedback and use it for further practice.

[0277] Emotion Recognition Process

[0278] 1. As the user practices calligraphy, the emotion engine analyzes the user's facial expressions and voice to recognize their emotional state in real time.

[0279] 2. The emotion engine provides feedback to the server based on the emotional state.

[0280] 3. The server receives information from the emotion engine, checks the progress of practice and motivation, and generates appropriate advice and encouraging messages.

[0281] 4. The device displays these messages to the user to help them stay motivated.

[0282] Optional Feature Process

[0283] Obtaining Dan / Kyu rank

[0284] 1. A user submits multiple practice pieces to the server and receives a certain evaluation.

[0285] 2. The server accumulates the user's practice data and notifies the user when the conditions for obtaining a rank or grade are met, allowing the user to check their own progress.

[0286] 3. The user submits works that meet the requirements, and the server performs a final evaluation to determine the rank and grade.

[0287] Remote instruction

[0288] 1. The user selects a calligraphy instructor who the user hopes to receive remote guidance within the app.

[0289] 2. The server sends a notification to the selected instructor and adjusts the guidance schedule.

[0290] 3. The user and the calligraphy instructor conduct remote guidance through a video call.

[0291] 4. The server records the guidance content so that the user can review it later.

[0292] 5. The emotion engine analyzes the emotional state of the user during the guidance and notifies the instructor of the result in real time. As a result, the instructor can provide guidance according to the emotional state of the user.

[0293] Specific Example

[0294] For example, when the user wants to practice the character "美" (beautiful), first enter "美" into the application. The terminal sends this information to the server, and the server retrieves the corresponding model data from the database and sends it to the terminal. The user practices while looking at the presented model, takes a photo of the completed work with a smartphone, and uploads it to the terminal. The server analyzes the image, sends the evaluation and correction results to the user, and the user can use them for the next practice.

[0295] Furthermore, during the user's practice, the emotion engine analyzes the user's expression and feeds back to the server the state of being concentrated or tired. Based on this, the server can send messages to the user such as "You seem to be concentrating. Keep it up like this" or "It might be good to take a short break."

[0296] Users aiming to obtain a rank or grade receive notifications from the server, submit work that meets the requirements, and are ultimately evaluated to obtain the rank or grade. Furthermore, if external instruction is required, users can select their preferred instructor within the app and receive remote instruction. This process allows users to receive high-quality calligraphy instruction from the comfort of their own home. The emotional engine enables more effective learning while maintaining users' motivation and concentration.

[0297] The processing flow will be explained below.

[0298] Modeling process

[0299] Step 1:

[0300] The user inputs the characters they want to write into the terminal.

[0301] Specifically, the user opens the application and enters the character they want to write, such as "beauty," in the text input field.

[0302] Step 2:

[0303] The terminal transmits the input character information to the server.

[0304] Specifically, the terminal issues a request to send the character data entered by the user to the server via the API.

[0305] Step 3:

[0306] The server receives the text information, analyzes it, and searches the database for corresponding example data.

[0307] Specifically, the server receives the request and uses a character recognition module to query the database and retrieve the corresponding example data.

[0308] Step 4:

[0309] The server sends the search results to the terminal.

[0310] Specifically, the server transmits the acquired sample data to the terminal as a response.

[0311] Step 5:

[0312] The terminal displays the transmitted example data and presents it to the user.

[0313] Specifically, the terminal receives the response and displays the example image within the application.

[0314] Practice and submission process

[0315] Step 1:

[0316] The user practices calligraphy based on the displayed example.

[0317] Specifically, the user uses paper and a brush to write characters according to the provided example.

[0318] Step 2:

[0319] Once the practice piece is complete, the user takes a picture of it and imports it into their device.

[0320] Specifically, users take a photo of the work using their smartphone camera and upload the image to the app.

[0321] Step 3:

[0322] The terminal transmits the captured image data to the server.

[0323] Specifically, the terminal issues a request to upload image data to the server via the API.

[0324] Step 4:

[0325] The server receives the images and analyzes them using AI algorithms.

[0326] Specifically, the server passes the image data to an analysis engine, which evaluates it based on multiple criteria, such as brush strokes and character balance.

[0327] Step 5:

[0328] The server generates the scoring results and corrections and sends them to the terminal.

[0329] Specifically, the server calculates a score based on the analysis results, generates feedback including points for improvement, and sends it to the device.

[0330] Step 6:

[0331] The terminal displays the received results to the user.

[0332] Specifically, the device receives a response from the server and displays the scoring results and corrections within the app.

[0333] Emotion Recognition Process

[0334] Step 1:

[0335] As the user practices calligraphy, the emotion engine analyzes the user's facial expressions and voice to recognize their emotional state in real time.

[0336] Specifically, the emotion engine collects and analyzes the user's facial expressions and tone of voice through the device's camera and microphone.

[0337] Step 2:

[0338] An emotional engine provides feedback to the server based on the emotional state.

[0339] Specifically, the emotion engine analyzes the user's current emotional state (e.g., focused, tired) and sends that data to the server.

[0340] Step 3:

[0341] The server receives information from the emotion engine, checks the progress of practice and motivation, and generates appropriate advice and encouraging messages.

[0342] Specifically, the server generates messages based on emotional data, such as "You seem to be concentrating, keep it up" or "Maybe it would be good to take a short break."

[0343] Step 4:

[0344] The device displays these messages to the user to help them stay motivated.

[0345] Specifically, the terminal displays the message sent from the server on the screen.

[0346] Optional Feature Process

[0347] Obtaining Dan / Kyu rank

[0348] Step 1:

[0349] A user submits multiple practice pieces to the server and receives a certain evaluation.

[0350] Specifically, users continue to upload their work to the app and accumulate ratings.

[0351] Step 2:

[0352] The server accumulates the user's practice data and notifies them when they meet the requirements for obtaining a rank or grade.

[0353] Specifically, the server checks the user's evaluation log and determines whether the acquisition conditions are met. If the conditions are met, the server notifies the user.

[0354] Step 3:

[0355] Users submit works that meet the requirements, and the server performs a final evaluation to determine rank and grade.

[0356] Specifically, users submit their work to be evaluated through a specified procedure, and the server performs a final evaluation to determine the rank and grade.

[0357] Remote instruction

[0358] Step 1:

[0359] The user selects the calligraphy instructor from whom they would like to receive remote instruction within the app.

[0360] Specifically, the user selects the desired instructor from a list of instructors in the app.

[0361] Step 2:

[0362] The server notifies the selected instructor and schedules the instruction.

[0363] Specifically, the server sends requests to instructors and provides an interface for scheduling.

[0364] Step 3:

[0365] The user and the calligraphy instructor will conduct remote instruction via video call.

[0366] Specifically, the user and instructor will provide instruction via a video call app at a specified time.

[0367] Step 4:

[0368] The server records the guidance content so that the user can recheck it later.

[0369] Specifically, the server saves the recording data of the video call and provides it so that the user can recheck it as needed.

[0370] Step 5:

[0371] The emotion engine analyzes the emotional state of the user during guidance and notifies the instructor of the result in real time.

[0372] Specifically, the emotion engine sends the analysis results of the user's expression and voice to the server, and the server displays the results on the instructor's screen.

[0373] Specific example

[0374] For example, when the user wants to practice the character "美", first enter "美" into the application. The terminal sends this information to the server, and the server retrieves the corresponding model data from the database and sends it to the terminal. The user practices while looking at the presented model, takes a photo of the completed work with the smartphone, and uploads it to the terminal. The server analyzes the image, sends the evaluation and correction results to the user, and the user can use them for the next practice.

[0375] Furthermore, during the user's practice, the emotion engine analyzes the user's expression and feedbacks the concentrated state or tired state to the server. Based on this, the server can send messages to the user such as "You seem to be concentrating. Keep it up like this" or "It might be good to take a little break".

[0376] Users aiming to obtain a rank or grade receive notifications from the server, submit work that meets the requirements, and are ultimately evaluated to obtain the rank or grade. Furthermore, if external instruction is required, users can select their preferred instructor within the app and receive remote instruction. This process allows users to receive high-quality calligraphy instruction from the comfort of their own home. The emotional engine enables more effective learning while maintaining users' motivation and concentration.

[0377] Example 2

[0378] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0379] Conventional calligraphy education systems make it difficult for users to receive high-quality feedback, limiting instruction especially from remote locations. Furthermore, they lack the means to properly manage users' emotions and motivation, making efficient learning difficult. The process for users to acquire ranks and grades is also inefficient, requiring a great deal of time and effort to master advanced calligraphy techniques.

[0380] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0381] In this invention, the server includes a means for inputting characters to be written, a means for analyzing the input character information and searching for model data, a means for presenting the model data, a means for capturing the work created by the user as an image, a means for analyzing the captured image and performing scoring and corrections, a means for presenting the scoring results and corrections, a means for analyzing the user's facial expressions and voice to recognize emotions, and a means for providing feedback based on the emotion recognition results. This allows for high-quality calligraphy education even in a remote environment, and enables the user's emotions to be analyzed in real time and appropriate feedback to be provided. Furthermore, since ranks and grades can be acquired efficiently, users can more effectively acquire calligraphy techniques.

[0382] The "means for inputting characters to be written" refers to an input field or operation means for the user to input characters to be written in calligraphy practice into the application.

[0383] The "means for analyzing input character information and searching for example data" refers to an algorithm or program for receiving character information input by a user, analyzing it, and searching for corresponding example data from a database.

[0384] The "means for presenting model data" refers to a display device or software for receiving the model data sent from the server and displaying it on the user's terminal.

[0385] The "means for capturing an image of a work created by a user" refers to a camera function or an image input device that allows a user to take an image of a calligraphy work and capture it into the terminal.

[0386] The "means of analyzing the captured image and grading and correcting" refers to an AI algorithm or analysis program that analyzes the captured image, evaluates the brush strokes and balance of the characters, and performs grading and correction.

[0387] The "means for presenting the marking results and corrections" refers to a UI component or notification system for displaying the marking results and corrections generated by the server to the user.

[0388] "Means for recognizing emotions by analyzing the user's facial expressions and voice" refers to facial recognition technology and voice analysis technology that analyze the user's facial expressions and tone of voice to understand their current emotional state.

[0389] The "means for providing feedback based on emotion recognition results" refers to software or a system for generating appropriate advice or encouraging messages based on the user's emotion recognition results and providing them to the user.

[0390] "Means for notifying the conditions for obtaining a rank or grade" refers to algorithms and communication methods for confirming the conditions necessary for a user to obtain a rank or grade and notifying the user of the status of achievement.

[0391] The "means for evaluating works that satisfy the acquisition conditions" refers to an examination algorithm or evaluation system that allows users to submit works that satisfy the conditions for acquiring a rank or grade and evaluates them.

[0392] The "means for determining ranks and grades" refers to a judging system for determining final ranks and grades based on the evaluation results of works submitted by users.

[0393] The "means for selecting an instructor from whom the user wishes to receive remote instruction" refers to a selection screen or system that allows the user to select an instructor from whom the user wishes to receive remote instruction within the application.

[0394] The "means for notifying the selected instructor and arranging the schedule for instruction" refers to a scheduling system or communication means for sending notification to the selected instructor and arranging the schedule for instruction.

[0395] The "means for providing instruction via video call" refers to a video call system or software that allows a user and an instructor to provide instruction remotely via video call.

[0396] The "means for recording the instruction content" refers to a recording device or software that records the instruction content given during the video call in real time so that the user can review it later.

[0397] "Means for analyzing the user's emotions during training and notifying the instructor" refers to an analysis system or communication means for analyzing the user's facial expressions and voice during a video call and notifying the instructor of their emotional state in real time.

[0398] This invention is a system for providing high-quality calligraphy education, designed to enable users to effectively learn calligraphy even in remote environments. This system is composed of a user terminal, a server, an interface, and an emotion engine.

[0399] System configuration

[0400] 1. User Device: Calligraphy lessons are conducted using devices such as tablets, smartphones, or PCs. These devices have built-in cameras that can be used to take photos of the works created by users.

[0401] 2. Server: This is the core part of the system, providing the database of calligraphy examples, AI image analysis, and scoring and correction functions. Specifically, AI frameworks such as TensorFlow and PyTorch are used for image analysis.

[0402] 3. Interface: The application or software that allows users to interact with the system. Through this interface, users can view calligraphy examples, upload practice pieces, and receive feedback.

[0403] 4. Emotion Engine: Technology that analyzes the user's facial expressions and tone of voice to recognize their emotional state. For example, OpenCV is used for facial recognition, and Google Cloud Speech-to-Text API is used for voice analysis.

[0404] Program processing

[0405] The program of this system performs the following processing.

[0406] Modeling process

[0407] When a user inputs the characters they want to write into the device, the device sends the character information to the server. The server analyzes the character information and searches for corresponding model data in a database. The resulting model data is sent to the device and presented to the user. This series of processes allows users to check calligraphy models from the comfort of their own home.

[0408] Practice and submission process

[0409] The user practices calligraphy based on the provided model and takes a photo of the completed work with their smartphone camera. The device sends the captured image data to a server, which receives the image and analyzes it using an AI algorithm. The analysis includes details such as brush strokes and character balance. The server generates an evaluation and corrections, which are sent to the device and displayed to the user. This allows the user to receive feedback that can be used to improve their next practice.

[0410] Emotion Recognition Process

[0411] As the user practices calligraphy, the emotion engine analyzes the user's facial expressions and voice in real time. The emotion engine then feeds back the analysis results to the server, which then uses this information to check the user's practice progress and motivation and generate appropriate advice and encouraging messages. The device displays these messages to the user, enabling more effective learning while maintaining the user's motivation.

[0412] Optional Features

[0413] This system provides, as options, the acquisition of ranks and levels and a remote guidance function. Regarding the acquisition of ranks and levels, when a user transmits multiple practice works to the server and receives a certain evaluation, the server records and accumulates the data and sends a notification when the acquisition conditions for ranks and levels are met. In the remote guidance function, the user selects a desired instructor, and the server notifies the instructor to adjust the guidance schedule. The user and the instructor conduct remote guidance via a video call, and the server records the guidance content. The emotion engine analyzes the emotion of the user during guidance and provides real-time feedback to the instructor.

[0414] Specific example

[0415] For example, when a user wants to practice the character "美" (beautiful), first enter "美" into the application. The terminal transmits this information to the server, and the server retrieves the corresponding model data from the database and transmits it to the terminal. The user practices while looking at the presented model, takes a photo of the completed work with a smartphone, and uploads it to the terminal. The server analyzes the image, conducts an evaluation and correction, and transmits the results to the user, who can use them for the next practice.

[0416] Furthermore, during the user's practice, the emotion engine analyzes the user's expression and provides feedback to the server on the state of concentration or fatigue. Based on this, the server can send messages such as "You seem to be concentrating. Keep it up like this" or "It might be good to take a short break."

[0417] Users aiming to acquire ranks and levels can submit works that meet the conditions upon receiving a notification from the server and finally be evaluated to acquire ranks and levels. Furthermore, when external guidance is required, the desired instructor can be selected within the application to receive remote guidance. Through this series of processes, it becomes possible to receive high-quality calligraphy education even from home. By using the emotion engine, more effective learning is possible while maintaining the user's motivation and concentration.

[0418] Example of prompt text

[0419] Use the following prompt text to explain the detailed processing content of the system to the generative AI model.

[0420] "In the calligraphy learning support system, please explain the specific processing steps of displaying a model of the character 'beautiful' and evaluating it after the user finishes practicing."

[0421] With this prompt text, the generative AI model explains each process of the system in an easy-to-understand natural language.

[0422] The flow of the specific process in Example 2 will be described using FIG. 13.

[0423] Model presentation process

[0424] Step 1:

[0425] The user inputs the character to be written on the terminal. The user inputs a character such as 'beautiful' in the input field of the application. The input for this step is the user's input character, and the output is character information ('beautiful', etc.).

[0426] Step 2:

[0427] The terminal sends the input character information to the server. Here, the HTTP request is used to send the character information to the server. The input for this step is the character information, and the output is the request transmission to the server.

[0428] Step 3:

[0429] The server receives and analyzes the character information. The server analyzes the character information using a natural language processing (NLP) algorithm and searches for the corresponding model data from the database. The input for this step is the received character information, and the output is the search result of the model data.

[0430] Step 4:

[0431] The server sends the search results to the terminal. The server uses an HTTP response to return the sample data to the terminal. The input for this step is the sample data, and the output is the data transmission to the terminal.

[0432] Step 5:

[0433] The terminal displays the sample data and presents it to the user. The terminal displays the received sample data on the user interface (UI). This includes an image viewer and SVG rendering. The input for this step is the sample data, and the output is the display of the sample.

[0434] Practice and work submission process

[0435] Step 1:

[0436] The user practices calligraphy based on the displayed sample. The user uses paper and pen to write the character 'beautiful'. The input for this step is the sample data, and the output is the practice work.

[0437] Step 2:

[0438] The user takes a photo of the completed work with the smartphone camera. The input for this step is the practice work, and the output is the image data.

[0439] Step 3:

[0440] The terminal sends the captured image data to the server. The image data is sent to the server via an HTTP POST request. The input for this step is the image data, and the output is the request transmission to the server.

[0441] Step 4:

[0442] The server analyzes the received image. The server uses an AI algorithm (for example, a model using TensorFlow or PyTorch) to analyze the image and evaluate the brush strokes, balance of the characters, etc. The input for this step is the image data, and the output is the evaluation and corrections.

[0443] Step 5:

[0444] The server generates the evaluation and corrections and sends them to the terminal. The input of this step is the evaluation result, and the output is the data sent to the terminal.

[0445] Step 6:

[0446] The terminal displays the evaluation results to the user. The terminal displays the evaluation results and corrections on the user interface. The input of this step is the evaluation data, and the output is the display of feedback.

[0447] Emotion Recognition Process

[0448] Step 1:

[0449] As the user practices, the emotion engine analyzes the user's facial expressions and voice. The emotion engine collects data in real time using the device's built-in camera and microphone. The input for this step is the user's facial expressions and voice, and the output is emotion analysis data.

[0450] Step 2:

[0451] The emotion engine feeds back the analysis results to the server. The analysis results are sent to the server via an HTTP request. The input of this step is the emotion analysis data, and the output is a request sent to the server.

[0452] Step 3:

[0453] The server receives the feedback and generates an appropriate message for the user. Based on the emotion data, the server generates messages such as "You seem to be concentrating, keep it up" or "Maybe it's time to take a short break." The input of this step is the emotion feedback, and the output is the generated message.

[0454] Step 4:

[0455] The terminal displays the generated message to the user. The terminal displays the message as a notification popup or a detailed feedback screen. The input to this step is the generated message and the output is the display of the message.

[0456] Rank and grade acquisition process

[0457] Step 1:

[0458] The user submits a practice piece to the server. The user submits multiple practice pieces together to the server for evaluation. The input of this step is the practice piece data, and the output is a request sent to the server.

[0459] Step 2:

[0460] The server accumulates practice data and checks the conditions for obtaining a rank or grade. The server saves the user's practice data in a database and notifies them when certain criteria have been met. The input for this step is practice data, and the output is a notification that the conditions have been met.

[0461] Step 3:

[0462] The server sends a notification when the conditions for obtaining a rank or grade are met. The server notifies the user of the conditions that have been met. The input to this step is the notification of the condition being met, and the output is a notification message to the user.

[0463] Step 4:

[0464] The user submits a work that meets the criteria for final judging. The user submits additional work for final evaluation. The input of this step is the final work, and the output is a request to the server.

[0465] Step 5:

[0466] The server performs the final evaluation and determines the rank. The server evaluates the submitted works and determines the final rank. The input for this step is the final judged work, and the output is the evaluation results and the determination of the rank.

[0467] Remote teaching process

[0468] Step 1:

[0469] The user selects the instructor they want within the app. The user then selects the instructor they want to receive remote instruction from on the screen. The input for this step is the instructor's selection information, and the output is the selection result.

[0470] Step 2:

[0471] The server notifies the instructor and schedules the instruction. The server notifies the selected instructor and schedules the instruction. The input of this step is the instructor selection information, and the output is a schedule adjustment notification.

[0472] Step 3:

[0473] The user and instructor receive remote instruction via video call at a specified date and time. WebRTC is used for the video call. The input of this step is the connection information for the video call, and the output is the start of the video call.

[0474] Step 4:

[0475] The server records the video call in real time and saves the call for the user to review later. The input of this step is the video call data and the output is the recording data.

[0476] Step 5:

[0477] The emotion engine analyzes the user's facial expressions and voice and notifies the instructor of the results in real time, so that the instructor can adjust the content of instruction according to the situation. The input of this step is emotion analysis data, and the output is feedback notification to the instructor.

[0478] (Application example 2)

[0479] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0480] Conventional calligraphy learning systems have limited access to high-quality calligraphy education, particularly in the absence of remote feedback and instruction. They also lack effective support for users' learning motivation and progress. Furthermore, conventional systems struggle to provide feedback that reflects users' emotions, creating a need for methods to improve individual learning experiences.

[0481] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting characters to be written, means for analyzing the input character information and searching for model data, means for presenting the model data, means for capturing a work created by the user as an image, means for analyzing the captured image and performing scoring and corrections, means for presenting the scoring results and corrections, means for performing emotion analysis, and means for generating and presenting feedback based on the analyzed emotion information. This not only enables users to receive high-quality calligraphy education, but also enables them to study effectively from home, and the learning experience can be improved by individual feedback based on emotions.

[0482] The "means for inputting the characters you want to write" is an interface that allows the user to input the characters they want to learn or use as a model.

[0483] The "means for analyzing input character information and searching for model data" is a function that analyzes character information input by the user and searches for and retrieves the corresponding calligraphy model data from a database.

[0484] The "means for presenting model data" is a function for visually presenting the retrieved calligraphy model data to the user.

[0485] "Means for capturing a work created by a user as an image" refers to a function for capturing a work written by a user as a digital image using a device such as a camera or scanner.

[0486] The "means for analyzing imported images and grading and correcting" is a function that analyzes the imported image of the user's calligraphy work and grades and corrects it based on evaluation criteria such as brush strokes and balance.

[0487] The "means for presenting the scoring results and corrections" is a function for visually presenting the scoring results and corrections based on the analysis results to the user.

[0488] "Means for emotion analysis" refers to technology for identifying a user's emotional state by analyzing their facial expressions and tone of voice.

[0489] The "means for generating and presenting feedback based on the analyzed emotional information" is a function that generates a feedback message to improve the user's learning experience based on the results of the emotional analysis and presents it to the user.

[0490] "Means for notifying the conditions for obtaining a rank or grade" is a function for notifying the user of the conditions necessary to obtain a rank or grade.

[0491] "Means for evaluating works that meet the acquisition conditions" is a function for evaluating whether or not the conditions for obtaining a rank or grade are met.

[0492] The "means for determining rank and grade" is a function for determining the final rank and grade based on the user's practice work and evaluation results.

[0493] The "means for selecting an instructor from whom the user wishes to receive remote instruction" is an interface for the user to select an instructor from whom the user wishes to receive remote calligraphy instruction.

[0494] The "means for notifying the selected instructor and adjusting the schedule for instruction" is a function for sending a notification to the selected instructor and adjusting the schedule for instruction to suit the convenience of the user and the instructor.

[0495] The "means for providing instruction via video call" is a function that allows an instructor and a user to receive calligraphy instruction in real time via video call.

[0496] "Means for recording instruction content" is a function that allows you to record instruction content given via video calls, etc., so that you can review it later.

[0497] The "means for providing appropriate feedback according to the progress of instruction" is a function for performing emotion analysis and providing appropriate advice and encouraging messages to instructors and users according to the progress of instruction.

[0498] The present invention relates to a calligraphy learning support system. This system is designed to enable users who want to learn calligraphy to receive high-quality calligraphy instruction from home. In particular, by combining it with sentiment analysis technology, it can provide individually optimized feedback to improve the user's learning experience.

[0499] System configuration

[0500] The system consists of the following main components:

[0501] 1. User device (smartphone, tablet, or PC):

[0502] An interface for users to enter the characters they want to write.

[0503] Camera function for capturing images of calligraphy works.

[0504] Camera and microphone for sentiment analysis.

[0505] 2. Server:

[0506] A function that analyzes text information and searches for example data.

[0507] A function to send sample data to the user's terminal.

[0508] Runs AI algorithms that analyze user-uploaded images and provide scoring and correction.

[0509] A function that generates feedback based on the results of sentiment analysis.

[0510] 3. Emotion Engine:

[0511] Technologies that analyze facial expressions and tone of voice to identify a user's emotional state in real time (e.g., the DeepFace library).

[0512] 4. Interface:

[0513] An application for presenting exemplar data and evaluating user-created work.

[0514] An interface with video calling capabilities for remote instruction.

[0515] A UI that displays the scoring results and corrections.

[0516] Example of operation

[0517] 1. Example process:

[0518] The user inputs the characters they want to write into the device. For example, they input the character "beauty" into the input field. The device sends the input character information to the server, which analyzes it and searches for corresponding example data from a database. The search results are sent to the user's device, and the example data is presented. This allows the user to check high-quality examples.

[0519] 2. Practice and submission process:

[0520] Users practice calligraphy based on the provided model. They then take a photo of their work with their smartphone and import it into the device. The image data is sent to a server and analyzed by an AI algorithm. The calligraphy is scored and corrected based on factors such as brush strokes and character balance, and the results are sent back to the device. Users can use this feedback to improve their next practice.

[0521] 3. Emotion recognition process:

[0522] During practice, the emotion engine analyzes the user's facial expressions and voice to identify their emotional state in real time. The analysis results are sent to a server, which generates appropriate feedback based on the results. The feedback is displayed on the device to help maintain the user's motivation.

[0523] Specific technologies used

[0524] DeepFace: A library for sentiment analysis.

[0525] TextBlob: A library for generating feedback messages using natural language processing.

[0526] OpenCV: A library for image processing.

[0527] Specific examples

[0528] For example, when a user wants to learn the character "beautiful", the user inputs "beautiful" into the application. The terminal sends the character information to the server, and the server sends the model data to the terminal. The user practices while looking at the presented model, takes a photo of the completed work with a smartphone, and uploads it to the application. The server analyzes the image and performs evaluation and correction. The emotion engine particularly analyzes the state of being highly concentrated or tired, and based on this, generates messages such as "You are concentrated. Keep it up like this." or "It might be good to take a little break."

[0529] Example of prompt sentence

[0530] For example, when analyzing the emotion during calligraphy practice and detecting the state of "being concentrated", the prompt sentence is as follows:

[0531] User's emotion: Concentrated

[0532] Instruction message: Great concentration! Keep it up like this.

[0533] As described above, this system is designed so that users can receive high-quality calligraphy education even from home and can effectively proceed with learning. By utilizing emotion analysis technology, the learning experience of users can be made more fulfilling.

[0534] The flow of the specific process in Application Example 2 will be described using FIG. 14.

[0535] Step 1:

[0536] The user inputs the character to be written.

[0537] Input: The user inputs a character such as "beautiful" into the terminal.

[0538] Process: The terminal acquires the input character information and sends it to the server.

[0539] Output: Text information sent to the server.

[0540] Step 2:

[0541] Search and display exemplar data.

[0542] Input: Text information sent to the server.

[0543] Processing: The server analyzes the character information and searches for corresponding example data from the example database.

[0544] Output: The retrieved example data is sent to the terminal and presented to the user.

[0545] Step 3:

[0546] The user practices calligraphy.

[0547] Input: exemplar data.

[0548] Processing: The user practices calligraphy using paper and a brush based on the presented example data.

[0549] Output: User-created work.

[0550] Step 4:

[0551] Take a photo of your work and import it into your device.

[0552] Input: A user-created calligraphy piece.

[0553] Processing: The user takes a photo of the work using the smartphone camera and imports it into the device as image data.

[0554] Output: The captured image data.

[0555] Step 5:

[0556] The image data is sent to the server.

[0557] Input: Image data of the work.

[0558] Processing: The image data captured by the terminal is sent to the server.

[0559] Output: Image data sent to the server.

[0560] Step 6:

[0561] Analyze image data and perform scoring and correction.

[0562] Input: Image data sent to the server.

[0563] Processing: The server uses AI algorithms to analyze the image, scoring and correcting it based on criteria such as brushstrokes and balance.

[0564] Output: Marking results and corrections.

[0565] Step 7:

[0566] The scoring results and corrections will be presented.

[0567] Input: Marking results and corrections.

[0568] Processing: The server sends the generated scoring results and corrections to the terminal and presents them to the user.

[0569] Output: The marks and corrections presented to the user.

[0570] Step 8:

[0571] Conduct sentiment analysis.

[0572] Input: User's facial expression and tone of voice.

[0573] Processing: The emotion engine analyzes the user's facial expressions and tone of voice to recognize their emotional state in real time.

[0574] Output: Parsed emotion information.

[0575] Step 9:

[0576] Generate and present feedback based on emotional information.

[0577] Input: Parsed emotion information.

[0578] Processing: The server generates an appropriate feedback message (e.g., "You're concentrating, so keep it up") based on the emotional information and sends it to the device.

[0579] Output: The feedback message presented to the user.

[0580] Step 10:

[0581] Users get feedback that helps them improve their practice next time.

[0582] Input: Feedback message.

[0583] Action: The user checks the feedback message provided and adjusts the content of the next practice based on it.

[0584] Output: Your next calligraphy piece based on your improved practice.

[0585] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0586] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0587] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0588] [Second embodiment]

[0589] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0590] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0591] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0592] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0593] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0594] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0595] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0596] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0597] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0598] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0599] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0600] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0601] The present invention relates to a calligraphy learning support system that enables users who wish to learn calligraphy to receive high-quality calligraphy instruction and enables remote calligraphy instruction using artificial intelligence (AI).

[0602] System configuration

[0603] This system consists of the following main elements:

[0604] 1. User device: Tablet, smartphone, or PC. Users use these devices to study calligraphy.

[0605] 2. Server: The core part of the system installed in the center. It provides a database of calligraphy models, image analysis by AI, and grading and commenting functions.

[0606] 3. Interface: Software or applications for the user and the system to interact with each other.

[0607] Program processing

[0608] Model presentation process

[0609] 1. The user inputs the characters they want to write on the terminal. For example, input the character "美" into the input field of the application.

[0610] 2. The terminal sends the input character information to the server.

[0611] 3. The server receives the character information, analyzes it, and searches for the corresponding model data from the database.

[0612] 4. The server sends the search results to the terminal.

[0613] 5. The terminal displays the sent model data and presents it to the user. In this way, the user can view the model while at home.

[0614] Practice and work submission process​​​​​​​​​​​​​​​4. The server receives the image and analyzes it using an AI algorithm, scoring it based on multiple criteria, including brush strokes and balance of the characters.

[0619] 5. The server generates the scoring results and corrections and sends them to the terminal.

[0620] 6. The device displays the received results to the user, allowing the user to receive feedback and use it for further practice.

[0621] Optional Feature Process

[0622] 1. Dan / Kyu rank acquisition:

[0623] A user submits multiple practice pieces to the server and receives a certain evaluation.

[0624] The server accumulates the user's practice data and notifies them when they meet the requirements for obtaining a rank or grade.

[0625] Users submit works that meet the requirements, and the server performs a final evaluation to determine rank and grade.

[0626] 2. Remote instruction:

[0627] The user selects the calligraphy instructor from whom they would like to receive remote instruction within the app.

[0628] The server notifies the selected instructor and schedules the instruction.

[0629] The user and the calligraphy instructor will conduct remote instruction via video call.

[0630] The server records the instruction content so that the user can review it later.

[0631] Specific examples

[0632] For example, if a user wants to practice the character "美", first enter "美" into the application. The terminal sends this information to the server, and the server retrieves the corresponding model data from the database and sends it to the terminal. The user practices while looking at the presented model, takes a photo of the completed work with a smartphone, and uploads it to the terminal. The server analyzes the image, sends the results of evaluation and correction to the user, and the user can use them for the next practice.

[0633] In addition, users aiming to obtain ranks or grades can submit works that meet the conditions upon receiving a notification from the server, and finally be evaluated to obtain ranks or grades. Furthermore, if external guidance is required, the user can select a desired instructor within the application and receive remote guidance. Through this series of processes, it is realized that high-quality calligraphy education can be received even from home.

[0634] The following describes the process flow.

[0635] Model Presentation Process

[0636] Step 1:

[0637] The user inputs the character to be written on the terminal.

[0638] Specifically, the user opens the application and inputs the character to be written, such as "美", into the text input field.

[0639] Step 2:

[0640] The terminal sends the input character information to the server.

[0641] Specifically, the terminal issues a request to send the character data input by the user to the server through the API.

[0642] Step 3:

[0643] The server receives the text information, analyzes it, and searches the database for corresponding example data.

[0644] Specifically, the server receives the request and uses a character recognition module to query the database and retrieve the corresponding example data.

[0645] Step 4:

[0646] The server sends the search results to the terminal.

[0647] Specifically, the server transmits the acquired sample data to the terminal as a response.

[0648] Step 5:

[0649] The terminal displays the transmitted example data and presents it to the user.

[0650] Specifically, the terminal receives the response and displays the example image within the application.

[0651] Practice and submission process

[0652] Step 1:

[0653] The user practices calligraphy based on the displayed example.

[0654] Specifically, the user uses paper and a brush to write characters according to the provided example.

[0655] Step 2:

[0656] Once the practice piece is complete, the user takes a picture of it and imports it into their device.

[0657] Specifically, users take a photo of the work using their smartphone camera and upload the image to the app.

[0658] Step 3:

[0659] The terminal transmits the captured image data to the server.

[0660] Specifically, the terminal issues a request to upload image data to the server via the API.

[0661] Step 4:

[0662] The server receives the images and analyzes them using AI algorithms.

[0663] Specifically, the server passes the image data to an analysis engine, which evaluates it based on multiple criteria, such as brush strokes and character balance.

[0664] Step 5:

[0665] The server generates the scoring results and corrections and sends them to the terminal.

[0666] Specifically, the server calculates a score based on the analysis results, generates feedback including points for improvement, and sends it to the device.

[0667] Step 6:

[0668] The terminal displays the received results to the user.

[0669] Specifically, the device receives a response from the server and displays the scoring results and corrections within the app.

[0670] Optional Feature Process

[0671] Obtaining Dan / Kyu rank

[0672] Step 1:

[0673] A user submits multiple practice pieces to the server and receives a certain evaluation.

[0674] Specifically, users continue to upload their work to the app and accumulate ratings.

[0675] Step 2:

[0676] The server accumulates the user's practice data and notifies them when they meet the requirements for obtaining a rank or grade.

[0677] Specifically, the server checks the user's evaluation log and determines whether the acquisition conditions are met. If the conditions are met, the server notifies the user.

[0678] Step 3:

[0679] Users submit works that meet the requirements, and the server performs a final evaluation to determine rank and grade.

[0680] Specifically, users submit their work to be evaluated through a specified procedure, and the server performs a final evaluation to determine the rank and grade.

[0681] Remote instruction

[0682] Step 1:

[0683] The user selects the calligraphy instructor from whom they would like to receive remote instruction within the app.

[0684] Specifically, the user selects the desired instructor from a list of instructors in the app.

[0685] Step 2:

[0686] The server notifies the selected instructor and schedules the instruction.

[0687] Specifically, the server sends requests to instructors and provides an interface for scheduling.

[0688] Step 3:

[0689] The user and the calligraphy instructor will conduct remote instruction via video call.

[0690] Specifically, the user and instructor will provide instruction via a video call app at a specified time.

[0691] Step 4:

[0692] The server records the instruction content so that the user can review it later.

[0693] Specifically, the server stores recorded data of video calls and makes it available for users to review if necessary.

[0694] Example 1

[0695] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0696] Conventional calligraphy education systems require face-to-face instruction and practice using paper and brush in order for users to receive high-quality calligraphy training, which can be difficult for users living in remote locations. There are also issues with the lack of prompt feedback on the work users have practiced and evaluation of their ranks and grades. Furthermore, there is a lack of a flexible system that allows for real-time remote instruction. To address these issues, an effective and efficient calligraphy learning support system is needed.

[0697] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0698] In this invention, the server includes means for a user to input characters that the user wants to write, means for analyzing the input character information and searching for model data, means for presenting the searched model data, means for capturing an image of a work created by the user, means for analyzing the captured image using an artificial intelligence algorithm and grading and correcting the image, means for presenting the grading results and corrections to the user, means for notifying the user of the conditions for obtaining a rank or grade, means for evaluating works that satisfy the conditions, means for determining the rank or grade, means for the user to select an instructor from whom they wish to receive remote instruction, means for notifying the selected instructor and arranging a schedule for instruction, means for providing instruction via video call, and means for recording the content of instruction. This enables users to receive high-quality calligraphy education from the comfort of their own homes, receive prompt feedback and evaluation, and receive real-time remote instruction.

[0699] "User" refers to a user of the system who intends to receive calligraphy education.

[0700] A "terminal" is a device that allows a user to use the system, and includes a tablet, smartphone, or PC.

[0701] The "server" is the central computer of the system, and is a device that provides functions for searching sample data, analyzing images, and scoring and correcting.

[0702] "Model data" is calligraphy sample data that is presented as a standard for the characters that the user wants to learn.

[0703] "Means for capturing images" refers to devices and functions necessary for inputting a user's work into the system as a digital image, and includes camera functions, etc.

[0704] An "artificial intelligence algorithm" is a program that runs on a server and analyzes, grades, and corrects works created by users.

[0705] "Scoring result" is the evaluation score given to the user's work by the artificial intelligence algorithm.

[0706] "Correction content" is feedback information indicating corrections and improvements provided to the user's work.

[0707] "Rank / Grade" refers to the qualification grade that indicates the level of the user's calligraphy skills.

[0708] "Acquisition conditions" are the requirements that a user must meet in order to acquire a rank or grade.

[0709] The "instructor who wishes to receive remote instruction" is a calligraphy instructor that the user selects within the system to receive calligraphy instruction via video call.

[0710] "Video calling" is a means of communication that allows for the exchange of video and audio in real time over the Internet.

[0711] "Means for recording instruction content" refers to the ability to digitally save instruction given during a video call.

[0712] This invention is a calligraphy learning support system that allows users to receive high-quality calligraphy training. The system mainly consists of a user terminal, a server, and a user interface.

[0713] System configuration

[0714] 1. User device: A device such as a tablet, smartphone, or PC that users use to learn calligraphy.

[0715] 2. Server: This is the core device of the system, providing a database of calligraphy samples, image analysis using AI, and scoring and correction functions.

[0716] 3. User Interface: Software or applications that allow users to interact with a system.

[0717] Model Prompt

[0718] 1. The user inputs the characters they want to write in the input field of the application. For example, input the character "美".

[0719] 2. The terminal sends the input character information to the server. Specifically, it uses an HTTP request to send the character information to the server.

[0720] 3. The server receives the character information, analyzes it, and searches for the corresponding model data from the database. The server executes an analysis script written in Python and uses MySQL for the database.

[0721] 4. The server sends the search results to the terminal, and the terminal displays the received model data on the application.

[0722] Practice and Submission of Works

[0723] The user practices calligraphy referring to the model. Using paper and pen, for example, write the character "美". When the practice work is completed, the user uses the camera function of the smartphone to take a photo of the work and import it into the terminal as an image. The terminal sends the captured image data to the server via an HTTP POST request.

[0724] The server receives the image data and performs analysis using an AI algorithm based on TensorFlow. It evaluates aspects such as the stroke movement and the balance of the characters, and generates a scoring result and correction content. The generated data is in JSON format, and the server sends this to the terminal. Then, the terminal displays the received result on the application. The evaluation and correction content are visually presented using HTML and CSS.

[0725] Obtaining Dan or Ji Ranks

[0726] The user sends multiple practice works to the server and receives evaluations. The server accumulates the user's practice data in the database and sends a notification when the conditions for obtaining a dan or kyū rank are met. The works that meet the conditions for the final evaluation are sent again, and the server finally determines the dan or kyū rank and sends the evaluation results to the terminal.

[0727] Remote guidance

[0728] When the user wishes to receive remote guidance within the application, an instructor is selected using the interface within the system. The server notifies the selected instructor and arranges the guidance schedule. WebRTC technology is used for video calls to enable real-time guidance. The content of the guidance is saved to cloud storage and recorded using Amazon S3.

[0729] Specific example

[0730] For example, if the user wants to practice the character "美", they input "美" into the application. The terminal sends this information to the server, and the server retrieves the corresponding model data from the database and sends it to the terminal. The user practices while looking at the presented model, takes a photo of the completed work with a smartphone, and uploads it to the terminal. The server analyzes the image, sends the results of the evaluation and correction to the user, which can be used for the next practice. Also, when obtaining a dan or kyū rank, multiple works are submitted for evaluation, and a notification is received when the criteria are met, making it possible to obtain a dan or kyū rank.

[0731] Examples of prompt sentences

[0732] Please display a model for writing the character "美".

[0733] Search for and display the model of the input character "美" from the database. Also, please take a photo of the user's practice work and evaluate it.

[0734] The flow of the specific process in Example 1 will be described using FIG. 11.

[0735] Step 1:

[0736] The user inputs the characters they want to write in the input field of the application. The data to be input is Chinese characters such as "beautiful". The user inputs characters in the input field and performs the operation of pressing the send button.

[0737] Step 2:

[0738] The terminal sends the input character information to the server. The input is the character information input by the user, and the output is an HTTP request to the server. The data is sent in JSON format. At this time, the terminal performs the operation of sending a request to the server using network communication.

[0739] Step 3:

[0740] The server receives the character information and analyzes the characters. The input is the character information sent from the terminal, and the output is the character code as the analysis result. The server executes a Python script and performs the operation of analyzing the specified characters.

[0741] Step 4:

[0742] The server searches for the corresponding template data from the database based on the character code. The input is the analyzed character code, and the output is the template data. The server executes an SQL query against MySQL and performs the operation of obtaining the relevant template data.

[0743] Step 5:

[0744] The server sends the search result to the terminal in JSON format. The input is the template data obtained from the database, and the output is an HTTP response to the terminal. The server converts the obtained data into JSON format and performs the operation of sending it as an HTTP response.

[0745] Step 6:

[0746] The example data received by the device is displayed on the application. The input is the example data sent from the server, and the output is the example displayed on the user interface. The device uses JavaScript to draw the example data.

[0747] Step 7:

[0748] The user practices calligraphy based on the displayed example. Using paper and a brush, the user writes the character "beauty."

[0749] Step 8:

[0750] Once the practice piece is complete, the user takes a photo of the piece using the smartphone's camera function and imports the image into the device. The input is the image of the completed piece, and the output is a digital image saved on the device. The user uses the camera function to acquire the image.

[0751] Step 9:

[0752] The device sends the captured image data to the server using an HTTP POST request. The input is the image data captured by the user, and the output is a request to the server. The device uses network communication to send the image data to the server.

[0753] Step 10:

[0754] The server receives the image data and analyzes it using an AI algorithm. The input is the transmitted image data, and the output is the evaluation data as the analysis result. The server uses TensorFlow to analyze the image and evaluate the brush strokes and balance of the characters.

[0755] Step 11:

[0756] The server generates the scoring results and corrections and sends them to the device. The input is the analysis result, and the output is the evaluation results and corrections sent to the device. The server converts the evaluation data into JSON format and sends it as an HTTP response.

[0757] Step 12:

[0758] The terminal displays the results it receives on the application. The input is the evaluation results and corrections sent from the server, and the output is the feedback displayed on the user interface. The terminal uses HTML and CSS to visually display the results.

[0759] Step 13:

[0760] The user performs further practice based on the provided feedback. The input is the displayed feedback, and the output is an improved practice piece. The user then uses the feedback to perform the writing action again.

[0761] (Application example 1)

[0762] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0763] Conventional calligraphy learning systems provide paper-based examples and feedback, making real-time instruction and evaluation difficult. Furthermore, due to limited learning support in remote locations, students must attend a calligraphy class in order to receive high-quality calligraphy education. Furthermore, there are currently insufficient methods for improving the efficiency of calligraphy learning using smart devices.

[0764] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0765] In this invention, the server includes means for inputting characters to be written, means for analyzing the input character information and searching for model data, means for presenting the model data, means for capturing a work created by the user as an image, means for analyzing the captured image and performing scoring and corrections, means for presenting the scoring results and corrections, means for displaying the model data using smart glasses, means for photographing the created work using the camera function of the smart glasses, means for transmitting the image data to a cloud server and analyzing it using an AI algorithm, and means for displaying the analysis results and feedback on the smart glasses. This enables real-time feedback, enables high-quality calligraphy education even from remote locations, and realizes efficient calligraphy learning by utilizing smart devices.

[0766] The "means for inputting characters to be written" is an interface for inputting characters to be written into a device when a user is learning calligraphy.

[0767] The "means for analyzing input character information and searching for model data" is a function that analyzes character information input by the user and searches for corresponding calligraphy model data from a database.

[0768] The "means for presenting example data" is a function for displaying the retrieved example data on the display of the terminal so that the user can confirm it.

[0769] The "means for capturing a work created by a user as an image" is a function for capturing a calligraphy work that a user has written on paper or the like as a digital image using a camera or the like.

[0770] The "means for analyzing the imported image and scoring and correcting it" refers to an AI algorithm that analyzes the imported image of the calligraphy work and automatically scores and corrects it.

[0771] The "means for presenting the scoring results and corrections" is a function for displaying the analysis results and corrections in an easy-to-understand manner to the user.

[0772] "Means for displaying example data using smart glasses" refers to a function that uses the display of the smart glasses to virtually display example data within the user's field of view.

[0773] "Means for photographing a work created using the camera function of the smart glasses" refers to a function for photographing a work written by a user using the camera built into the smart glasses.

[0774] "Means for sending image data to a cloud server and analyzing it using an AI algorithm" refers to a function that uploads image data of calligraphy works sent from smart glasses or other devices to a server on the cloud and uses an AI algorithm to analyze the data.

[0775] "Means for displaying analysis results and feedback on the smart glasses" is a function for displaying the analysis results and feedback returned from the cloud server on the display of the smart glasses.

[0776] This invention is a system that utilizes smart glasses to provide real-time calligraphy learning support. This system allows users who want to learn calligraphy to receive high-quality calligraphy education using smart glasses and a cloud server.

[0777] System Configuration

[0778] 1. User Device

[0779] Smart glasses (e.g., Google Glass) are used as the user terminal. This device has the function of displaying the sample data, taking a photo of the written characters, and sending the photo to a cloud server.

[0780] 2. Cloud Server

[0781] The cloud server provides a database of calligraphy models, AI-based image analysis, and grading and correction functions. It functions as an infrastructure for real-time data processing and analysis.

[0782] 3. Interface

[0783] An interface is provided for the user to interact with the system through the display, operation panel, and voice recognition function of the smart glasses.

[0784] Outline of Program Execution

[0785] Model Presentation Process

[0786] The server provides a means for the user to input the characters they want to write through voice or touchpad operations using the smart glasses. For example, when the user inputs the character "beautiful", the server analyzes the character information and searches the database for the corresponding model data. The model data is displayed on the smart glasses, and the user can practice calligraphy while viewing it in real time.

[0787] Practice and Work Submission Process

[0788] The user practices calligraphy while viewing the model displayed through the smart glasses. After practice, the user uses the camera function of the smart glasses to take a picture of the work and transmits the image data to the cloud server. The cloud server analyzes the image using an AI algorithm and evaluates aspects such as the stroke movement and character balance. The analysis results are displayed on the smart glasses together with the calligraphy correction content, and the user can obtain real-time feedback.

[0789] Hardware and Software to be Used

[0790] This system operates with smart glasses, a cloud server, and an AI analysis model (using, for example, TensorFlow) as its main components. The smart glasses function as a user interface, and the cloud server provides data processing and storage solutions. An image analysis model is used for AI analysis to perform a detailed evaluation of the user's calligraphy works.

[0791] Examples of specific cases and prompt sentences

[0792] For example, when the user wants to practice the character "beautiful", first, the user inputs the command "display the model" to the smart glasses by voice or touchpad operation. The following are examples of prompt sentences.

[0793] Prompt sentence: "Please input the model character."

[0794] When the user practices calligraphy and the character is completed, next, the user activates the camera with the gesture function of the smart glasses and takes a photo of the work. The following are examples of prompt sentences.

[0795] Prompt sentence: "When the character is completed, please activate the camera and take a photo."

[0796] The captured image data is sent to the cloud server, and the AI model performs analysis and displays the results and feedback on the display of the smart glasses. When the analysis is completed, the following prompt sentence is displayed.

[0797] <s Prompt sentence: "The analysis is complete. Please check the feedback."

[0798] In this way, a system is provided that allows the user to practice calligraphy and receive feedback in real time.

[0799] The flow of specific processing in Application Example 1 will be described using FIG. 12.

[0800] Step 1:

[0801] The user wears smart glasses and inputs the characters to be written. The input method is voice recognition or touchpad operation. The user inputs "beautiful" in response to the prompt "Please input the reference characters". The input character information is sent by the smart glasses to the cloud server. The server analyzes the received character information and searches for the corresponding reference data from the reference database.

[0802] Step 2:

[0803] The server sends the reference data to the smart glasses, and the smart glasses display the data in the user's field of vision. Thereby, the user can start practicing calligraphy while checking the reference in real time. The display of the reference data is projected onto the display of the smart glasses, and the user writes characters on the paper while looking at the reference.

[0804] Step 3:

[0805] After the user finishes writing the characters referring to the reference, the user uses the camera function of the smart glasses to take a picture of the work. The user follows the prompt "Please start the camera and take a picture when the characters are completed" using gesture operation or the touchpad to take a picture of the work. The captured image data is sent by the smart glasses to the cloud server.

[0806] Step 4:

[0807] The server analyzes the received image data using an AI algorithm. As data processing, preprocessing of the image and feature extraction are performed, and analysis is performed using evaluation criteria such as the stroke movement, character balance, and line thickness. A generative AI model such as TensorFlow is used for this analysis. As a result of the analysis, scores and detailed correction contents are generated.

[0808] Step 5:

[0809] The server sends the generated analysis results and feedback to the smart glasses, which then display the received results in the user's field of view. Following the prompt "Analysis completed. Please check your feedback," feedback is presented to the user, allowing the user to use this information to improve their practice next time.

[0810] Step 6:

[0811] When a user practices multiple times and meets certain conditions, the conditions for obtaining a rank or grade are notified to the smart glasses. The server accumulates the user's practice data and notifies the user when the conditions are met. The user then photographs and submits works that meet the conditions, and the server performs a final evaluation to determine the rank or grade.

[0812] Step 7:

[0813] When a user requests remote instruction, they select an instructor through the smart glasses interface. The server then notifies the instructor and schedules the instruction. The instructor provides real-time instruction to the user through a video call function. The instruction is then recorded on the server for later review by the user.

[0814] Through this series of steps, the calligraphy learning support system, which utilizes smart glasses and a cloud server, provides users with an efficient and high-quality learning environment.

[0815] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0816] The present invention relates to a calligraphy learning support system. This system is for users who want to learn calligraphy to receive high-quality calligraphy education and enables remote calligraphy guidance using artificial intelligence (AI). Also, by combining an emotion engine that analyzes the emotions of users, the learning experience of users can be further improved.

[0817] Configuration of the System

[0818] This system is composed of the following main elements.

[0819] 1. User terminal: Such as a tablet, smartphone, or personal computer. The user uses these terminals to conduct calligraphy learning.

[0820] 2. Server: The core part of the system installed centrally. It provides a database of calligraphy models, image analysis by AI, and scoring and correction functions.

[0821] 3. Interface: Software or applications for the user and the system to interact with each other.

[0822] 4. Emotion engine: A device or software that analyzes the user's facial expressions and tone of voice and recognizes the emotional state.

[0823] Processing of the Program

[0824] Model Presentation Process

[0825] 1. The user inputs the characters they want to write on the terminal. For example, input the character "beautiful" into the input field of the application.

[0826] 2. The terminal sends the input character information to the server.

[0827] 3. The server receives the character information, analyzes it, and searches for the corresponding model data from the database.

[0828] 4. The server sends the search results to the device.

[0829] 5. The device displays the example data and presents it to the user, allowing the user to check the example from the comfort of their own home.

[0830] Practice and submission process

[0831] 1. The user practices calligraphy based on the displayed model, for example, writing the character "beauty" using paper and a brush.

[0832] 2. Once the practice piece is complete, the user takes a photo of it and imports it into their device, often using the camera function on their smartphone.

[0833] 3. The device sends the captured image data to the server.

[0834] 4. The server receives the image and analyzes it using an AI algorithm, scoring it based on multiple criteria, including brush strokes and balance of the characters.

[0835] 5. The server generates the scoring results and corrections and sends them to the terminal.

[0836] 6. The device displays the received results to the user, allowing the user to receive feedback and use it for further practice.

[0837] Emotion Recognition Process

[0838] 1. As the user practices calligraphy, the emotion engine analyzes the user's facial expressions and voice to recognize their emotional state in real time.

[0839] 2. The emotion engine provides feedback to the server based on the emotional state.

[0840] 3. The server receives information from the emotion engine, checks the progress of practice and motivation, and generates appropriate advice and encouraging messages.

[0841] 4. The device displays these messages to the user to help them stay motivated.

[0842] Optional Feature Process

[0843] Obtaining Dan / Kyu rank

[0844] 1. A user submits multiple practice pieces to the server and receives a certain evaluation.

[0845] 2. The server accumulates the user's practice data and notifies the user when the conditions for obtaining a rank or grade are met, allowing the user to check their own progress.

[0846] 3. The user submits works that meet the requirements, and the server performs a final evaluation to determine the rank and grade.

[0847] Remote instruction

[0848] 1. The user selects the calligraphy instructor with whom they would like to receive remote instruction within the app.

[0849] 2. The server notifies the selected instructor and schedules the instruction.

[0850] 3. The user and the calligraphy instructor will conduct remote instruction via video call.

[0851] 4. The server records the instruction so that the user can review it later.

[0852] 5. The emotion engine analyzes the user's emotional state during training and notifies the instructor of the results in real time, allowing the instructor to provide instruction that is appropriate for the user's emotional state.

[0853] Specific examples

[0854] For example, when a user wants to practice the character "beautiful", first enter "beautiful" into the application. The terminal sends this information to the server, and the server retrieves the corresponding model data from the database and sends it to the terminal. The user practices while looking at the presented model, takes a photo of the completed work with a smartphone, and uploads it to the terminal. The server analyzes the image, conducts evaluation and correction, and sends the results to the user, who can use them for the next practice.

[0855] Furthermore, during the user's practice, the emotion engine analyzes the user's expression and feedbacks to the server the state of concentration or fatigue. Based on this, the server can send messages to the user such as "You seem to be concentrating. Keep it up" or "It might be good to take a little break".

[0856] Users aiming to obtain a dan or kyū rank receive a notification from the server, submit works that meet the conditions, and can finally be evaluated to obtain a dan or kyū rank. Furthermore, if external guidance is required, the user can select a desired instructor within the application and receive remote guidance. Through this series of processes, it is realized that high-quality calligraphy education can be received even from home. By using the emotion engine, more effective learning becomes possible while maintaining the user's motivation and concentration.

[0857] The following describes the processing flow.

[0858] Model Presentation Process

[0859] Step 1:

[0860] The user inputs the character to be written on the terminal.

[0861] Specifically, the user opens the application and inputs the character to be written, such as "beautiful", into the text input field.

[0862] Step 2:

[0863] The terminal transmits the input character information to the server.

[0864] Specifically, the terminal issues a request to send the character data entered by the user to the server via the API.

[0865] Step 3:

[0866] The server receives the text information, analyzes it, and searches the database for corresponding example data.

[0867] Specifically, the server receives the request and uses a character recognition module to query the database and retrieve the corresponding example data.

[0868] Step 4:

[0869] The server sends the search results to the terminal.

[0870] Specifically, the server transmits the acquired sample data to the terminal as a response.

[0871] Step 5:

[0872] The terminal displays the transmitted example data and presents it to the user.

[0873] Specifically, the terminal receives the response and displays the example image within the application.

[0874] Practice and submission process

[0875] Step 1:

[0876] The user practices calligraphy based on the displayed example.

[0877] Specifically, the user uses paper and a brush to write characters according to the provided example.

[0878] Step 2:

[0879] Once the practice piece is complete, the user takes a picture of it and imports it into their device.

[0880] Specifically, users take a photo of the work using their smartphone camera and upload the image to the app.

[0881] Step 3:

[0882] The terminal transmits the captured image data to the server.

[0883] Specifically, the terminal issues a request to upload image data to the server via the API.

[0884] Step 4:

[0885] The server receives the images and analyzes them using AI algorithms.

[0886] Specifically, the server passes the image data to an analysis engine, which evaluates it based on multiple criteria, such as brush strokes and character balance.

[0887] Step 5:

[0888] The server generates the scoring results and corrections and sends them to the terminal.

[0889] Specifically, the server calculates a score based on the analysis results, generates feedback including points for improvement, and sends it to the device.

[0890] Step 6:

[0891] The terminal displays the received results to the user.

[0892] Specifically, the device receives a response from the server and displays the scoring results and corrections within the app.

[0893] Emotion Recognition Process

[0894] Step 1:

[0895] As the user practices calligraphy, the emotion engine analyzes the user's facial expressions and voice to recognize their emotional state in real time.

[0896] Specifically, the emotion engine collects and analyzes the user's facial expressions and tone of voice through the device's camera and microphone.

[0897] Step 2:

[0898] An emotional engine provides feedback to the server based on the emotional state.

[0899] Specifically, the emotion engine analyzes the user's current emotional state (e.g., focused, tired) and sends that data to the server.

[0900] Step 3:

[0901] The server receives information from the emotion engine, checks the progress of practice and motivation, and generates appropriate advice and encouraging messages.

[0902] Specifically, the server generates messages based on emotional data, such as "You seem to be concentrating, keep it up" or "Maybe it would be good to take a short break."

[0903] Step 4:

[0904] The device displays these messages to the user to help them stay motivated.

[0905] Specifically, the terminal displays the message sent from the server on the screen.

[0906] Optional Feature Process

[0907] Obtaining Dan / Kyu rank

[0908] Step 1:

[0909] A user submits multiple practice pieces to the server and receives a certain evaluation.

[0910] Specifically, users continue to upload their work to the app and accumulate ratings.

[0911] Step 2:

[0912] The server accumulates the user's practice data and notifies them when they meet the requirements for obtaining a rank or grade.

[0913] Specifically, the server checks the user's evaluation log and determines whether the acquisition conditions are met. If the conditions are met, the server notifies the user.

[0914] Step 3:

[0915] Users submit works that meet the requirements, and the server performs a final evaluation to determine rank and grade.

[0916] Specifically, users submit their work to be evaluated through a specified procedure, and the server performs a final evaluation to determine the rank and grade.

[0917] Remote instruction

[0918] Step 1:

[0919] The user selects the calligraphy instructor from whom they would like to receive remote instruction within the app.

[0920] Specifically, the user selects the desired instructor from a list of instructors in the app.

[0921] Step 2:

[0922] The server notifies the selected instructor and schedules the instruction.

[0923] Specifically, the server sends requests to instructors and provides an interface for scheduling.

[0924] Step 3:

[0925] The user and the calligraphy instructor will conduct remote instruction via video call.

[0926] Specifically, the user and instructor will provide instruction via a video call app at a specified time.

[0927] Step 4:

[0928] The server records the instruction content so that the user can review it later.

[0929] Specifically, the server stores recorded data of video calls and makes it available for users to review if necessary.

[0930] Step 5:

[0931] The emotion engine analyzes the user's emotional state during instruction and notifies the instructor of the results in real time.

[0932] Specifically, the emotion engine sends the results of analyzing the user's facial expressions and voice to the server, which then displays the results on the instructor's screen.

[0933] Specific examples

[0934] For example, when a user wants to practice the character "beautiful", first enter "beautiful" into the application. The terminal sends this information to the server, and the server retrieves the corresponding model data from the database and sends it to the terminal. The user practices while looking at the presented model, takes a photo of the completed work with a smartphone, and uploads it to the terminal. The server analyzes the image, conducts evaluation and correction, and sends the results to the user, who can use them for the next practice.

[0935] Furthermore, during the user's practice, the emotion engine analyzes the user's expression and provides feedback to the server on the state of concentration or fatigue. Based on this, the server can send messages to the user such as "You seem to be concentrating. Keep it up like this" or "It might be good to take a short break."

[0936] Users aiming to obtain ranks or grades can receive notifications from the server, submit works that meet the conditions, and ultimately be evaluated to obtain ranks or grades. Furthermore, if external guidance is needed, the user can select a desired instructor within the application and receive remote guidance. Through this series of processes, it is realized that high-quality calligraphy education can be received even from home. By using the emotion engine, more effective learning becomes possible while maintaining the user's motivation and concentration.

[0937] (Example 2)

[0938] Next, Example 2 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0939] In a conventional calligraphy education system, it is difficult for users to receive high-quality feedback, especially there are limitations in guidance from remote locations. Also, due to the lack of means to appropriately manage the user's emotions and motivation, efficient learning has been difficult. The process for users to obtain ranks or grades is also inefficient, and it requires a lot of time and effort to acquire advanced calligraphy techniques.

[0940] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0941] In this invention, the server includes a means for inputting characters to be written, a means for analyzing the input character information and searching for model data, a means for presenting the model data, a means for capturing the work created by the user as an image, a means for analyzing the captured image and performing scoring and corrections, a means for presenting the scoring results and corrections, a means for analyzing the user's facial expressions and voice to recognize emotions, and a means for providing feedback based on the emotion recognition results. This allows for high-quality calligraphy education even in a remote environment, and enables the user's emotions to be analyzed in real time and appropriate feedback to be provided. Furthermore, since ranks and grades can be acquired efficiently, users can more effectively acquire calligraphy techniques.

[0942] The "means for inputting characters to be written" refers to an input field or operation means for the user to input characters to be written in calligraphy practice into the application.

[0943] The "means for analyzing input character information and searching for example data" refers to an algorithm or program for receiving character information input by a user, analyzing it, and searching for corresponding example data from a database.

[0944] The "means for presenting model data" refers to a display device or software for receiving the model data sent from the server and displaying it on the user's terminal.

[0945] The "means for capturing an image of a work created by a user" refers to a camera function or an image input device that allows a user to take an image of a calligraphy work and capture it into the terminal.

[0946] The "means of analyzing the captured image and grading and correcting" refers to an AI algorithm or analysis program that analyzes the captured image, evaluates the brush strokes and balance of the characters, and performs grading and correction.

[0947] The "means for presenting the marking results and corrections" refers to a UI component or notification system for displaying the marking results and corrections generated by the server to the user.

[0948] "Means for recognizing emotions by analyzing the user's facial expressions and voice" refers to facial recognition technology and voice analysis technology that analyze the user's facial expressions and tone of voice to understand their current emotional state.

[0949] The "means for providing feedback based on emotion recognition results" refers to software or a system for generating appropriate advice or encouraging messages based on the user's emotion recognition results and providing them to the user.

[0950] "Means for notifying the conditions for obtaining a rank or grade" refers to algorithms and communication methods for confirming the conditions necessary for a user to obtain a rank or grade and notifying the user of the status of achievement.

[0951] The "means for evaluating works that satisfy the acquisition conditions" refers to an examination algorithm or evaluation system that allows users to submit works that satisfy the conditions for acquiring a rank or grade and evaluates them.

[0952] The "means for determining ranks and grades" refers to a judging system for determining final ranks and grades based on the evaluation results of works submitted by users.

[0953] The "means for selecting an instructor from whom the user wishes to receive remote instruction" refers to a selection screen or system that allows the user to select an instructor from whom the user wishes to receive remote instruction within the application.

[0954] The "means for notifying the selected instructor and arranging the schedule for instruction" refers to a scheduling system or communication means for sending notification to the selected instructor and arranging the schedule for instruction.

[0955] The "means for providing instruction via video call" refers to a video call system or software that allows a user and an instructor to provide instruction remotely via video call.

[0956] The "means for recording the instruction content" refers to a recording device or software that records the instruction content given during the video call in real time so that the user can review it later.

[0957] "Means for analyzing the user's emotions during training and notifying the instructor" refers to an analysis system or communication means for analyzing the user's facial expressions and voice during a video call and notifying the instructor of their emotional state in real time.

[0958] This invention is a system for providing high-quality calligraphy education, designed to enable users to effectively learn calligraphy even in remote environments. This system is composed of a user terminal, a server, an interface, and an emotion engine.

[0959] System configuration

[0960] 1. User Device: Calligraphy lessons are conducted using devices such as tablets, smartphones, or PCs. These devices have built-in cameras that can be used to take photos of the works created by users.

[0961] 2. Server: This is the core part of the system, providing the database of calligraphy examples, AI image analysis, and scoring and correction functions. Specifically, AI frameworks such as TensorFlow and PyTorch are used for image analysis.

[0962] 3. Interface: The application or software that allows users to interact with the system. Through this interface, users can view calligraphy examples, upload practice pieces, and receive feedback.

[0963] 4. Emotion Engine: Technology that analyzes the user's facial expressions and tone of voice to recognize their emotional state. For example, OpenCV is used for facial recognition, and Google Cloud Speech-to-Text API is used for voice analysis.

[0964] Program processing

[0965] The program of this system performs the following processing.

[0966] Modeling process

[0967] When a user inputs the characters they want to write into the device, the device sends the character information to the server. The server analyzes the character information and searches for corresponding model data in a database. The resulting model data is sent to the device and presented to the user. This series of processes allows users to check calligraphy models from the comfort of their own home.

[0968] Practice and submission process

[0969] The user practices calligraphy based on the provided model and takes a photo of the completed work with their smartphone camera. The device sends the captured image data to a server, which receives the image and analyzes it using an AI algorithm. The analysis includes details such as brush strokes and character balance. The server generates an evaluation and corrections, which are sent to the device and displayed to the user. This allows the user to receive feedback that can be used to improve their next practice.

[0970] Emotion Recognition Process

[0971] During the process of the user practicing calligraphy, the emotion engine analyzes the user's expression and voice in real time. When the emotion engine feedbacks the analysis results to the server, the server confirms the progress and motivation of the practice based on them, and generates appropriate advice and encouraging messages. By the terminal displaying these messages to the user, more effective learning becomes possible while maintaining the user's motivation.

[0972] Optional function

[0973] This system provides the acquisition of ranks and grades and the remote guidance function as options. Regarding the acquisition of ranks and grades, when the user sends multiple practice works to the server and receives a certain evaluation, the server records and accumulates the data and notifies the user when the acquisition conditions for ranks and grades are met. In the remote guidance function, the user selects a desired instructor, the server notifies the instructor, and adjusts the guidance schedule. The user and the instructor conduct remote guidance through a video call, and the server records the guidance content. The emotion engine analyzes the emotion of the user during the guidance and provides real-time feedback to the instructor.

[0974] Specific example

[0975] For example, when the user wants to practice the character "美", first enter "美" into the application. The terminal sends this information to the server, and the server retrieves the corresponding model data from the database and sends it to the terminal. The user practices while looking at the presented model, takes a photo of the completed work with the smartphone, and uploads it to the terminal. The server analyzes the image, sends the evaluation and correction results to the user, and the user can use them for the next practice.

[0976] Furthermore, during the user's practice, the emotion engine analyzes the user's expression and feedbacks the concentrated state or tired state to the server. Based on this, the server can send messages such as "You seem to be concentrating. Keep it up like this" or "It might be good to take a little break".

[0977] Users aiming to obtain ranks or grades can receive notifications from the server, submit works that meet the conditions, and ultimately be evaluated to obtain ranks or grades. Additionally, if external guidance is required, users can select a desired instructor within the app and receive remote guidance. Through this series of processes, it becomes possible to receive high-quality calligraphy education even from home. By using the emotion engine, more effective learning is possible while maintaining the user's motivation and concentration.

[0978] Examples of prompt sentences

[0979] Use the following prompt sentences to explain the detailed processing content of the system to the generative AI model.

[0980] "In the calligraphy learning support system, please explain the specific processing steps of displaying a copybook of the character 'beautiful' and evaluating it after the user finishes practicing."

[0981] With this prompt sentence, the generative AI model explains each process of the system in an easy-to-understand natural language.

[0982] Explain the flow of the specific process in Example 2 using FIG. 13.

[0983] Copybook presentation process

[0984] Step 1:

[0985] The user inputs the character they want to write on the terminal. The user inputs a character such as 'beautiful' in the input field of the application. The input for this step is the user's input character, and the output is character information (such as 'beautiful').

[0986] Step 2:

[0987] The terminal sends the input character information to the server. Here, an HTTP request is used to send the character information to the server. The input for this step is the character information, and the output is the request sending to the server.

[0988] Step 3:

[0989] The server receives and analyzes the character information. The server uses a natural language processing (NLP) algorithm to analyze the character information and searches for corresponding model data from the database. The input for this step is the received character information, and the output is the search result of the model data.

[0990] Step 4:

[0991] The server sends the search result to the terminal. The server uses an HTTP response to send the model data to the terminal. The input for this step is the model data, and the output is the data sending to the terminal.

[0992] Step 5:

[0993] The terminal displays and presents the model data to the user. The terminal displays the received model data on the user interface (UI). This includes an image viewer and SVG rendering. The input for this step is the model data, and the output is the display of the model.

[0994] Practice and work submission process

[0995] Step 1:

[0996] The user practices calligraphy based on the displayed model. The user uses paper and pen to write the character 'beautiful'. The input for this step is the model data, and the output is the practice work.

[0997] Step 2:

[0998] The user takes a photo of the completed work with the smartphone camera. The input of this step is the practice work, and the output is image data.

[0999] Step 3:

[1000] The device sends the captured image data to the server. The image data is sent to the server via an HTTP POST request. The input of this step is the image data, and the output is a request sent to the server.

[1001] Step 4:

[1002] The server analyzes the received image. The server uses an AI algorithm (for example, a model using TensorFlow or PyTorch) to analyze the image and evaluate the brush strokes, balance of the characters, etc. The input for this step is the image data, and the output is the evaluation and corrections.

[1003] Step 5:

[1004] The server generates the evaluation and corrections and sends them to the terminal. The input of this step is the evaluation result, and the output is the data sent to the terminal.

[1005] Step 6:

[1006] The terminal displays the evaluation results to the user. The terminal displays the evaluation results and corrections on the user interface. The input of this step is the evaluation data, and the output is the display of feedback.

[1007] Emotion Recognition Process

[1008] Step 1:

[1009] As the user practices, the emotion engine analyzes the user's facial expressions and voice. The emotion engine collects data in real time using the device's built-in camera and microphone. The input for this step is the user's facial expressions and voice, and the output is emotion analysis data.

[1010] Step 2:

[1011] The emotion engine feeds back the analysis results to the server. The analysis results are sent to the server via an HTTP request. The input of this step is the emotion analysis data, and the output is a request sent to the server.

[1012] Step 3:

[1013] The server receives the feedback and generates an appropriate message for the user. Based on the emotion data, the server generates messages such as "You seem to be concentrating, keep it up" or "Maybe it's time to take a short break." The input of this step is the emotion feedback, and the output is the generated message.

[1014] Step 4:

[1015] The terminal displays the generated message to the user. The terminal displays the message as a notification popup or a detailed feedback screen. The input to this step is the generated message and the output is the display of the message.

[1016] Rank and grade acquisition process

[1017] Step 1:

[1018] The user submits a practice piece to the server. The user submits multiple practice pieces together to the server for evaluation. The input of this step is the practice piece data, and the output is a request sent to the server.

[1019] Step 2:

[1020] The server accumulates practice data and checks the conditions for obtaining a rank or grade. The server saves the user's practice data in a database and notifies them when certain criteria have been met. The input for this step is practice data, and the output is a notification that the conditions have been met.

[1021] Step 3:

[1022] The server sends a notification when the conditions for obtaining a rank or grade are met. The server notifies the user of the conditions that have been met. The input to this step is the notification of the condition being met, and the output is a notification message to the user.

[1023] Step 4:

[1024] The user submits a work that meets the criteria for final judging. The user submits additional work for final evaluation. The input of this step is the final work, and the output is a request to the server.

[1025] Step 5:

[1026] The server performs the final evaluation and determines the rank. The server evaluates the submitted works and determines the final rank. The input for this step is the final judged work, and the output is the evaluation results and the determination of the rank.

[1027] Remote teaching process

[1028] Step 1:

[1029] The user selects the instructor they want within the app. The user then selects the instructor they want to receive remote instruction from on the screen. The input for this step is the instructor's selection information, and the output is the selection result.

[1030] Step 2:

[1031] The server notifies the instructor and schedules the instruction. The server notifies the selected instructor and schedules the instruction. The input of this step is the instructor selection information, and the output is a schedule adjustment notification.

[1032] Step 3:

[1033] The user and instructor receive remote instruction via video call at a specified date and time. WebRTC is used for the video call. The input of this step is the connection information for the video call, and the output is the start of the video call.

[1034] Step 4:

[1035] The server records the video call in real time and saves the call for the user to review later. The input of this step is the video call data and the output is the recording data.

[1036] Step 5:

[1037] The emotion engine analyzes the user's facial expressions and voice and notifies the instructor of the results in real time, so that the instructor can adjust the content of instruction according to the situation. The input of this step is emotion analysis data, and the output is feedback notification to the instructor.

[1038] (Application example 2)

[1039] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1040] Conventional calligraphy learning systems have limited access to high-quality calligraphy education, particularly in the absence of remote feedback and instruction. They also lack effective support for users' learning motivation and progress. Furthermore, conventional systems struggle to provide feedback that reflects users' emotions, creating a need for methods to improve individual learning experiences.

[1041] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting characters to be written, means for analyzing the input character information and searching for model data, means for presenting the model data, means for capturing a work created by the user as an image, means for analyzing the captured image and performing scoring and corrections, means for presenting the scoring results and corrections, means for performing emotion analysis, and means for generating and presenting feedback based on the analyzed emotion information. This not only enables users to receive high-quality calligraphy education, but also enables them to study effectively from home, and the learning experience can be improved by individual feedback based on emotions.

[1042] The "means for inputting the characters you want to write" is an interface that allows the user to input the characters they want to learn or use as a model.

[1043] The "means for analyzing input character information and searching for model data" is a function that analyzes character information input by the user and searches for and retrieves the corresponding calligraphy model data from a database.

[1044] The "means for presenting model data" is a function for visually presenting the retrieved calligraphy model data to the user.

[1045] "Means for capturing a work created by a user as an image" refers to a function for capturing a work written by a user as a digital image using a device such as a camera or scanner.

[1046] The "means for analyzing imported images and grading and correcting" is a function that analyzes the imported image of the user's calligraphy work and grades and corrects it based on evaluation criteria such as brush strokes and balance.

[1047] The "means for presenting the scoring results and corrections" is a function for visually presenting the scoring results and corrections based on the analysis results to the user.

[1048] "Means for emotion analysis" refers to technology for identifying a user's emotional state by analyzing their facial expressions and tone of voice.

[1049] The "means for generating and presenting feedback based on the analyzed emotional information" is a function that generates a feedback message to improve the user's learning experience based on the results of the emotional analysis and presents it to the user.

[1050] "Means for notifying the conditions for obtaining a rank or grade" is a function for notifying the user of the conditions necessary to obtain a rank or grade.

[1051] "Means for evaluating works that meet the acquisition conditions" is a function for evaluating whether or not the conditions for obtaining a rank or grade are met.

[1052] The "means for determining rank and grade" is a function for determining the final rank and grade based on the user's practice work and evaluation results.

[1053] The "means for selecting an instructor from whom the user wishes to receive remote instruction" is an interface for the user to select an instructor from whom the user wishes to receive remote calligraphy instruction.

[1054] The "means for notifying the selected instructor and adjusting the schedule for instruction" is a function for sending a notification to the selected instructor and adjusting the schedule for instruction to suit the convenience of the user and the instructor.

[1055] The "means for providing instruction via video call" is a function that allows an instructor and a user to receive calligraphy instruction in real time via video call.

[1056] "Means for recording instruction content" is a function that allows you to record instruction content given via video calls, etc., so that you can review it later.

[1057] The "means for providing appropriate feedback according to the progress of instruction" is a function for performing emotion analysis and providing appropriate advice and encouraging messages to instructors and users according to the progress of instruction.

[1058] The present invention relates to a calligraphy learning support system. This system is designed to enable users who want to learn calligraphy to receive high-quality calligraphy instruction from home. In particular, by combining it with sentiment analysis technology, it can provide individually optimized feedback to improve the user's learning experience.

[1059] System configuration

[1060] The system consists of the following main components:

[1061] 1. User device (smartphone, tablet, or PC):

[1062] An interface for users to enter the characters they want to write.

[1063] Camera function for capturing images of calligraphy works.

[1064] Camera and microphone for sentiment analysis.

[1065] 2. Server:

[1066] A function that analyzes text information and searches for example data.

[1067] A function to send sample data to the user's terminal.

[1068] Runs AI algorithms that analyze user-uploaded images and provide scoring and correction.

[1069] A function that generates feedback based on the results of sentiment analysis.

[1070] 3. Emotion Engine:

[1071] Technologies that analyze facial expressions and tone of voice to identify a user's emotional state in real time (e.g., the DeepFace library).

[1072] 4. Interface:

[1073] An application for presenting exemplar data and evaluating user-created work.

[1074] An interface with video calling capabilities for remote instruction.

[1075] A UI that displays the scoring results and corrections.

[1076] Example of operation

[1077] 1. Example process:

[1078] The user inputs the characters they want to write into the device. For example, they input the character "beauty" into the input field. The device sends the input character information to the server, which analyzes it and searches for corresponding example data from a database. The search results are sent to the user's device, and the example data is presented. This allows the user to check high-quality examples.

[1079] 2. Practice and submission process:

[1080] Users practice calligraphy based on the provided model. They then take a photo of their work with their smartphone and import it into the device. The image data is sent to a server and analyzed by an AI algorithm. The calligraphy is scored and corrected based on factors such as brush strokes and character balance, and the results are sent back to the device. Users can use this feedback to improve their next practice.

[1081] 3. Emotion recognition process:

[1082] During practice, the emotion engine analyzes the user's facial expressions and voice to identify their emotional state in real time. The analysis results are sent to a server, which generates appropriate feedback based on the results. The feedback is displayed on the device to help maintain the user's motivation.

[1083] Specific technologies used

[1084] DeepFace: A library for sentiment analysis.

[1085] TextBlob: A library for generating feedback messages using natural language processing.

[1086] OpenCV: A library for image processing.

[1087] Specific examples

[1088] For example, if a user wants to learn the character "beauty," they enter "beauty" into the application. The device sends the character information to the server, and the server sends example data to the device. The user practices using the provided example, then takes a photo of the completed work with their smartphone and uploads it to the application. The server analyzes the image and evaluates and corrects it. The emotion engine analyzes the user's state of particular concentration or fatigue, and based on this generates messages such as "You're concentrating. Keep it up" or "It might be a good idea to take a short break."

[1089] Prompt Sentence Examples

[1090] For example, if you analyze emotions while practicing calligraphy and detect a "state of concentration," the prompt sentence is as follows:

[1091] User Sentiment: Concentration

[1092] Guidance message: Great focus! Keep it up!

[1093] As described above, this system is designed so that users can receive high-quality calligraphy education even from home and effectively progress in their learning. By utilizing sentiment analysis technology, the learning experience of users can be further enhanced.

[1094] The flow of the specific process in Application Example 2 will be described using FIG. 14.

[1095] Step 1:

[1096] The user inputs the character to be written.

[1097] Input: The user inputs characters such as "beautiful" into the terminal.

[1098] Process: The terminal acquires the input character information and transmits it to the server.

[1099] Output: The character information transmitted to the server.

[1100] Step 2:

[1101] Search for and display the model data.

[1102] Input: The character information transmitted to the server.

[1103] Process: The server analyzes the character information and searches for the corresponding model data from the model database.

[1104] Output: The searched model data is transmitted to the terminal and presented to the user.

[1105] Step 3:

[1106] The user practices calligraphy.

[1107] Input: The model data.

[1108] Process: The user uses paper and pen to practice calligraphy based on the presented model data.

[1109] Output: User-created work.

[1110] Step 4:

[1111] Take a photo of your work and import it into your device.

[1112] Input: A user-created calligraphy piece.

[1113] Processing: The user takes a photo of the work using the smartphone camera and imports it into the device as image data.

[1114] Output: The captured image data.

[1115] Step 5:

[1116] The image data is sent to the server.

[1117] Input: Image data of the work.

[1118] Processing: The image data captured by the terminal is sent to the server.

[1119] Output: Image data sent to the server.

[1120] Step 6:

[1121] Analyze image data and perform scoring and correction.

[1122] Input: Image data sent to the server.

[1123] Processing: The server uses AI algorithms to analyze the image, scoring and correcting it based on criteria such as brushstrokes and balance.

[1124] Output: Marking results and corrections.

[1125] Step 7:

[1126] The scoring results and corrections will be presented.

[1127] Input: Marking results and corrections.

[1128] Processing: The server sends the generated scoring results and corrections to the terminal and presents them to the user.

[1129] Output: The marks and corrections presented to the user.

[1130] Step 8:

[1131] Conduct sentiment analysis.

[1132] Input: User's facial expression and tone of voice.

[1133] Processing: The emotion engine analyzes the user's facial expressions and tone of voice to recognize their emotional state in real time.

[1134] Output: Parsed emotion information.

[1135] Step 9:

[1136] Generate and present feedback based on emotional information.

[1137] Input: Parsed emotion information.

[1138] Processing: The server generates an appropriate feedback message (e.g., "You're concentrating, so keep it up") based on the emotional information and sends it to the device.

[1139] Output: The feedback message presented to the user.

[1140] Step 10:

[1141] Users get feedback that helps them improve their practice next time.

[1142] Input: Feedback message.

[1143] Action: The user checks the feedback message provided and adjusts the content of the next practice based on it.

[1144] Output: Your next calligraphy piece based on your improved practice.

[1145] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1146] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1147] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1148] [Third embodiment]

[1149] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1150] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1151] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1152] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1153] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1154] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1155] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1156] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1157] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1158] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1159] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1160] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1161] The present invention relates to a calligraphy learning support system that enables users who wish to learn calligraphy to receive high-quality calligraphy instruction and enables remote calligraphy instruction using artificial intelligence (AI).

[1162] System configuration

[1163] This system consists of the following main elements:

[1164] 1. User device: Tablet, smartphone, or PC. Users use these devices to study calligraphy.

[1165] 2. Server: The core part of the system installed in the center. It provides a database of calligraphy models, image analysis by AI, and grading and correction functions.

[1166] 3. Interface: Software or applications for the user and the system to interact with each other.

[1167] Program processing

[1168] Model presentation process

[1169] 1. The user inputs the characters they want to write on the terminal. For example, input the character "美" into the input field of the application.

[1170] 2. The terminal sends the input character information to the server.

[1171] 3. The server receives the character information, analyzes it, and searches for the corresponding model data from the database.

[1172] 4. The server sends the search results to the terminal.

[1173] 5. The terminal displays the sent model data and presents it to the user. In this way, the user can view the model while staying at home.

[1174] Practice and work submission process

[1175] 1. The user practices calligraphy based on the displayed model. For example, use paper and pen to write the character "美".

[1176] 2. When the practice work is completed, the user takes a photo of the work as an image and imports it into the terminal. The camera function of a smartphone is often used.

[1177] 3. The terminal sends the captured image data to the server.

[1178] 4. The server receives the image and analyzes it using an AI algorithm, scoring it based on multiple criteria, including brush strokes and balance of the characters.

[1179] 5. The server generates the scoring results and corrections and sends them to the terminal.

[1180] 6. The device displays the received results to the user, allowing the user to receive feedback and use it for further practice.

[1181] Optional Feature Process

[1182] 1. Dan / Kyu rank acquisition:

[1183] A user submits multiple practice pieces to the server and receives a certain evaluation.

[1184] The server accumulates the user's practice data and notifies them when they meet the requirements for obtaining a rank or grade.

[1185] Users submit works that meet the requirements, and the server performs a final evaluation to determine rank and grade.

[1186] 2. Remote instruction:

[1187] The user selects the calligraphy instructor from whom they would like to receive remote instruction within the app.

[1188] The server notifies the selected instructor and schedules the instruction.

[1189] The user and the calligraphy instructor will conduct remote instruction via video call.

[1190] The server records the instruction content so that the user can review it later.

[1191] Specific examples

[1192] For example, if a user wants to practice the character "美", first enter "美" into the application. The terminal sends this information to the server, and the server retrieves the corresponding model data from the database and sends it to the terminal. The user practices while looking at the presented model, takes a photo of the completed work with a smartphone, and uploads it to the terminal. The server analyzes the image, sends the results of evaluation and correction to the user, and the user can use it for the next practice.

[1193] In addition, users aiming to obtain ranks or grades can submit works that meet the conditions upon receiving a notification from the server, and finally be evaluated to obtain ranks or grades. Furthermore, if external guidance is required, users can select a desired instructor within the application and receive remote guidance. Through this series of processes, it is realized that high-quality calligraphy education can be received even from home.

[1194] The following describes the processing flow.

[1195] Model Presentation Process

[1196] Step 1:

[1197] The user inputs the character to be written on the terminal.

[1198] Specifically, the user opens the application and inputs the character to be written, such as "美", into the text input field.

[1199] Step 2:

[1200] The terminal sends the input character information to the server.

[1201] Specifically, the terminal issues a request to send the character data input by the user to the server through the API.

[1202] Step 3:

[1203] The server receives the text information, analyzes it, and searches the database for corresponding example data.

[1204] Specifically, the server receives the request and uses a character recognition module to query the database and retrieve the corresponding example data.

[1205] Step 4:

[1206] The server sends the search results to the terminal.

[1207] Specifically, the server transmits the acquired sample data to the terminal as a response.

[1208] Step 5:

[1209] The terminal displays the transmitted example data and presents it to the user.

[1210] Specifically, the terminal receives the response and displays the example image within the application.

[1211] Practice and submission process

[1212] Step 1:

[1213] The user practices calligraphy based on the displayed example.

[1214] Specifically, the user uses paper and a brush to write characters according to the provided example.

[1215] Step 2:

[1216] Once the practice piece is complete, the user takes a picture of it and imports it into their device.

[1217] Specifically, users take a photo of the work using their smartphone camera and upload the image to the app.

[1218] Step 3:

[1219] The terminal transmits the captured image data to the server.

[1220] Specifically, the terminal issues a request to upload image data to the server via the API.

[1221] Step 4:

[1222] The server receives the images and analyzes them using AI algorithms.

[1223] Specifically, the server passes the image data to an analysis engine, which evaluates it based on multiple criteria, such as brush strokes and character balance.

[1224] Step 5:

[1225] The server generates the scoring results and corrections and sends them to the terminal.

[1226] Specifically, the server calculates a score based on the analysis results, generates feedback including points for improvement, and sends it to the device.

[1227] Step 6:

[1228] The terminal displays the received results to the user.

[1229] Specifically, the device receives a response from the server and displays the scoring results and corrections within the app.

[1230] Optional Feature Process

[1231] Obtaining Dan / Kyu rank

[1232] Step 1:

[1233] A user submits multiple practice pieces to the server and receives a certain evaluation.

[1234] Specifically, users continue to upload their work to the app and accumulate ratings.

[1235] Step 2:

[1236] The server accumulates the user's practice data and notifies them when they meet the requirements for obtaining a rank or grade.

[1237] Specifically, the server checks the user's evaluation log and determines whether the acquisition conditions are met. If the conditions are met, the server notifies the user.

[1238] Step 3:

[1239] Users submit works that meet the requirements, and the server performs a final evaluation to determine rank and grade.

[1240] Specifically, users submit their work to be evaluated through a specified procedure, and the server performs a final evaluation to determine the rank and grade.

[1241] Remote instruction

[1242] Step 1:

[1243] The user selects the calligraphy instructor from whom they would like to receive remote instruction within the app.

[1244] Specifically, the user selects the desired instructor from a list of instructors in the app.

[1245] Step 2:

[1246] The server notifies the selected instructor and schedules the instruction.

[1247] Specifically, the server sends requests to instructors and provides an interface for scheduling.

[1248] Step 3:

[1249] The user and the calligraphy instructor will conduct remote instruction via video call.

[1250] Specifically, the user and instructor will provide instruction via a video call app at a specified time.

[1251] Step 4:

[1252] The server records the instruction content so that the user can review it later.

[1253] Specifically, the server stores recorded data of video calls and makes it available for users to review if necessary.

[1254] Example 1

[1255] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1256] Conventional calligraphy education systems require face-to-face instruction and practice using paper and brush in order for users to receive high-quality calligraphy training, which can be difficult for users living in remote locations. There are also issues with the lack of prompt feedback on the work users have practiced and evaluation of their ranks and grades. Furthermore, there is a lack of a flexible system that allows for real-time remote instruction. To address these issues, an effective and efficient calligraphy learning support system is needed.

[1257] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1258] In this invention, the server includes means for a user to input characters that the user wants to write, means for analyzing the input character information and searching for model data, means for presenting the searched model data, means for capturing an image of a work created by the user, means for analyzing the captured image using an artificial intelligence algorithm and grading and correcting the image, means for presenting the grading results and corrections to the user, means for notifying the user of the conditions for obtaining a rank or grade, means for evaluating works that satisfy the conditions, means for determining the rank or grade, means for the user to select an instructor from whom they wish to receive remote instruction, means for notifying the selected instructor and arranging a schedule for instruction, means for providing instruction via video call, and means for recording the content of instruction. This enables users to receive high-quality calligraphy education from the comfort of their own homes, receive prompt feedback and evaluation, and receive real-time remote instruction.

[1259] "User" refers to a user of the system who intends to receive calligraphy education.

[1260] A "terminal" is a device that allows a user to use the system, and includes a tablet, smartphone, or PC.

[1261] The "server" is the central computer of the system, and is a device that provides functions for searching sample data, analyzing images, and scoring and correcting.

[1262] "Model data" is calligraphy sample data that is presented as a standard for the characters that the user wants to learn.

[1263] "Means for capturing images" refers to devices and functions necessary for inputting a user's work into the system as a digital image, and includes camera functions, etc.

[1264] An "artificial intelligence algorithm" is a program that runs on a server and analyzes, grades, and corrects works created by users.

[1265] "Scoring result" is the evaluation score given to the user's work by the artificial intelligence algorithm.

[1266] "Correction content" is feedback information indicating corrections and improvements provided to the user's work.

[1267] "Rank / Grade" refers to the qualification grade that indicates the level of the user's calligraphy skills.

[1268] "Acquisition conditions" are the requirements that a user must meet in order to acquire a rank or grade.

[1269] The "instructor who wishes to receive remote instruction" is a calligraphy instructor that the user selects within the system to receive calligraphy instruction via video call.

[1270] "Video calling" is a means of communication that allows for the exchange of video and audio in real time over the Internet.

[1271] "Means for recording instruction content" refers to the ability to digitally save instruction given during a video call.

[1272] This invention is a calligraphy learning support system that allows users to receive high-quality calligraphy training. The system mainly consists of a user terminal, a server, and a user interface.

[1273] System configuration

[1274] 1. User device: A device such as a tablet, smartphone, or PC that users use to learn calligraphy.

[1275] 2. Server: This is the core device of the system, providing a database of calligraphy samples, image analysis using AI, and scoring and correction functions.

[1276] 3. User Interface: Software or applications that allow users to interact with a system.

[1277] Model Prompt

[1278] 1. The user enters the characters they want to write in the input field of the application. For example, enter the character "美".

[1279] 2. The terminal sends the entered character information to the server. Specifically, use an HTTP request to send the character information to the server.

[1280] 3. The server receives the character information, analyzes it, and searches for the corresponding model data from the database. The server executes an analysis script written in Python and uses MySQL for the database.

[1281] 4. The server sends the search results to the terminal, and the terminal displays the received model data on the application.

[1282] Practice and Submission of Works

[1283] The user practices calligraphy referring to the model. Using paper and pen, for example, write the character "美". When the practice work is completed, the user uses the camera function of the smartphone to take a photo of the work and import it into the terminal as an image. The terminal sends the captured image data to the server via an HTTP POST request.

[1284] The server receives the image data and performs analysis using an AI algorithm based on TensorFlow. Evaluate aspects such as the stroke movement and the balance of the characters, and generate a scoring result and correction content. The generated data is in JSON format, and the server sends this to the terminal. Then, the terminal displays the received results on the application. The evaluation and correction content are visually displayed using HTML and CSS.

[1285] Obtaining Dan and Ji Grades

[1286] The user sends multiple practice works to the server and receives an evaluation. The server accumulates the user's practice data in the database and sends a notification when the conditions for obtaining a rank or grade are met. Resend the works that meet the conditions for the final evaluation, and the server finally determines the rank or grade and sends the evaluation result to the terminal.

[1287] Remote guidance

[1288] When the user wishes to receive remote guidance within the application, select an instructor using the interface within the system. The server notifies the selected instructor and adjusts the guidance schedule. The WebRTC technology is used for video calls to enable real-time guidance. The content of the guidance is saved in cloud storage and recorded using Amazon S3.

[1289] Specific example

[1290] For example, if the user wants to practice the character "美", enter "美" in the application. The terminal sends this information to the server, and the server retrieves the corresponding model data from the database and sends it to the terminal. The user practices while looking at the presented model, takes a photo of the completed work with a smartphone, and uploads it to the terminal. The server analyzes the image, sends the results of the evaluation and correction to the user, and can be used for the next practice. Also, when obtaining a rank or grade, submit multiple works for evaluation, receive a notification when the criteria are met, and it is possible to obtain a rank or grade.

[1291] Examples of prompt sentences

[1292] Please display a model for writing the character "美".

[1293] Search for and display the model of the character "美" entered in the database. Also, please take a photo of the user's practice work and evaluate it.

[1294] The flow of the specific process in Example 1 will be described using FIG. 11.

[1295] Step 1:

[1296] The user inputs the characters they want to write in the input field of the application. The data to be input is Chinese characters such as "beautiful". The user inputs characters in the input field and performs the operation of pressing the send button.

[1297] Step 2:

[1298] The terminal sends the input character information to the server. The input is the character information entered by the user, and the output is an HTTP request to the server. The data is sent in JSON format. At this time, the terminal performs the operation of sending a request to the server using network communication.

[1299] Step 3:

[1300] The server receives the character information and analyzes the characters. The input is the character information sent from the terminal, and the output is the character code as the analysis result. The server executes a Python script and performs the operation of analyzing the specified characters.

[1301] Step 4:

[1302] The server searches for the corresponding model data from the database based on the character code. The input is the analyzed character code, and the output is the model data. The server executes an SQL query against MySQL and performs the operation of obtaining the relevant model data.

[1303] Step 5:

[1304] The server sends the search results to the terminal in JSON format. The input is the model data obtained from the database, and the output is an HTTP response to the terminal. The server converts the obtained data into JSON format and performs the operation of sending it as an HTTP response.

[1305] Step 6:

[1306] The example data received by the device is displayed on the application. The input is the example data sent from the server, and the output is the example displayed on the user interface. The device uses JavaScript to draw the example data.

[1307] Step 7:

[1308] The user practices calligraphy based on the displayed example. Using paper and a brush, the user writes the character "beauty."

[1309] Step 8:

[1310] Once the practice piece is complete, the user takes a photo of the piece using the smartphone's camera function and imports the image into the device. The input is the image of the completed piece, and the output is a digital image saved on the device. The user uses the camera function to acquire the image.

[1311] Step 9:

[1312] The device sends the captured image data to the server using an HTTP POST request. The input is the image data captured by the user, and the output is a request to the server. The device uses network communication to send the image data to the server.

[1313] Step 10:

[1314] The server receives the image data and analyzes it using an AI algorithm. The input is the transmitted image data, and the output is the evaluation data as the analysis result. The server uses TensorFlow to analyze the image and evaluate the brush strokes and balance of the characters.

[1315] Step 11:

[1316] The server generates the scoring results and corrections and sends them to the device. The input is the analysis result, and the output is the evaluation results and corrections sent to the device. The server converts the evaluation data into JSON format and sends it as an HTTP response.

[1317] Step 12:

[1318] The terminal displays the results it receives on the application. The input is the evaluation results and corrections sent from the server, and the output is the feedback displayed on the user interface. The terminal uses HTML and CSS to visually display the results.

[1319] Step 13:

[1320] The user performs further practice based on the provided feedback. The input is the displayed feedback, and the output is an improved practice piece. The user then uses the feedback to perform the writing action again.

[1321] (Application example 1)

[1322] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1323] Conventional calligraphy learning systems provide paper-based examples and feedback, making real-time instruction and evaluation difficult. Furthermore, due to limited learning support in remote locations, students must attend a calligraphy class in order to receive high-quality calligraphy education. Furthermore, there are currently insufficient methods for improving the efficiency of calligraphy learning using smart devices.

[1324] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1325] In this invention, the server includes means for inputting characters to be written, means for analyzing the input character information and searching for model data, means for presenting the model data, means for capturing a work created by the user as an image, means for analyzing the captured image and performing scoring and corrections, means for presenting the scoring results and corrections, means for displaying the model data using smart glasses, means for photographing the created work using the camera function of the smart glasses, means for transmitting the image data to a cloud server and analyzing it using an AI algorithm, and means for displaying the analysis results and feedback on the smart glasses. This enables real-time feedback, enables high-quality calligraphy education even from remote locations, and realizes efficient calligraphy learning by utilizing smart devices.

[1326] The "means for inputting characters to be written" is an interface for inputting characters to be written into a device when a user is learning calligraphy.

[1327] The "means for analyzing input character information and searching for model data" is a function that analyzes character information input by the user and searches for corresponding calligraphy model data from a database.

[1328] The "means for presenting example data" is a function for displaying the retrieved example data on the display of the terminal so that the user can confirm it.

[1329] The "means for capturing a work created by a user as an image" is a function for capturing a calligraphy work that a user has written on paper or the like as a digital image using a camera or the like.

[1330] The "means for analyzing the imported image and scoring and correcting it" refers to an AI algorithm that analyzes the imported image of the calligraphy work and automatically scores and corrects it.

[1331] The "means for presenting the scoring results and corrections" is a function for displaying the analysis results and corrections in an easy-to-understand manner to the user.

[1332] "Means for displaying example data using smart glasses" refers to a function that uses the display of the smart glasses to virtually display example data within the user's field of view.

[1333] "Means for photographing a work created using the camera function of the smart glasses" refers to a function for photographing a work written by a user using the camera built into the smart glasses.

[1334] "Means for sending image data to a cloud server and analyzing it using an AI algorithm" refers to a function that uploads image data of calligraphy works sent from smart glasses or other devices to a server on the cloud and uses an AI algorithm to analyze the data.

[1335] "Means for displaying analysis results and feedback on the smart glasses" is a function for displaying the analysis results and feedback returned from the cloud server on the display of the smart glasses.

[1336] This invention is a system that utilizes smart glasses to provide real-time calligraphy learning support. This system allows users who want to learn calligraphy to receive high-quality calligraphy education using smart glasses and a cloud server.

[1337] System Configuration

[1338] 1. User Device

[1339] Smart glasses (e.g., Google Glass) are used as the user terminal. This device has the function of displaying the sample data, taking a photo of the written characters, and sending the photo to a cloud server.

[1340] 2. Cloud Server

[1341] The cloud server provides a database of calligraphy models, image analysis by AI, and a grading and correction function. It functions as an infrastructure for real-time data processing and analysis.

[1342] 3. Interface

[1343] An interface is provided for the user to interact with the system through the display, operation panel, and voice recognition function of the smart glasses.

[1344] Outline of Program Execution

[1345] Model Presentation Process

[1346] The server provides a means for the user to input the characters they want to write through voice or touchpad operations using the smart glasses. For example, when the user inputs the character "beautiful", the server analyzes the character information and searches the database for the corresponding model data. The model data is displayed on the smart glasses, and the user can practice calligraphy while viewing it in real time.

[1347] Practice and Work Submission Process

[1348] The user practices calligraphy while viewing the model displayed through the smart glasses. After practice, the user uses the camera function of the smart glasses to take a picture of the work and send the image data to the cloud server. The cloud server analyzes the image using an AI algorithm and evaluates aspects such as the stroke movement and character balance. The analysis results are displayed on the smart glasses together with the calligraphy correction content, and the user can obtain real-time feedback.

[1349] Hardware and Software to be Used

[1350] This system operates with smart glasses, a cloud server, and an AI analysis model (using, for example, TensorFlow) as its main components. The smart glasses function as a user interface, and the cloud server provides data processing and storage solutions. An image analysis model is used for AI analysis to conduct a detailed evaluation of the user's calligraphy works.

[1351] Examples of specific cases and prompt sentences

[1352] For example, if the user wants to practice the character "beautiful", first, the user inputs the command "display the model" to the smart glasses by voice or touchpad operation. The following are examples of prompt sentences.

[1353] Prompt sentence: "Please input the model character."

[1354] When the user practices calligraphy and the character is completed, next, the user activates the camera using the gesture function of the smart glasses and takes a photo of the work. The following are examples of prompt sentences.

[1355] Prompt sentence: "When the character is completed, please activate the camera and take a photo."

[1356] The captured image data is sent to the cloud server, and the AI model conducts analysis and displays the results and feedback on the display of the smart glasses. When the analysis is completed, the following prompt sentence is displayed.

[1357] Prompt sentence: "The analysis is completed. Please check the feedback."

[1358] In this way, a system is provided that enables the user to practice calligraphy and receive feedback in real time.

[1359] The flow of specific processing in Application Example 1 will be described using FIG. 12.

[1360] Step 1:

[1361] The user wears smart glasses and inputs the characters they want to write. The input method can be voice recognition or touchpad operation. When the user inputs "beautiful" in response to the prompt "Please input the model characters", the input character information is sent by the smart glasses to the cloud server. The server analyzes the received character information and searches for the corresponding model data from the model database.

[1362] Step 2:

[1363] The server sends the model data to the smart glasses, and the smart glasses display the data in the user's field of vision. Thereby, the user can start practicing calligraphy while checking the model in real time. The display of the model data is projected onto the display of the smart glasses, and the user writes characters on paper while looking at the model.

[1364] Step 3:

[1365] After the user finishes writing the characters referring to the model, the user uses the camera function of the smart glasses to take a picture of the work. The user follows the prompt "Please activate the camera and take a picture when the characters are completed" using gesture operation or the touchpad to take a picture of the work. The captured image data is sent by the smart glasses to the cloud server.

[1366] Step 4:

[1367] The server analyzes the received image data using an AI algorithm. As data processing, preprocessing of the image and feature extraction are performed, and analysis is carried out using evaluation criteria such as the movement of the pen, the balance of the characters, and the thickness of the lines. For this analysis, a generative AI model such as TensorFlow is used. As a result of the analysis, scores and detailed correction contents are generated.

[1368] Step 5:

[1369] The server sends the generated analysis results and feedback to the smart glasses, which then display the received results in the user's field of view. Following the prompt "Analysis completed. Please check your feedback," feedback is presented to the user, allowing the user to use this information to improve their practice next time.

[1370] Step 6:

[1371] When a user practices multiple times and meets certain conditions, the conditions for obtaining a rank or grade are notified to the smart glasses. The server accumulates the user's practice data and notifies the user when the conditions are met. The user then photographs and submits works that meet the conditions, and the server performs a final evaluation to determine the rank or grade.

[1372] Step 7:

[1373] When a user requests remote instruction, they select an instructor through the smart glasses interface. The server then notifies the instructor and schedules the instruction. The instructor provides real-time instruction to the user through a video call function. The instruction is then recorded on the server for later review by the user.

[1374] Through this series of steps, the calligraphy learning support system, which utilizes smart glasses and a cloud server, provides users with an efficient and high-quality learning environment.

[1375] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1376] The present invention relates to a calligraphy learning support system. This system is for users who want to learn calligraphy to receive high-quality calligraphy education and enables remote calligraphy instruction using artificial intelligence (AI). Additionally, by combining an emotion engine that analyzes the emotions of users, the learning experience of users can be further improved.

[1377] System Configuration

[1378] This system consists of the following main elements.

[1379] 1. User terminal: Such as a tablet, smartphone, or personal computer. The user uses these terminals to conduct calligraphy learning.

[1380] 2. Server: The core part of the system installed centrally. It provides a database of calligraphy models, image analysis by AI, and a grading and correction function.

[1381] 3. Interface: Software or applications for the user and the system to interact with each other.

[1382] 4. Emotion engine: A device or software that analyzes the user's facial expressions and voice tones to recognize the emotional state.

[1383] Program Processing

[1384] Model Presentation Process

[1385] 1. The user inputs the character they want to write into the terminal. For example, input the character "beautiful" into the input field of the application.

[1386] 2. The terminal sends the input character information to the server.

[1387] 3. The server receives the character information, analyzes it, and searches for the corresponding model data from the database.

[1388] 4. The server sends the search results to the device.

[1389] 5. The device displays the example data and presents it to the user, allowing the user to check the example from the comfort of their own home.

[1390] Practice and submission process

[1391] 1. The user practices calligraphy based on the displayed model, for example, writing the character "beauty" using paper and a brush.

[1392] 2. Once the practice piece is complete, the user takes a photo of it and imports it into their device, often using the camera function on their smartphone.

[1393] 3. The device sends the captured image data to the server.

[1394] 4. The server receives the image and analyzes it using an AI algorithm, scoring it based on multiple criteria, including brush strokes and balance of the characters.

[1395] 5. The server generates the scoring results and corrections and sends them to the terminal.

[1396] 6. The device displays the received results to the user, allowing the user to receive feedback and use it for further practice.

[1397] Emotion Recognition Process

[1398] 1. As the user practices calligraphy, the emotion engine analyzes the user's facial expressions and voice to recognize their emotional state in real time.

[1399] 2. The emotion engine provides feedback to the server based on the emotional state.

[1400] 3. The server receives information from the emotion engine, checks the progress of practice and motivation, and generates appropriate advice and encouraging messages.

[1401] 4. The device displays these messages to the user to help them stay motivated.

[1402] Optional Feature Process

[1403] Obtaining Dan / Kyu rank

[1404] 1. A user submits multiple practice pieces to the server and receives a certain evaluation.

[1405] 2. The server accumulates the user's practice data and notifies the user when the conditions for obtaining a rank or grade are met, allowing the user to check their own progress.

[1406] 3. The user submits works that meet the requirements, and the server performs a final evaluation to determine the rank and grade.

[1407] Remote instruction

[1408] 1. The user selects the calligraphy instructor with whom they would like to receive remote instruction within the app.

[1409] 2. The server notifies the selected instructor and schedules the instruction.

[1410] 3. The user and the calligraphy instructor will conduct remote instruction via video call.

[1411] 4. The server records the instruction so that the user can review it later.

[1412] 5. The emotion engine analyzes the user's emotional state during training and notifies the instructor of the results in real time, allowing the instructor to provide instruction that is appropriate for the user's emotional state.

[1413] Specific examples

[1414] For example, if a user wants to practice the character "美", first enter "美" into the application. The terminal sends this information to the server, and the server retrieves the corresponding model data from the database and sends it to the terminal. The user practices while looking at the presented model, takes a photo of the completed work with a smartphone, and uploads it to the terminal. The server analyzes the image, performs evaluation and correction, and sends the results to the user, who can use them for the next practice.

[1415] Furthermore, while the user is practicing, the emotion engine analyzes the user's expression and feedbacks to the server the state of concentration or fatigue. Based on this, the server can send messages to the user such as "You seem to be concentrating. Keep it up like this" or "It might be good to take a little break".

[1416] Users aiming to obtain ranks or grades receive notifications from the server, submit works that meet the conditions, and can finally be evaluated to obtain ranks or grades. Furthermore, if external guidance is required, the user can select a desired instructor within the application and receive remote guidance. Through this series of processes, it is realized that high-quality calligraphy education can be received even from home. By using the emotion engine, more effective learning becomes possible while maintaining the user's motivation and concentration.

[1417] The following describes the processing flow.

[1418] Model Presentation Process

[1419] Step 1:

[1420] [[ID=Z7]]The user inputs the character to be written on the terminal.

[1421] Specifically, the user opens the application and inputs the character to be written, such as "美", into the text input field.

[1422] Step 2:

[1423] The terminal transmits the input character information to the server.

[1424] Specifically, the terminal issues a request to send the character data entered by the user to the server via the API.

[1425] Step 3:

[1426] The server receives the text information, analyzes it, and searches the database for corresponding example data.

[1427] Specifically, the server receives the request and uses a character recognition module to query the database and retrieve the corresponding example data.

[1428] Step 4:

[1429] The server sends the search results to the terminal.

[1430] Specifically, the server transmits the acquired sample data to the terminal as a response.

[1431] Step 5:

[1432] The terminal displays the transmitted example data and presents it to the user.

[1433] Specifically, the terminal receives the response and displays the example image within the application.

[1434] Practice and submission process

[1435] Step 1:

[1436] The user practices calligraphy based on the displayed example.

[1437] Specifically, the user uses paper and a brush to write characters according to the provided example.

[1438] Step 2:

[1439] Once the practice piece is complete, the user takes a picture of it and imports it into their device.

[1440] Specifically, users take a photo of the work using their smartphone camera and upload the image to the app.

[1441] Step 3:

[1442] The terminal transmits the captured image data to the server.

[1443] Specifically, the terminal issues a request to upload image data to the server via the API.

[1444] Step 4:

[1445] The server receives the images and analyzes them using AI algorithms.

[1446] Specifically, the server passes the image data to an analysis engine, which evaluates it based on multiple criteria, such as brush strokes and character balance.

[1447] Step 5:

[1448] The server generates the scoring results and corrections and sends them to the terminal.

[1449] Specifically, the server calculates a score based on the analysis results, generates feedback including points for improvement, and sends it to the device.

[1450] Step 6:

[1451] The terminal displays the received results to the user.

[1452] Specifically, the device receives a response from the server and displays the scoring results and corrections within the app.

[1453] Emotion Recognition Process

[1454] Step 1:

[1455] As the user practices calligraphy, the emotion engine analyzes the user's facial expressions and voice to recognize their emotional state in real time.

[1456] Specifically, the emotion engine collects and analyzes the user's facial expressions and tone of voice through the device's camera and microphone.

[1457] Step 2:

[1458] An emotional engine provides feedback to the server based on the emotional state.

[1459] Specifically, the emotion engine analyzes the user's current emotional state (e.g., focused, tired) and sends that data to the server.

[1460] Step 3:

[1461] The server receives information from the emotion engine, checks the progress of practice and motivation, and generates appropriate advice and encouraging messages.

[1462] Specifically, the server generates messages based on emotional data, such as "You seem to be concentrating, keep it up" or "Maybe it would be good to take a short break."

[1463] Step 4:

[1464] The device displays these messages to the user to help them stay motivated.

[1465] Specifically, the terminal displays the message sent from the server on the screen.

[1466] Optional Feature Process

[1467] Obtaining Dan / Kyu rank

[1468] Step 1:

[1469] A user submits multiple practice pieces to the server and receives a certain evaluation.

[1470] Specifically, users continue to upload their work to the app and accumulate ratings.

[1471] Step 2:

[1472] The server accumulates the user's practice data and notifies them when they meet the requirements for obtaining a rank or grade.

[1473] Specifically, the server checks the user's evaluation log and determines whether the acquisition conditions are met. If the conditions are met, the server notifies the user.

[1474] Step 3:

[1475] Users submit works that meet the requirements, and the server performs a final evaluation to determine rank and grade.

[1476] Specifically, users submit their work to be evaluated through a specified procedure, and the server performs a final evaluation to determine the rank and grade.

[1477] Remote instruction

[1478] Step 1:

[1479] The user selects the calligraphy instructor from whom they would like to receive remote instruction within the app.

[1480] Specifically, the user selects the desired instructor from a list of instructors in the app.

[1481] Step 2:

[1482] The server notifies the selected instructor and schedules the instruction.

[1483] Specifically, the server sends requests to instructors and provides an interface for scheduling.

[1484] Step 3:

[1485] The user and the calligraphy instructor will conduct remote instruction via video call.

[1486] Specifically, the user and instructor will provide instruction via a video call app at a specified time.

[1487] Step 4:

[1488] The server records the instruction content so that the user can review it later.

[1489] Specifically, the server stores recorded data of video calls and makes it available for users to review if necessary.

[1490] Step 5:

[1491] The emotion engine analyzes the user's emotional state during instruction and notifies the instructor of the results in real time.

[1492] Specifically, the emotion engine sends the results of analyzing the user's facial expressions and voice to the server, which then displays the results on the instructor's screen.

[1493] Specific examples

[1494] For example, if a user wants to practice the character "美" (beautiful), first the user inputs "美" into the application. The terminal sends this information to the server, and the server retrieves the corresponding model data from the database and sends it to the terminal. The user practices while looking at the presented model, takes a photo of the completed work with a smartphone, and uploads it to the terminal. The server analyzes the image, conducts evaluation and correction, and sends the results to the user, who can use them for the next practice.

[1495] Furthermore, while the user is practicing, the emotion engine analyzes the user's expression and provides feedback to the server on whether the user is in a concentrated state or a tired state. Based on this, the server can send messages to the user such as "You seem to be concentrating. Keep it up like this" or "It might be good to take a short break."

[1496] Users aiming to obtain ranks or grades receive notifications from the server, submit works that meet the conditions, and can finally be evaluated to obtain ranks or grades. Furthermore, if external guidance is needed, the user can select a desired instructor within the application and receive remote guidance. Through this series of processes, it is realized that high-quality calligraphy education can be received even from home. By using the emotion engine, more effective learning is possible while maintaining the user's motivation and concentration.

[1497] (Example 2)

[1498] Next, Example 2 will be described. In the following description, the data processing device 12 is referred to as the "server", and the headset-type terminal 314 is referred to as the "terminal".

[1499] In a conventional calligraphy education system, it is difficult for users to receive high-quality feedback, especially there are limitations in guidance from remote locations. Also, due to the lack of means to appropriately manage the user's emotions and motivation, efficient learning has been difficult. The process for users to obtain ranks or grades is also inefficient, and it requires a lot of time and effort to acquire advanced calligraphy techniques.

[1500] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1501] In this invention, the server includes a means for inputting characters to be written, a means for analyzing the input character information and searching for model data, a means for presenting the model data, a means for capturing the work created by the user as an image, a means for analyzing the captured image and performing scoring and corrections, a means for presenting the scoring results and corrections, a means for analyzing the user's facial expressions and voice to recognize emotions, and a means for providing feedback based on the emotion recognition results. This allows for high-quality calligraphy education even in a remote environment, and enables the user's emotions to be analyzed in real time and appropriate feedback to be provided. Furthermore, since ranks and grades can be acquired efficiently, users can more effectively acquire calligraphy techniques.

[1502] The "means for inputting characters to be written" refers to an input field or operation means for the user to input characters to be written in calligraphy practice into the application.

[1503] The "means for analyzing input character information and searching for example data" refers to an algorithm or program for receiving character information input by a user, analyzing it, and searching for corresponding example data from a database.

[1504] The "means for presenting model data" refers to a display device or software for receiving the model data sent from the server and displaying it on the user's terminal.

[1505] The "means for capturing an image of a work created by a user" refers to a camera function or an image input device that allows a user to take an image of a calligraphy work and capture it into the terminal.

[1506] The "means of analyzing the captured image and grading and correcting" refers to an AI algorithm or analysis program that analyzes the captured image, evaluates the brush strokes and balance of the characters, and performs grading and correction.

[1507] The "means for presenting the marking results and corrections" refers to a UI component or notification system for displaying the marking results and corrections generated by the server to the user.

[1508] "Means for recognizing emotions by analyzing the user's facial expressions and voice" refers to facial recognition technology and voice analysis technology that analyze the user's facial expressions and tone of voice to understand their current emotional state.

[1509] The "means for providing feedback based on emotion recognition results" refers to software or a system for generating appropriate advice or encouraging messages based on the user's emotion recognition results and providing them to the user.

[1510] "Means for notifying the conditions for obtaining a rank or grade" refers to algorithms and communication methods for confirming the conditions necessary for a user to obtain a rank or grade and notifying the user of the status of achievement.

[1511] The "means for evaluating works that satisfy the acquisition conditions" refers to an examination algorithm or evaluation system that allows users to submit works that satisfy the conditions for acquiring a rank or grade and evaluates them.

[1512] The "means for determining ranks and grades" refers to a judging system for determining final ranks and grades based on the evaluation results of works submitted by users.

[1513] The "means for selecting an instructor from whom the user wishes to receive remote instruction" refers to a selection screen or system that allows the user to select an instructor from whom the user wishes to receive remote instruction within the application.

[1514] The "means for notifying the selected instructor and arranging the schedule for instruction" refers to a scheduling system or communication means for sending notification to the selected instructor and arranging the schedule for instruction.

[1515] The "means for providing instruction via video call" refers to a video call system or software that allows a user and an instructor to provide instruction remotely via video call.

[1516] The "means for recording the instruction content" refers to a recording device or software that records the instruction content given during the video call in real time so that the user can review it later.

[1517] "Means for analyzing the user's emotions during training and notifying the instructor" refers to an analysis system or communication means for analyzing the user's facial expressions and voice during a video call and notifying the instructor of their emotional state in real time.

[1518] This invention is a system for providing high-quality calligraphy education, designed to enable users to effectively learn calligraphy even in remote environments. This system is composed of a user terminal, a server, an interface, and an emotion engine.

[1519] System configuration

[1520] 1. User Device: Calligraphy lessons are conducted using devices such as tablets, smartphones, or PCs. These devices have built-in cameras that can be used to take photos of the works created by users.

[1521] 2. Server: This is the core part of the system, providing the database of calligraphy examples, AI image analysis, and scoring and correction functions. Specifically, AI frameworks such as TensorFlow and PyTorch are used for image analysis.

[1522] 3. Interface: The application or software that allows users to interact with the system. Through this interface, users can view calligraphy examples, upload practice pieces, and receive feedback.

[1523] 4. Emotion Engine: Technology that analyzes the user's facial expressions and tone of voice to recognize their emotional state. For example, OpenCV is used for facial recognition, and Google Cloud Speech-to-Text API is used for voice analysis.

[1524] Program processing

[1525] The program of this system performs the following processing.

[1526] Modeling process

[1527] When a user inputs the characters they want to write into the device, the device sends the character information to the server. The server analyzes the character information and searches for corresponding model data in a database. The resulting model data is sent to the device and presented to the user. This series of processes allows users to check calligraphy models from the comfort of their own home.

[1528] Practice and submission process

[1529] The user practices calligraphy based on the provided model and takes a photo of the completed work with their smartphone camera. The device sends the captured image data to a server, which receives the image and analyzes it using an AI algorithm. The analysis includes details such as brush strokes and character balance. The server generates an evaluation and corrections, which are sent to the device and displayed to the user. This allows the user to receive feedback that can be used to improve their next practice.

[1530] Emotion Recognition Process

[1531] In the process of the user practicing calligraphy, the emotion engine analyzes the user's expression and voice in real time. When the emotion engine feedbacks the analysis result to the server, the server checks the progress and motivation of the practice based on it, and generates appropriate advice and encouraging messages. By the terminal displaying these messages to the user, more effective learning becomes possible while maintaining the user's motivation.

[1532] Optional function

[1533] This system provides, as options, the acquisition of ranks and grades and the remote guidance function. Regarding the acquisition of ranks and grades, when the user sends multiple practice works to the server and receives a certain evaluation, the server records and accumulates the data and notifies the user when the acquisition conditions for ranks and grades are met. In the remote guidance function, the user selects a desired instructor, and the server notifies the instructor to adjust the guidance schedule. The user and the instructor conduct remote guidance through a video call, and the server records the guidance content. The emotion engine analyzes the emotion of the user during the guidance and provides real-time feedback to the instructor.

[1534] Specific example

[1535] For example, when the user wants to practice the character "美", first enter "美" into the application. The terminal sends this information to the server, and the server retrieves the corresponding model data from the database and sends it to the terminal. The user practices while looking at the presented model, and takes a photo of the completed work with the smartphone and uploads it to the terminal. The server analyzes the image, sends the evaluation and correction results to the user, and the user can use them for the next practice.

[1536] Furthermore, during the user's practice, the emotion engine analyzes the user's expression and feedbacks the concentrated state or tired state to the server. Based on this, the server can send messages such as "You seem to be concentrating. Keep it up like this" or "It might be good to take a little break".

[1537] Users aiming to obtain ranks or grades can submit works that meet the conditions upon receiving notifications from the server and finally be evaluated to obtain ranks or grades. Furthermore, if external guidance is required, users can select a desired instructor within the app and receive remote guidance. Through this series of processes, it becomes possible to receive high-quality calligraphy education even from home. By using the emotion engine, more effective learning is made possible while maintaining users' motivation and concentration.

[1538] Examples of prompt sentences

[1539] Use the following prompt sentences to explain the detailed processing content of the system to the generative AI model.

[1540] "In the calligraphy learning support system, please explain the specific processing steps of displaying a model of the character 'beautiful' and evaluating it after the user's practice."

[1541] With this prompt sentence, the generative AI model explains each process of the system in an easy-to-understand natural language.

[1542] The flow of the specific process in Example 2 will be described using FIG. 13.

[1543] Model presentation process

[1544] [[ID=2⑨]] Step 1:

[1545] The user inputs the character they want to write on the terminal. The user inputs a character such as 'beautiful' in the input field of the application. The input for this step is the user's input character, and the output is character information (such as 'beautiful').

[1546] Step 2:

[1547] The terminal sends the input character information to the server. Here, an HTTP request is used to send the character information to the server. The input for this step is the character information, and the output is the request transmission to the server.

[1548] Step 3:

[1549] The server receives and analyzes the character information. The server uses a natural language processing (NLP) algorithm to analyze the character information and searches for corresponding model data from the database. The input for this step is the received character information, and the output is the search result of the model data.

[1550] Step 4:

[1551] The server sends the search result to the terminal. The server uses an HTTP response to send the model data to the terminal. The input for this step is the model data, and the output is the data transmission to the terminal.

[1552] Step 5:

[1553] The terminal displays and presents the model data to the user. The terminal displays the received model data on the user interface (UI). This includes an image viewer and SVG rendering. The input for this step is the model data, and the output is the display of the model.

[1554] Practice and work submission process

[1555] Step 1:

[1556] The user practices calligraphy based on the displayed model. The user uses paper and pen to write the character "beautiful". The input for this step is the model data, and the output is the practice work.

[1557] Step 2:

[1558] The user takes a photo of the completed work with the smartphone camera. The input of this step is the practice work, and the output is image data.

[1559] Step 3:

[1560] The device sends the captured image data to the server. The image data is sent to the server via an HTTP POST request. The input of this step is the image data, and the output is a request sent to the server.

[1561] Step 4:

[1562] The server analyzes the received image. The server uses an AI algorithm (for example, a model using TensorFlow or PyTorch) to analyze the image and evaluate the brush strokes, balance of the characters, etc. The input for this step is the image data, and the output is the evaluation and corrections.

[1563] Step 5:

[1564] The server generates the evaluation and corrections and sends them to the terminal. The input of this step is the evaluation result, and the output is the data sent to the terminal.

[1565] Step 6:

[1566] The terminal displays the evaluation results to the user. The terminal displays the evaluation results and corrections on the user interface. The input of this step is the evaluation data, and the output is the display of feedback.

[1567] Emotion Recognition Process

[1568] Step 1:

[1569] As the user practices, the emotion engine analyzes the user's facial expressions and voice. The emotion engine collects data in real time using the device's built-in camera and microphone. The input for this step is the user's facial expressions and voice, and the output is emotion analysis data.

[1570] Step 2:

[1571] The emotion engine feeds back the analysis results to the server. The analysis results are sent to the server via an HTTP request. The input of this step is the emotion analysis data, and the output is a request sent to the server.

[1572] Step 3:

[1573] The server receives the feedback and generates an appropriate message for the user. Based on the emotion data, the server generates messages such as "You seem to be concentrating, keep it up" or "Maybe it's time to take a short break." The input of this step is the emotion feedback, and the output is the generated message.

[1574] Step 4:

[1575] The terminal displays the generated message to the user. The terminal displays the message as a notification popup or a detailed feedback screen. The input to this step is the generated message and the output is the display of the message.

[1576] Rank and grade acquisition process

[1577] Step 1:

[1578] The user submits a practice piece to the server. The user submits multiple practice pieces together to the server for evaluation. The input of this step is the practice piece data, and the output is a request sent to the server.

[1579] Step 2:

[1580] The server accumulates practice data and checks the conditions for obtaining a rank or grade. The server saves the user's practice data in a database and notifies them when certain criteria have been met. The input for this step is practice data, and the output is a notification that the conditions have been met.

[1581] Step 3:

[1582] The server sends a notification when the conditions for obtaining a rank or grade are met. The server notifies the user of the conditions that have been met. The input to this step is the notification of the condition being met, and the output is a notification message to the user.

[1583] Step 4:

[1584] The user submits a work that meets the criteria for final judging. The user submits additional work for final evaluation. The input of this step is the final work, and the output is a request to the server.

[1585] Step 5:

[1586] The server performs the final evaluation and determines the rank. The server evaluates the submitted works and determines the final rank. The input for this step is the final judged work, and the output is the evaluation results and the determination of the rank.

[1587] Remote teaching process

[1588] Step 1:

[1589] The user selects the instructor they want within the app. The user then selects the instructor they want to receive remote instruction from on the screen. The input for this step is the instructor's selection information, and the output is the selection result.

[1590] Step 2:

[1591] The server notifies the instructor and schedules the instruction. The server notifies the selected instructor and schedules the instruction. The input of this step is the instructor selection information, and the output is a schedule adjustment notification.

[1592] Step 3:

[1593] The user and instructor receive remote instruction via video call at a specified date and time. WebRTC is used for the video call. The input of this step is the connection information for the video call, and the output is the start of the video call.

[1594] Step 4:

[1595] The server records the video call in real time and saves the call for the user to review later. The input of this step is the video call data and the output is the recording data.

[1596] Step 5:

[1597] The emotion engine analyzes the user's facial expressions and voice and notifies the instructor of the results in real time, so that the instructor can adjust the content of instruction according to the situation. The input of this step is emotion analysis data, and the output is feedback notification to the instructor.

[1598] (Application example 2)

[1599] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1600] Conventional calligraphy learning systems have limited access to high-quality calligraphy education, particularly in the absence of remote feedback and instruction. They also lack effective support for users' learning motivation and progress. Furthermore, conventional systems struggle to provide feedback that reflects users' emotions, creating a need for methods to improve individual learning experiences.

[1601] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting characters to be written, means for analyzing the input character information and searching for model data, means for presenting the model data, means for capturing a work created by the user as an image, means for analyzing the captured image and performing scoring and corrections, means for presenting the scoring results and corrections, means for performing emotion analysis, and means for generating and presenting feedback based on the analyzed emotion information. This not only enables users to receive high-quality calligraphy education, but also enables them to study effectively from home, and the learning experience can be improved by individual feedback based on emotions.

[1602] The "means for inputting the characters you want to write" is an interface that allows the user to input the characters they want to learn or use as a model.

[1603] The "means for analyzing input character information and searching for model data" is a function that analyzes character information input by the user and searches for and retrieves the corresponding calligraphy model data from a database.

[1604] The "means for presenting model data" is a function for visually presenting the retrieved calligraphy model data to the user.

[1605] "Means for capturing a work created by a user as an image" refers to a function for capturing a work written by a user as a digital image using a device such as a camera or scanner.

[1606] The "means for analyzing imported images and grading and correcting" is a function that analyzes the imported image of the user's calligraphy work and grades and corrects it based on evaluation criteria such as brush strokes and balance.

[1607] The "means for presenting the scoring results and corrections" is a function for visually presenting the scoring results and corrections based on the analysis results to the user.

[1608] "Means for emotion analysis" refers to technology for identifying a user's emotional state by analyzing their facial expressions and tone of voice.

[1609] The "means for generating and presenting feedback based on the analyzed emotional information" is a function that generates a feedback message to improve the user's learning experience based on the results of the emotional analysis and presents it to the user.

[1610] "Means for notifying the conditions for obtaining a rank or grade" is a function for notifying the user of the conditions necessary to obtain a rank or grade.

[1611] "Means for evaluating works that meet the acquisition conditions" is a function for evaluating whether or not the conditions for obtaining a rank or grade are met.

[1612] The "means for determining rank and grade" is a function for determining the final rank and grade based on the user's practice work and evaluation results.

[1613] The "means for selecting an instructor from whom the user wishes to receive remote instruction" is an interface for the user to select an instructor from whom the user wishes to receive remote calligraphy instruction.

[1614] The "means for notifying the selected instructor and adjusting the schedule for instruction" is a function for sending a notification to the selected instructor and adjusting the schedule for instruction to suit the convenience of the user and the instructor.

[1615] The "means for providing instruction via video call" is a function that allows an instructor and a user to receive calligraphy instruction in real time via video call.

[1616] "Means for recording instruction content" is a function that allows you to record instruction content given via video calls, etc., so that you can review it later.

[1617] The "means for providing appropriate feedback according to the progress of instruction" is a function for performing emotion analysis and providing appropriate advice and encouraging messages to instructors and users according to the progress of instruction.

[1618] The present invention relates to a calligraphy learning support system. This system is designed to enable users who want to learn calligraphy to receive high-quality calligraphy instruction from home. In particular, by combining it with sentiment analysis technology, it can provide individually optimized feedback to improve the user's learning experience.

[1619] System configuration

[1620] The system consists of the following main components:

[1621] 1. User device (smartphone, tablet, or PC):

[1622] An interface for users to enter the characters they want to write.

[1623] Camera function for capturing images of calligraphy works.

[1624] Camera and microphone for sentiment analysis.

[1625] 2. Server:

[1626] A function that analyzes text information and searches for example data.

[1627] A function to send sample data to the user's terminal.

[1628] Runs AI algorithms that analyze user-uploaded images and provide scoring and correction.

[1629] A function that generates feedback based on the results of sentiment analysis.

[1630] 3. Emotion Engine:

[1631] Technologies that analyze facial expressions and tone of voice to identify a user's emotional state in real time (e.g., the DeepFace library).

[1632] 4. Interface:

[1633] An application for presenting exemplar data and evaluating user-created work.

[1634] An interface with video calling capabilities for remote instruction.

[1635] A UI that displays the scoring results and corrections.

[1636] Example of operation

[1637] 1. Example process:

[1638] The user inputs the characters they want to write into the device. For example, they input the character "beauty" into the input field. The device sends the input character information to the server, which analyzes it and searches for corresponding example data from a database. The search results are sent to the user's device, and the example data is presented. This allows the user to check high-quality examples.

[1639] 2. Practice and submission process:

[1640] Users practice calligraphy based on the provided model. They then take a photo of their work with their smartphone and import it into the device. The image data is sent to a server and analyzed by an AI algorithm. The calligraphy is scored and corrected based on factors such as brush strokes and character balance, and the results are sent back to the device. Users can use this feedback to improve their next practice.

[1641] 3. Emotion recognition process:

[1642] During practice, the emotion engine analyzes the user's facial expressions and voice to identify their emotional state in real time. The analysis results are sent to a server, which generates appropriate feedback based on the results. The feedback is displayed on the device to help maintain the user's motivation.

[1643] Specific technologies used

[1644] DeepFace: A library for sentiment analysis.

[1645] TextBlob: A library for generating feedback messages using natural language processing.

[1646] OpenCV: A library for image processing.

[1647] Specific examples

[1648] For example, if a user wants to learn the character "beauty," they enter "beauty" into the application. The device sends the character information to the server, and the server sends example data to the device. The user practices using the provided example, then takes a photo of the completed work with their smartphone and uploads it to the application. The server analyzes the image and evaluates and corrects it. The emotion engine analyzes the user's state of particular concentration or fatigue, and based on this generates messages such as "You're concentrating. Keep it up" or "It might be a good idea to take a short break."

[1649] Prompt Sentence Examples

[1650] For example, if you analyze emotions while practicing calligraphy and detect a "state of concentration," the prompt sentence is as follows:

[1651] User Sentiment: Concentration

[1652] Guidance message: Great focus! Keep it up!

[1653] As described above, this system is designed so that users can receive high-quality calligraphy education even from home and effectively proceed with learning. By utilizing sentiment analysis technology, the learning experience of users can be further enriched.

[1654] The flow of the specific process in Application Example 2 will be described using FIG. 14.

[1655] Step 1:

[1656] The user inputs the characters they want to write.

[1657] Input: The user inputs characters such as "beautiful" into the terminal.

[1658] Process: The terminal acquires the input character information and sends it to the server. <00...... Output: The character information sent to the server.

[1660] Step 2:

[1661] Search for and display the model data.

[1662] Input: The character information sent to the server.

[1663] Process: The server analyzes the character information and searches for the corresponding model data from the model database.

[1664] Output: The searched model data is sent to the terminal and presented to the user.

[1665] Step 3:

[1666] The user practices calligraphy. <00050...... Input: The model data.

[1668] Process: Based on the model data presented to the user, the user uses paper and a brush to practice calligraphy.

[1669] Output: User-created work.

[1670] Step 4:

[1671] Take a photo of your work and import it into your device.

[1672] Input: A user-created calligraphy piece.

[1673] Processing: The user takes a photo of the work using the smartphone camera and imports it into the device as image data.

[1674] Output: The captured image data.

[1675] Step 5:

[1676] The image data is sent to the server.

[1677] Input: Image data of the work.

[1678] Processing: The image data captured by the terminal is sent to the server.

[1679] Output: Image data sent to the server.

[1680] Step 6:

[1681] Analyze image data and perform scoring and correction.

[1682] Input: Image data sent to the server.

[1683] Processing: The server uses AI algorithms to analyze the image, scoring and correcting it based on criteria such as brushstrokes and balance.

[1684] Output: Marking results and corrections.

[1685] Step 7:

[1686] The scoring results and corrections will be presented.

[1687] Input: Marking results and corrections.

[1688] Processing: The server sends the generated scoring results and corrections to the terminal and presents them to the user.

[1689] Output: The marks and corrections presented to the user.

[1690] Step 8:

[1691] Conduct sentiment analysis.

[1692] Input: User's facial expression and tone of voice.

[1693] Processing: The emotion engine analyzes the user's facial expressions and tone of voice to recognize their emotional state in real time.

[1694] Output: Parsed emotion information.

[1695] Step 9:

[1696] Generate and present feedback based on emotional information.

[1697] Input: Parsed emotion information.

[1698] Processing: The server generates an appropriate feedback message (e.g., "You're concentrating, so keep it up") based on the emotional information and sends it to the device.

[1699] Output: The feedback message presented to the user.

[1700] Step 10:

[1701] Users get feedback that helps them improve their practice next time.

[1702] Input: Feedback message.

[1703] Action: The user checks the feedback message provided and adjusts the content of the next practice based on it.

[1704] Output: Your next calligraphy piece based on your improved practice.

[1705] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1706] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1707] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1708] [Fourth embodiment]

[1709] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1710] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1711] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1712] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1713] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1714] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1715] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1716] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1717] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1718] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1719] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1720] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1721] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1722] The present invention relates to a calligraphy learning support system that enables users who wish to learn calligraphy to receive high-quality calligraphy instruction and enables remote calligraphy instruction using artificial intelligence (AI).

[1723] System Configuration

[1724] This system is composed of the following main elements.

[1725] 1. User Terminal: Such as a tablet, smartphone, or personal computer. The user uses these terminals to practice calligraphy.

[1726] 2. Server: The core part of the system installed centrally. It provides a database of calligraphy models, image analysis by AI, and scoring and correction functions.

[1727] 3. Interface: Software or applications for the user and the system to interact with each other.

[1728] Program Processing

[1729] Model Presentation Process

[1730] 1. The user inputs the characters they want to write on the terminal. For example, input the character "美" into the input field of the application.

[1731] 2. The terminal sends the input character information to the server.

[1732] 3. The server receives the character information, analyzes it, and searches for the corresponding model data from the database.

[1733] 4. The server sends the search results to the terminal.

[1734] 5. The terminal displays the sent model data and presents it to the user. Thus, the user can view the model while at home.

[1735] Practice and Work Submission Process

[1736] 1. The user practices calligraphy based on the displayed model. For example, use paper and pen to write the character "美".

[1737] 2. Once the practice piece is complete, the user takes a photo of it and imports it into their device, often using the camera function on their smartphone.

[1738] 3. The device sends the captured image data to the server.

[1739] 4. The server receives the image and analyzes it using an AI algorithm, scoring it based on multiple criteria, including brush strokes and balance of the characters.

[1740] 5. The server generates the scoring results and corrections and sends them to the terminal.

[1741] 6. The device displays the received results to the user, allowing the user to receive feedback and use it for further practice.

[1742] Optional Feature Process

[1743] 1. Dan / Kyu rank acquisition:

[1744] A user submits multiple practice pieces to the server and receives a certain evaluation.

[1745] The server accumulates the user's practice data and notifies them when they meet the requirements for obtaining a rank or grade.

[1746] Users submit works that meet the requirements, and the server performs a final evaluation to determine rank and grade.

[1747] 2. Remote instruction:

[1748] The user selects the calligraphy instructor from whom they would like to receive remote instruction within the app.

[1749] The server notifies the selected instructor and schedules the instruction.

[1750] The user and the calligraphy instructor conduct remote guidance through video calls.

[1751] The server records the guidance content so that the user can recheck it later.

[1752] Specific example

[1753] For example, when the user wants to practice the character "美", first enter "美" into the application. The terminal sends this information to the server, and the server retrieves the corresponding model data from the database and sends it to the terminal. The user practices while looking at the presented model, takes a photo of the completed work with a smartphone, and uploads it to the terminal. The server analyzes the image, sends the results of evaluation and correction to the user, and the user can use it for the next practice.

[1754] In addition, users aiming to obtain ranks or grades can receive notifications from the server, submit works that meet the conditions, and finally be evaluated to obtain ranks or grades. Furthermore, if external guidance is required, the user can select the desired instructor within the application and receive remote guidance. Through this series of processes, it is realized that high-quality calligraphy education can be received even from home.

[1755] The following explains the process flow.

[1756] Model presentation process <x

[1757] Step 1:

[1758] The user inputs the character to be written into the terminal.

[1759] Specifically, the user opens the application and inputs the character to be written, such as "美", into the text input field.

[1760] Step 2: <00x05328>

[1761] The terminal sends the input character information to the server.

[1762] Specifically, the terminal issues a request to send the character data entered by the user to the server via the API.

[1763] Step 3:

[1764] The server receives the text information, analyzes it, and searches the database for corresponding example data.

[1765] Specifically, the server receives the request and uses a character recognition module to query the database and retrieve the corresponding example data.

[1766] Step 4:

[1767] The server sends the search results to the terminal.

[1768] Specifically, the server transmits the acquired sample data to the terminal as a response.

[1769] Step 5:

[1770] The terminal displays the transmitted example data and presents it to the user.

[1771] Specifically, the terminal receives the response and displays the example image within the application.

[1772] Practice and submission process

[1773] Step 1:

[1774] The user practices calligraphy based on the displayed example.

[1775] Specifically, the user uses paper and a brush to write characters according to the provided example.

[1776] Step 2:

[1777] Once the practice piece is complete, the user takes a picture of it and imports it into their device.

[1778] Specifically, users take a photo of the work using their smartphone camera and upload the image to the app.

[1779] Step 3:

[1780] The terminal transmits the captured image data to the server.

[1781] Specifically, the terminal issues a request to upload image data to the server via the API.

[1782] Step 4:

[1783] The server receives the images and analyzes them using AI algorithms.

[1784] Specifically, the server passes the image data to an analysis engine, which evaluates it based on multiple criteria, such as brush strokes and character balance.

[1785] Step 5:

[1786] The server generates the scoring results and corrections and sends them to the terminal.

[1787] Specifically, the server calculates a score based on the analysis results, generates feedback including points for improvement, and sends it to the device.

[1788] Step 6:

[1789] The terminal displays the received results to the user.

[1790] Specifically, the device receives a response from the server and displays the scoring results and corrections within the app.

[1791] Optional Feature Process

[1792] Obtaining Dan / Kyu rank

[1793] Step 1:

[1794] A user submits multiple practice pieces to the server and receives a certain evaluation.

[1795] Specifically, users continue to upload their work to the app and accumulate ratings.

[1796] Step 2:

[1797] The server accumulates the user's practice data and notifies them when they meet the requirements for obtaining a rank or grade.

[1798] Specifically, the server checks the user's evaluation log and determines whether the acquisition conditions are met. If the conditions are met, the server notifies the user.

[1799] Step 3:

[1800] Users submit works that meet the requirements, and the server performs a final evaluation to determine rank and grade.

[1801] Specifically, users submit their work to be evaluated through a specified procedure, and the server performs a final evaluation to determine the rank and grade.

[1802] Remote instruction

[1803] Step 1:

[1804] The user selects the calligraphy instructor from whom they would like to receive remote instruction within the app.

[1805] Specifically, the user selects the desired instructor from a list of instructors in the app.

[1806] Step 2:

[1807] The server notifies the selected instructor and schedules the instruction.

[1808] Specifically, the server sends requests to instructors and provides an interface for scheduling.

[1809] Step 3:

[1810] The user and the calligraphy instructor will conduct remote instruction via video call.

[1811] Specifically, the user and instructor will provide instruction via a video call app at a specified time.

[1812] Step 4:

[1813] The server records the instruction content so that the user can review it later.

[1814] Specifically, the server stores recorded data of video calls and makes it available for users to review if necessary.

[1815] Example 1

[1816] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1817] Conventional calligraphy education systems require face-to-face instruction and practice using paper and brush in order for users to receive high-quality calligraphy training, which can be difficult for users living in remote locations. There are also issues with the lack of prompt feedback on the work users have practiced and evaluation of their ranks and grades. Furthermore, there is a lack of a flexible system that allows for real-time remote instruction. To address these issues, an effective and efficient calligraphy learning support system is needed.

[1818] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1819] In this invention, the server includes means for a user to input characters that the user wants to write, means for analyzing the input character information and searching for model data, means for presenting the searched model data, means for capturing an image of a work created by the user, means for analyzing the captured image using an artificial intelligence algorithm and grading and correcting the image, means for presenting the grading results and corrections to the user, means for notifying the user of the conditions for obtaining a rank or grade, means for evaluating works that satisfy the conditions, means for determining the rank or grade, means for the user to select an instructor from whom they wish to receive remote instruction, means for notifying the selected instructor and arranging a schedule for instruction, means for providing instruction via video call, and means for recording the content of instruction. This enables users to receive high-quality calligraphy education from the comfort of their own homes, receive prompt feedback and evaluation, and receive real-time remote instruction.

[1820] "User" refers to a user of the system who intends to receive calligraphy education.

[1821] A "terminal" is a device that allows a user to use the system, and includes a tablet, smartphone, or PC.

[1822] The "server" is the central computer of the system, and is a device that provides functions for searching sample data, analyzing images, and scoring and correcting.

[1823] "Model data" is calligraphy sample data that is presented as a standard for the characters that the user wants to learn.

[1824] "Means for capturing images" refers to devices and functions necessary for inputting a user's work into the system as a digital image, and includes camera functions, etc.

[1825] An "artificial intelligence algorithm" is a program that runs on a server and analyzes, grades, and corrects works created by users.

[1826] "Scoring result" is the evaluation score given to the user's work by the artificial intelligence algorithm.

[1827] "Correction content" is feedback information indicating corrections and improvements provided to the user's work.

[1828] "Rank / Grade" refers to the qualification grade that indicates the level of the user's calligraphy skills.

[1829] "Acquisition conditions" are the requirements that a user must meet in order to acquire a rank or grade.

[1830] The "instructor who wishes to receive remote instruction" is a calligraphy instructor that the user selects within the system to receive calligraphy instruction via video call.

[1831] "Video calling" is a means of communication that allows for the exchange of video and audio in real time over the Internet.

[1832] "Means for recording instruction content" refers to the ability to digitally save instruction given during a video call.

[1833] This invention is a calligraphy learning support system that allows users to receive high-quality calligraphy training. The system mainly consists of a user terminal, a server, and a user interface.

[1834] System configuration

[1835] 1. User device: A device such as a tablet, smartphone, or PC that users use to learn calligraphy.

[1836] 2. Server: This is the core device of the system, providing a database of calligraphy samples, image analysis using AI, and scoring and correction functions.

[1837] 3. User Interface: Software or applications that allow users to interact with a system.

[1838] Model Prompt

[1839] 1. The user enters the characters they want to write in the input field of the application. For example, enter the character "美".

[1840] 2. The terminal sends the entered character information to the server. Specifically, use an HTTP request to send the character information to the server.

[1841] 3. The server receives the character information, analyzes it, and searches for the corresponding model data from the database. The server executes an analysis script written in Python and uses MySQL for the database.

[1842] 4. The server sends the search results to the terminal, and the terminal displays the received model data on the application.

[1843] Practice and Submission of Works

[1844] The user practices calligraphy referring to the model. Using paper and pen, for example, write the character "美". When the practice work is completed, the user uses the camera function of the smartphone to take a photo of the work and import it into the terminal as an image. The terminal sends the captured image data to the server via an HTTP POST request.

[1845] The server receives the image data and performs analysis using an AI algorithm based on TensorFlow. Evaluate aspects such as the stroke movement and the balance of the characters, and generate a scoring result and correction content. The generated data is in JSON format, and the server sends this to the terminal. Then, the terminal displays the received result on the application. The evaluation and correction content are visually displayed using HTML and CSS.

[1846] Obtaining Dan and Ji Ranks

[1847] The user sends multiple practice works to the server and receives evaluations. The server accumulates the user's practice data in the database and sends a notification when the conditions for obtaining ranks or grades are met. The works that meet the conditions for the final evaluation are sent again, and the server finally determines the ranks or grades and sends the evaluation results to the terminal.

[1848] Remote guidance

[1849] When the user desires remote guidance within the application, an instructor is selected using the interface within the system. The server notifies the selected instructor and adjusts the guidance schedule. The WebRTC technology is used for video calls to enable real-time guidance. The content of the guidance is saved in cloud storage and recorded using Amazon S3.

[1850] Specific example

[1851] For example, when the user wants to practice the character "美" (beautiful), the user inputs "美" into the application. The terminal sends this information to the server, and the server retrieves the corresponding model data from the database and sends it to the terminal. The user practices while looking at the presented model, takes a photo of the completed work with a smartphone, and uploads it to the terminal. The server analyzes the image, sends the evaluation and correction results to the user, and these can be used for the next practice. Also, when obtaining ranks or grades, multiple works are submitted for evaluation, and a notification is received when the criteria are met, making it possible to obtain ranks or grades.

[1852] Examples of prompt sentences

[1853] Please display a model for writing the character "美".

[1854] Search for and display the model of the character "美" entered in the database. Also, please take a photo of the user's practice work and evaluate it.

[1855] The flow of specific processing in Example 1 will be described using FIG. 11.

[1856] Step 1:

[1857] The user inputs the characters they want to write in the input field of the application. The data to be input is Chinese characters such as "beautiful". The user inputs characters in the input field and performs the operation of pressing the send button.

[1858] Step 2:

[1859] The terminal sends the input character information to the server. The input is the character information input by the user, and the output is an HTTP request to the server. The data is sent in JSON format. At this time, the terminal performs the operation of sending a request to the server using network communication.

[1860] Step 3:

[1861] The server receives the character information and analyzes the characters. The input is the character information sent from the terminal, and the output is the character code as the analysis result. The server executes a Python script and performs the operation of analyzing the specified characters.

[1862] Step 4:

[1863] The server searches for the corresponding model data from the database based on the character code. The input is the analyzed character code, and the output is the model data. The server executes an SQL query against MySQL and performs the operation of obtaining the relevant model data.

[1864] Step 5:

[1865] The server sends the search results to the terminal in JSON format. The input is the model data obtained from the database, and the output is an HTTP response to the terminal. The server converts the obtained data into JSON format and performs the operation of sending it as an HTTP response.

[1866] Step 6:

[1867] The example data received by the device is displayed on the application. The input is the example data sent from the server, and the output is the example displayed on the user interface. The device uses JavaScript to draw the example data.

[1868] Step 7:

[1869] The user practices calligraphy based on the displayed example. Using paper and a brush, the user writes the character "beauty."

[1870] Step 8:

[1871] Once the practice piece is complete, the user takes a photo of the piece using the smartphone's camera function and imports the image into the device. The input is the image of the completed piece, and the output is a digital image saved on the device. The user uses the camera function to acquire the image.

[1872] Step 9:

[1873] The device sends the captured image data to the server using an HTTP POST request. The input is the image data captured by the user, and the output is a request to the server. The device uses network communication to send the image data to the server.

[1874] Step 10:

[1875] The server receives the image data and analyzes it using an AI algorithm. The input is the transmitted image data, and the output is the evaluation data as the analysis result. The server uses TensorFlow to analyze the image and evaluate the brush strokes and balance of the characters.

[1876] Step 11:

[1877] The server generates the scoring results and corrections and sends them to the device. The input is the analysis result, and the output is the evaluation results and corrections sent to the device. The server converts the evaluation data into JSON format and sends it as an HTTP response.

[1878] Step 12:

[1879] The terminal displays the results it receives on the application. The input is the evaluation results and corrections sent from the server, and the output is the feedback displayed on the user interface. The terminal uses HTML and CSS to visually display the results.

[1880] Step 13:

[1881] The user performs further practice based on the provided feedback. The input is the displayed feedback, and the output is an improved practice piece. The user then uses the feedback to perform the writing action again.

[1882] (Application example 1)

[1883] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1884] Conventional calligraphy learning systems provide paper-based examples and feedback, making real-time instruction and evaluation difficult. Furthermore, due to limited learning support in remote locations, students must attend a calligraphy class in order to receive high-quality calligraphy education. Furthermore, there are currently insufficient methods for improving the efficiency of calligraphy learning using smart devices.

[1885] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1886] In this invention, the server includes means for inputting characters to be written, means for analyzing the input character information and searching for model data, means for presenting the model data, means for capturing a work created by the user as an image, means for analyzing the captured image and performing scoring and corrections, means for presenting the scoring results and corrections, means for displaying the model data using smart glasses, means for photographing the created work using the camera function of the smart glasses, means for transmitting the image data to a cloud server and analyzing it using an AI algorithm, and means for displaying the analysis results and feedback on the smart glasses. This enables real-time feedback, enables high-quality calligraphy education even from remote locations, and realizes efficient calligraphy learning by utilizing smart devices.

[1887] The "means for inputting characters to be written" is an interface for inputting characters to be written into a device when a user is learning calligraphy.

[1888] The "means for analyzing input character information and searching for model data" is a function that analyzes character information input by the user and searches for corresponding calligraphy model data from a database.

[1889] The "means for presenting example data" is a function for displaying the retrieved example data on the display of the terminal so that the user can confirm it.

[1890] The "means for capturing a work created by a user as an image" is a function for capturing a calligraphy work that a user has written on paper or the like as a digital image using a camera or the like.

[1891] The "means for analyzing the imported image and scoring and correcting it" refers to an AI algorithm that analyzes the imported image of the calligraphy work and automatically scores and corrects it.

[1892] The "means for presenting the scoring results and corrections" is a function for displaying the analysis results and corrections in an easy-to-understand manner to the user.

[1893] "Means for displaying example data using smart glasses" refers to a function that uses the display of the smart glasses to virtually display example data within the user's field of view.

[1894] "Means for photographing a work created using the camera function of the smart glasses" refers to a function for photographing a work written by a user using the camera built into the smart glasses.

[1895] "Means for sending image data to a cloud server and analyzing it using an AI algorithm" refers to a function that uploads image data of calligraphy works sent from smart glasses or other devices to a server on the cloud and uses an AI algorithm to analyze the data.

[1896] "Means for displaying analysis results and feedback on the smart glasses" is a function for displaying the analysis results and feedback returned from the cloud server on the display of the smart glasses.

[1897] This invention is a system that utilizes smart glasses to provide real-time calligraphy learning support. This system allows users who want to learn calligraphy to receive high-quality calligraphy education using smart glasses and a cloud server.

[1898] System Configuration

[1899] 1. User Device

[1900] Smart glasses (e.g., Google Glass) are used as the user terminal. This device has the function of displaying the sample data, taking a photo of the written characters, and sending the photo to a cloud server.

[1901] 2. Cloud Server

[1902] The cloud server provides a database of calligraphy models, image analysis by AI, and grading and correction functions. It functions as an infrastructure for real-time data processing and analysis.

[1903] 3. Interface

[1904] An interface is provided for the user to interact with the system through the display, operation panel, and voice recognition function of the smart glasses.

[1905] Outline of Program Execution

[1906] Model Presentation Process

[1907] The server provides a means for the user to input the characters they want to write through voice or touchpad operations using the smart glasses. For example, when the user inputs the character "美", the server analyzes the character information and searches the database for the corresponding model data. The model data is displayed on the smart glasses, and the user can practice calligraphy while viewing it in real time.

[1908] Practice and Work Submission Process

[1909] The user practices calligraphy while viewing the model displayed through the smart glasses. After practice, the user uses the camera function of the smart glasses to take a picture of the work and transmits the image data to the cloud server. The cloud server analyzes the image using an AI algorithm and evaluates aspects such as the stroke movement and character balance. The analysis results are displayed on the smart glasses together with the calligraphy correction content, and the user can obtain real-time feedback.

[1910]

[1577] Hardware and Software to be Used

[1911] This system operates with smart glasses, a cloud server, and an AI analysis model (using, for example, TensorFlow) as its main components. The smart glasses function as a user interface, and the cloud server provides data processing and storage solutions. An image analysis model is used for AI analysis to perform a detailed evaluation of the user's calligraphy works.

[1912] Examples of specific cases and prompt sentences

[1913] For example, if the user wants to practice the character "beautiful", first enter the command "display a model" into the smart glasses by voice or touchpad operation. The following are examples of prompt sentences.

[1914] Prompt sentence: "Please enter the model character."

[1915] When the user practices calligraphy and the character is completed, next, activate the camera with the gesture function of the smart glasses and take a picture of the work. The following are examples of prompt sentences.

[1916] Prompt sentence: "When the character is completed, please activate the camera and take a picture."

[1917] The captured image data is sent to the cloud server, and the AI model performs analysis and displays the results and feedback on the display of the smart glasses. When the analysis is completed, the following prompt sentence is displayed.

[1918] Prompt sentence: "The analysis is complete. Please check the feedback."

[1919] In this way, a system is provided that allows the user to practice calligraphy and receive feedback in real time.

[1920] The flow of specific processing in Application Example 1 will be described using FIG. 12.

[1921] Step 1:

[1922] The user wears smart glasses and enters the characters they want to write. The input method is voice recognition or touchpad operation. When the user enters "beautiful" in response to the prompt "Please enter the model characters", the entered character information is sent by the smart glasses to the cloud server. The server analyzes the received character information and searches for the corresponding model data from the model database.

[1923] Step 2:

[1924] The server sends the model data to the smart glasses, and the smart glasses display the data in the user's field of vision. As a result, the user can start practicing calligraphy while checking the model in real time. The display of the model data is projected onto the display of the smart glasses, and the user writes characters on paper while looking at the model.

[1925] Step 3:

[1926] After the user finishes writing characters referring to the model, they use the camera function of the smart glasses to take a picture of the work. The user follows the prompt "Please activate the camera and take a picture when the characters are completed" using gesture operation or the touchpad to take a picture of the work. The captured image data is sent by the smart glasses to the cloud server.

[1927] Step 4:

[1928] The server analyzes the received image data using an AI algorithm. As data processing, preprocessing of the image and feature extraction are performed, and analysis is carried out using evaluation criteria such as the movement of the pen, the balance of the characters, and the thickness of the lines. For this analysis, a generative AI model such as TensorFlow is used. As a result of the analysis, scores and detailed correction contents are generated.

[1929] Step 5:

[1930] The server sends the generated analysis results and feedback to the smart glasses, which then display the received results in the user's field of view. Following the prompt "Analysis completed. Please check your feedback," feedback is presented to the user, allowing the user to use this information to improve their practice next time.

[1931] Step 6:

[1932] When a user practices multiple times and meets certain conditions, the conditions for obtaining a rank or grade are notified to the smart glasses. The server accumulates the user's practice data and notifies the user when the conditions are met. The user then photographs and submits works that meet the conditions, and the server performs a final evaluation to determine the rank or grade.

[1933] Step 7:

[1934] When a user requests remote instruction, they select an instructor through the smart glasses interface. The server then notifies the instructor and schedules the instruction. The instructor provides real-time instruction to the user through a video call function. The instruction is then recorded on the server for later review by the user.

[1935] Through this series of steps, the calligraphy learning support system, which utilizes smart glasses and a cloud server, provides users with an efficient and high-quality learning environment.

[1936] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1937] The present invention relates to a calligraphy learning support system. This system is for users who want to learn calligraphy to receive high-quality calligraphy education and enables remote calligraphy instruction using artificial intelligence (AI). Additionally, by combining an emotion engine that analyzes the emotions of the user, the learning experience of the user can be further improved.

[1938] System Configuration

[1939] This system is composed of the following main elements.

[1940] 1. User terminal: Such as a tablet, smartphone, or personal computer. The user uses these terminals to conduct calligraphy learning.

[1941] 2. Server: The core part of the system installed centrally. It provides a database of calligraphy models, image analysis by AI, and a scoring and commenting function.

[1942] 3. Interface: Software or applications for the user and the system to interact with each other.

[1943] 4. Emotion engine: A device or software that analyzes the user's facial expressions and voice tones to recognize the emotional state.

[1944] Program Processing

[1945] Model Presentation Process

[1946] 1. The user inputs the character they want to write into the terminal. For example, input the character "beautiful" into the input field of the application.

[1947] 2. The terminal sends the input character information to the server.

[1948] 3. The server receives the character information, analyzes it, and searche...

Claims

1. A way to input the characters you want to write, A means for analyzing input character information and searching for example data; a means for presenting exemplar data; A means for capturing user-created works as images; A means for analyzing the captured images and performing grading and correction; A means of presenting the results of the marks and corrections; A system including:

2. A means of notifying the conditions for obtaining ranks and grades; a means of evaluating works that meet the acquisition criteria; 10. The system of claim 1, further comprising means for determining rank.

3. a means for selecting an instructor who wishes to provide remote instruction; a means of notifying the selected instructor and arranging the instruction schedule; A means of providing instruction via video calls; The system of claim 1 further comprising means for recording instruction content.

Images