system

Abstract

We provide the system. [Solution] A data acquisition means for acquiring the user's oral cavity condition as image data and audio data, A data transmission means that encrypts the acquired data and sends it to an external server, An analytical means that analyzes the user's oral health status based on received data and predicts the risk of periodontal disease, A notification system that sends a notification to the user when an anomaly is detected based on the analysis results, A plan generation means that generates an individualized oral care plan considering the user's lifestyle information, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance that responds to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

[0006] "Data acquisition means" refers to a function for acquiring the user's oral cavity condition as image data and audio data.

[0007] "Data transmission means" refers to a function for encrypting acquired data and securely transmitting it to an external server.

[0008] "Analysis means" refers to a function that analyzes the user's oral health status based on received data and predicts the risk of periodontal disease.

[0009] "Notification method" refers to a function that sends alerts or notifications to the user when an anomaly is detected based on the analysis results.

[0010] "Plan generation means" refers to a function that generates individualized oral care plans, taking into account the user's lifestyle information.

[0011] A "machine learning algorithm" refers to a computational process that compares past data with current data to find patterns and relationships.

[0012] "Natural language generation technology" refers to technology that uses data to generate information in a language format that is easy for humans to understand. [Brief explanation of the drawing]

[0013] [Figure 1]This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, a numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is embodied as a system that periodically records the condition of the oral cavity using the user's smart device, analyzes the data using AI, and detects the risk of periodontal disease early, thereby providing preventive measures. This system mainly consists of the user's smart device, a server that processes the data, and a communication infrastructure that connects them.

[0035] Users capture images of their oral cavity and acquire audio data using a dedicated application on their smart device. This data is temporarily stored on the device and transmitted to a server using a secure communication method. The server employs a generative AI model that utilizes machine learning algorithms to analyze the received data. This AI analyzes the condition of the gums in detail from the image data and detects abnormalities in oral sounds using the audio data.

[0036] Based on the analysis results, the server assesses the user's periodontal disease risk and sends a notification to the device according to the risk level. This notification includes information about the area where abnormalities were detected, the degree of risk, and specific countermeasures. The device displays the received information to the user in an easy-to-understand visual format and provides guidance on appropriate care.

[0037] Furthermore, the server uses natural language generation technology to create a personalized oral care plan, taking into account the user's saved lifestyle data. This plan supports users in practicing self-care without getting lost due to lack of knowledge or time.

[0038] For example, if gum swelling is detected in the early stages from daily image data, the server notifies the terminal of the risk. Based on the user's eating habits, it provides advice on reducing excessive sugar intake and information on recommended oral care products. In this way, the present invention provides an oral care service that is useful for users to proactively manage their health and reduce the risk of developing periodontal disease.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The user launches a dedicated app on their smart device. The app instructs the user to take images of their oral cavity and simultaneously records necessary audio data. This obtains data about the user's current oral condition.

[0042] Step 2:

[0043] The device temporarily stores the acquired image and audio data and encrypts them. Encryption is performed to ensure the confidentiality of the data.

[0044] Step 3:

[0045] Encrypted data is sent from the terminal to the server. A secure communication protocol reduces the risk of data leakage during transit.

[0046] Step 4:

[0047] A generative AI model is used to analyze the data received by the server. The AI ​​analyzes the condition of the gums from image data and detects abnormal patterns. It also detects abnormalities based on brushing sounds and oral sounds from audio data.

[0048] Step 5:

[0049] The server evaluates the user's periodontal disease risk based on the analysis results and determines the risk level. The server then prepares to send an alert to the terminal according to the risk level.

[0050] Step 6:

[0051] The device receives alerts sent from the server and displays them visually to the user. The display includes detailed analysis results and recommended actions. The user can then take self-care actions based on this information.

[0052] Step 7:

[0053] Furthermore, the server generates a personalized oral care plan, taking into account the user's lifestyle and daily data. It utilizes natural language generation technology to send the information to the device in an easy-to-understand format.

[0054] (Example 1)

[0055] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0056] Continuously and regularly monitoring oral health and detecting the risk of periodontal disease early is a challenging task for individual users. Furthermore, providing appropriate oral care plans tailored to individual lifestyles and health conditions is essential for effective self-care. However, conventional methods lacked systems to address these challenges.

[0057] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0058] This invention includes a server that uses a machine learning model to analyze the user's oral health status and predict and evaluate the risk of periodontal disease; a server that sends a notification to the user if an abnormality is detected based on the analysis results; and a server that generates an individualized oral care plan considering the user's lifestyle information. This allows the user to easily understand the condition of their own oral cavity and receive specific health management and preventive measures tailored to their lifestyle.

[0059] "Data acquisition means" refers to a device or method for acquiring the condition of a user's oral cavity as image data and audio data.

[0060] "Data transmission means" refers to a device or method for encrypting acquired image data and audio data and securely transmitting them to an external processing device.

[0061] "Analysis means" refers to a device or method for analyzing the user's oral health status using a machine learning model based on received data, and for predicting and evaluating the risk of periodontal disease.

[0062] "Notification means" refers to a device or method for transmitting information to the user when an anomaly is detected based on the analysis results.

[0063] "Plan generation means" refers to a device or method for generating an individualized oral care plan while taking into account the user's lifestyle information.

[0064] A "machine learning model" is a mathematical model used to compare past and present data, recognize patterns, and make predictions about new data.

[0065] "Natural language generation technology" is a technology that enables computers to generate text in a user-friendly format based on text data.

[0066] This invention is a system that utilizes a user's smart device to periodically acquire images and audio data from the oral cavity, analyzes them, and provides an individualized oral care plan. It mainly consists of a smart device, a server, and a communication network.

[0067] Users install a dedicated application on their smart device and use it to capture images of their oral cavity and collect audio data. The device's built-in camera and microphone are used to acquire both image and audio data. The acquired data is temporarily stored on the smart device and then transmitted to the server using secure protocols such as SSL / TLS.

[0068] The server performs detailed analysis on the received data using a specific machine learning model. This generative AI model has the ability to identify the condition of the gums from image data and detect abnormalities in oral sounds from audio data. This allows for the prediction of periodontal disease risk and enables early intervention when necessary.

[0069] Based on the analysis, the server generates information about the user's health status and sends notifications to the user's smart device according to the risk level. These notifications include the specific location and severity of any abnormalities, as well as recommended countermeasures. The smart device also presents the information visually in an easy-to-understand manner, supporting intuitive understanding for the user.

[0070] Furthermore, this system uses natural language generation technology to create a personalized oral care plan, referencing the user's lifestyle information. This plan serves as a guide for self-care that the user can implement on a daily basis.

[0071] For example, if slight gum swelling is detected from captured image data, the server will notify the user of the risk and send advice to reduce sugar intake as a way to improve their diet. It will also provide product information that can be used for actual self-care.

[0072] An example of a prompt message is: "Analyze this user's most recent intraoral image and assess their health status. If any abnormalities are found, indicate their location and risk level, and suggest specific corrective measures."

[0073] In this way, the present invention operates as a system that supports users' autonomy in health management and plays a role in reducing the risk of periodontal disease.

[0074] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0075] Step 1:

[0076] The user launches an application on their smart device, takes an image of their mouth, and records audio. The input consists of image and audio data acquired using the smart device's camera and microphone. Specifically, the user takes a picture of their mouth with the camera and taps the audio input button to record audio. The output—image and audio data—is saved on the smart device.

[0077] Step 2:

[0078] The terminal encrypts the acquired data and sends it to the server. The input consists of image data and audio data stored on the terminal. As part of the data processing, the data is encrypted using the SSL / TLS protocol. Specifically, the terminal executes the data encryption process and establishes a communication channel with the server. As output, the encrypted data is sent to the server via a secure communication path.

[0079] Step 3:

[0080] The server prepares the received data for analysis. The input consists of encrypted image and audio data sent to the server. The data processing involves decrypting the data and converting it into an analyzable format. The output is data suitable for analysis.

[0081] Step 4:

[0082] The server activates a generation AI model and analyzes image and audio data using prompt text as input. For data processing, an image analysis algorithm examines the condition of the gums, and an audio analysis algorithm checks for abnormalities in the audio data. Specifically, the AI ​​model identifies abnormal areas in the image and identifies abnormal patterns in the audio. The output provides analysis results regarding health status and risk levels.

[0083] Step 5:

[0084] The server evaluates the user's periodontal disease risk based on the analysis results, creates a notification message, and sends it to the terminal. The input is the analysis results from the AI ​​model. As data processing, it performs specific actions to generate a message according to the risk level. As output, the notification message is sent to the user's terminal.

[0085] Step 6:

[0086] The terminal displays received notifications to the user. The input is the notification message sent from the server. Specifically, the terminal uses its display function to visually present the message to the user. The output provides information that guides the user's next course of action.

[0087] Step 7:

[0088] The server uses the user's lifestyle information to create a personalized oral care plan using a generative AI model. The input consists of the user's lifestyle data and the results of the AI ​​model's analysis. Specifically, natural language generation technology is used to create the care plan as text. The output is a user-specific oral care plan, which is then sent to the terminal.

[0089] (Application Example 1)

[0090] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0091] While it is important to properly manage oral health and detect the risk of periodontal disease early, providing specific care suggestions based on each user's health condition and lifestyle in real time is not easy. Furthermore, even in physical stores, there is a lack of means to provide consumers with useful oral care information on the spot and promote product use.

[0092] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0093] In this invention, the server includes data collection means, information transmission means, analysis means, information presentation means, and plan generation means. This allows users to understand their oral health status and receive individually optimized oral care plans in real time via their smart devices or video devices with audio output capabilities in physical stores.

[0094] "Data collection means" refers to devices and methods for acquiring the user's oral cavity condition as image and audio information.

[0095] "Information transmission means" refers to devices or technologies for encrypting collected oral cavity information and transmitting it to an external information processing device.

[0096] "Analysis means" refers to a technology or device for analyzing received information, evaluating the user's oral health status, and predicting the risk of periodontal disease.

[0097] A "notification means" is a device or method for sending a notification to a user when an anomaly is detected based on the analysis results.

[0098] "Plan generation means" refers to a technology or device for generating an individualized oral care plan, taking into account the user's lifestyle pattern information.

[0099] "Information presentation means" refers to a technology or device that uses a video device with audio output capabilities installed in a physical store to provide users with analysis results and recommended oral care methods visually and audibly.

[0100] This invention constructs a system for monitoring oral health and providing users with personalized care plans. The system utilizes the user's smart device and digital signage installed in physical stores to acquire information in real time and provide analysis results and care plans.

[0101] First, the user uses a dedicated application on their smart device to collect image and audio information from their oral cavity. The device encrypts this data and sends it to a server in the cloud. The server uses a generative AI model with machine learning algorithms to analyze the received data and assess the risk of periodontal disease and other oral health conditions. This analysis generates a personalized oral care plan based on the user's health condition.

[0102] The analysis results and care plan are not only notified to the user's smart device, but are also provided as visual and auditory information on video devices with audio output capabilities installed in physical stores. This allows users to receive consistent care advice both at home and in stores.

[0103] For example, a customer who undergoes an oral check at a drugstore might receive a notification about their periodontal disease risk on the spot, prompting them to immediately purchase appropriate oral care products. Another example of a prompt message for the generating AI model is: "Analyze the oral images, assess the periodontal disease risk, and generate recommended care methods based on the risk level."

[0104] This system enhances user convenience and streamlines the maintenance and management of oral health.

[0105] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0106] Step 1:

[0107] The user uses a dedicated application on a smart device to acquire image and audio information from inside their mouth. The user takes pictures of the inside of their mouth with a camera and collects audio data through a microphone. The input is images and audio information of the user's mouth, and the output is this data.

[0108] Step 2:

[0109] The device encrypts the image and audio information it acquires. The device uses encryption technologies such as the SSL / TLS protocol to ensure data security. The input to this process is the captured image and audio information, and the output is the encrypted image and audio information.

[0110] Step 3:

[0111] The device sends encrypted data to a server in the cloud. The device uses a secure communication method to send encrypted data to a predetermined server address. The input is encrypted image and audio information, and the output is the transmission of data to the server.

[0112] Step 4:

[0113] The server analyzes the received data using a generative AI model based on machine learning algorithms. The server utilizes this generative AI model to analyze the condition of the gums from image information and detect abnormalities in oral sounds from audio information. The input to this process is the decrypted result of the received encrypted data, and the output is the analysis result indicating the user's oral health status.

[0114] Step 5:

[0115] The server generates an oral care plan for the user based on the analysis results and creates a notification. The server generates a notification that takes into account the user's lifestyle and health data, considering their individual care plan, and converts it into text format. The inputs to this process are the analysis results and the user's lifestyle information, and the outputs are the personalized care plan and notification message.

[0116] Step 6:

[0117] The server generates notification messages and sends them to terminals and in-store video displays. The server sends notifications to terminals via SMS, email, or a dedicated app, and provides them to users visually and audibly in in-store settings. The inputs to this process are the generated care plans and notification messages, and the output is the information displayed on terminal screens and video displays.

[0118] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0119] This invention is a system for understanding the user's oral health status and providing preventive measures against periodontal disease. By combining it with an emotional engine, it realizes customized care that responds to the user's emotional state.

[0120] The user periodically acquires images and audio data of their oral cavity using a dedicated application installed on their smart device. This data is temporarily stored on the device, encrypted, and then sent to a server. The server uses a generative AI model to analyze the received data, examining the condition of the user's gums and the sounds of their brushing.

[0121] Furthermore, this system incorporates an emotion engine that can recognize the user's emotional state. The emotion engine identifies the user's emotions from factors such as voice tone and facial expressions, and evaluates their motivation and stress level regarding oral care. For example, if the system determines that the user is experiencing stress, the plan generation mechanism will provide advice and product suggestions to alleviate that stress.

[0122] The server integrates and analyzes the user's emotional state and oral health data to generate an optimal care plan. Using natural language generation technology, it sends notifications in a format easily understood by the user. These notifications can be emotionally sensitive and include a relaxed tone and encouraging words.

[0123] For example, if user data indicates a risk of periodontal disease and stress levels are also identified, the server will select an approach that promotes self-care in a calming atmosphere. It will also support the user's periodontal disease prevention efforts by recommending simple and effective care methods. In this way, the present invention provides a more effective and personalized care service that comprehensively considers the user's emotions and oral health status.

[0124] The following describes the processing flow.

[0125] Step 1:

[0126] The user launches a dedicated app on their smart device and takes a picture of the inside of their mouth. The app instructs the user to capture the image and record audio, thereby obtaining detailed data about the inside of their mouth.

[0127] Step 2:

[0128] The device encrypts the acquired data and sends it to the server while protecting privacy. Secure protocols such as TLS are used in this process.

[0129] Step 3:

[0130] The server analyzes the received data. Image analysis algorithms are used to assess the condition of the gums, and potential problems are detected from the audio data. At this stage, foundational data is generated to assess the risk of periodontal disease.

[0131] Step 4:

[0132] The emotion engine analyzes transmitted voice tone and facial expression data to recognize the user's emotional state. This identifies the user's current mood and stress level.

[0133] Step 5:

[0134] The server integrates the analysis results, combining emotional state and health data to more precisely assess the user's periodontal disease risk. If necessary, it detects anomalies in areas requiring specific attention.

[0135] Step 6:

[0136] A notification is sent from the server to the device. This notification includes the results of the periodontal disease risk assessment, along with personalized advice that takes into account the user's emotional state.

[0137] Step 7:

[0138] The plan generation system considers the user's lifestyle and emotional state, uses natural language generation technology to generate a specific self-care plan, and sends it to the device. The device then presents this plan to the user, supporting their oral care efforts.

[0139] (Example 2)

[0140] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0141] Conventional oral care support systems provide care plans without considering the user's emotional state, which can lead to insufficient user motivation and engagement with care. Therefore, there is a need to provide more effective care plans that take the user's emotional state into account.

[0142] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0143] In this invention, the server includes means for recognizing the user's emotional state from voice and image data, means for integrating and analyzing the emotional state and oral health status to generate an individualized care plan, and means for notifying the user of the generated care plan in natural language. This makes it possible to provide a care plan that takes the user's emotions into consideration.

[0144] A "user" is an individual who uses the system to provide oral care data and receive services.

[0145] "Oral condition" refers to information related to the user's oral health, specifically the condition of the gums and teeth, which is acquired as digital data.

[0146] "Image data" refers to a digital recording of visual information about the user's oral cavity.

[0147] "Audio data" refers to digital recordings of sounds produced when a user performs actions such as brushing their teeth.

[0148] "Device" is a general term for hardware and software used to acquire, transmit, and analyze data.

[0149] "Encryption" is the process of transforming the content of digital data using a specific algorithm in order to transmit it securely.

[0150] An "external device" is another computer system connected via a network for the purpose of processing and storing data.

[0151] "Analysis" is the process of examining the acquired data in detail and evaluating the user's oral health status.

[0152] "Risk" is an indicator that shows the potential for problems with the user's oral health.

[0153] "Notifications" refer to information that visually or audibly communicates analysis results and care plans to the user.

[0154] "Emotional state" refers to the user's psychological state, including their stress levels and relaxation levels.

[0155] "Creation" refers to the process of developing a care plan that is tailored to a specific purpose and is based on the user's characteristics.

[0156] "Natural language" refers to the language that humans use on a daily basis, and it is a means by which computer systems use this language to transmit information to users.

[0157] This system uses smart devices, a dedicated application, an external server, a generative AI model, and an emotion engine to assess the user's oral health and provide a care plan tailored to their emotional state.

[0158] The user utilizes an application installed on their smart device. This application uses the device's camera and microphone to acquire image data of the oral cavity and audio data of brushing. The acquired data is temporarily stored on the device, encrypted using technologies such as AES encryption, and then securely transmitted to an external server.

[0159] The server utilizes generative AI models to analyze the received data. Specifically, it uses machine learning libraries such as TENSORFLOW® and PyTorch to evaluate the user's gum condition and analyze the accuracy and intensity of brushing from the audio data.

[0160] Furthermore, the server implements an emotion engine that recognizes the user's emotional state by analyzing voice tone and facial expression data. This recognition uses OpenCV and NLP techniques to extract features from voice and images and determine stress and relaxation levels.

[0161] The server integrates this data to comprehensively evaluate the user's oral health and emotional state. Then, in the process of generating the optimal care plan for the user, it uses natural language generation technology to create and deliver notifications that are easy for the user to understand.

[0162] For example, if a user is experiencing stress and is at risk of periodontal disease, the server will suggest relaxation techniques along with specific advice to enhance daily self-care. This notification can include a relaxed tone and encouraging words.

[0163] For example, the prompt can be set as follows:

[0164] "Analyze the user's oral cavity images and voice data to generate a customized care plan that takes into account their health status and emotions, and notify them in natural language."

[0165] In this way, the present invention provides more effective and personalized care services.

[0166] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0167] Step 1:

[0168] The user launches the app, takes an image of their oral cavity with their smart device's camera, and records brushing sounds with the microphone. The input consists of image and audio data of the user's oral cavity. This data acquisition provides basic data necessary to understand the user's current condition.

[0169] Step 2:

[0170] The device temporarily stores the image and audio data it acquires and processes the data securely using AES encryption. The input is the image and audio data acquired in step 1, and the output is encrypted data. This prepares the device to securely transmit user data externally.

[0171] Step 3:

[0172] The terminal sends encrypted data to an external server. The input is the encrypted data, and the output is the completion of the data transmission to the server. The terminal's communication function plays a crucial role here, and the data is transmitted using protocols such as TLS.

[0173] Step 4:

[0174] The server decodes the received data and analyzes it through a generative AI model. Here, TensorFlow and PyTorch are used to evaluate the condition of the gums from image data and analyze the quality of brushing from audio data. The input is decoded image and audio data, and the output is the analysis result indicating the user's oral health status.

[0175] Step 5:

[0176] The server uses an emotion engine to analyze voice tone and facial image data to determine the user's emotional state. The input is the same decoded voice and image data, and the output is an evaluation of the user's emotional state. The server uses the engine to analyze stress and relaxation levels.

[0177] Step 6:

[0178] The server integrates health and emotional state data and generates an optimal care plan using an AI model. A personalized care plan is created using natural language generation technology and communicated to the user in the form of a notification. The input is the analysis results of the health state and the evaluation of the emotional state, and the output is a customized care plan. This process uses prompts to create notifications that are easy for the user to remember.

[0179] (Application Example 2)

[0180] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0181] In modern society, maintaining oral and mental health is crucial, yet they often influence each other. In particular, there is a lack of technology to mitigate the impact of stress on oral health while providing effective care plans tailored to individual circumstances. This invention aims to comprehensively analyze a user's oral and emotional state and provide appropriate care while reducing stress-related risks.

[0182] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0183] In this invention, the server includes information acquisition means for acquiring the user's oral cavity condition as image and audio information, information transmission means for encrypting the acquired information and transmitting it to an external processing device, and analysis means for analyzing the oral health condition and stress level based on the received information and the user's emotional state. This makes it possible to effectively provide the user with an individualized care plan and stress management measures.

[0184] "Information acquisition means" refers to a device equipped with the function of acquiring the user's oral cavity condition as image and audio information.

[0185] "Information transmission means" refers to a device equipped with the function of encrypting acquired information and transmitting it to an external processing device.

[0186] The "analysis means" is equipped with the function to analyze the user's oral health status based on the received information and predict the risk of periodontal disease, and further includes the function to identify the user's emotional state by recognizing emotions and to evaluate the stress level.

[0187] A "notification mechanism" is a system equipped with the functionality to send notifications to the user when an anomaly is detected based on the analysis results and emotional state.

[0188] The "plan generation means" is a system that takes into account the user's lifestyle information to generate an individualized oral care plan, and also has the function of providing it in a format that is easy for the user to understand using natural language generation technology.

[0189] A "suggestion tool" is a system equipped with functions to provide relaxation methods and security advice tailored to different stress levels.

[0190] This invention is a system that analyzes the user's oral cavity condition and emotional state, and provides personalized oral care and stress reduction measures. This system includes information acquisition means, information transmission means, analysis means, notification means, plan generation means, and proposal means.

[0191] The system uses a smartphone or other device to periodically acquire image and audio information of the inside of the mouth. It utilizes the camera and microphone of the smart device to acquire data via a dedicated application. The acquired data is encrypted on the spot and sent to an external processing unit, i.e., a server.

[0192] The server is equipped with a generative AI model for analyzing received data. This AI model analyzes the user's oral health status and predicts the risk of periodontal disease. It also has an emotion engine for emotion recognition, which evaluates the user's emotional state, particularly stress level, based on voice tone and facial expression data. This utilizes speech recognition and facial recognition technologies.

[0193] Based on the analysis results, the server sends a notification to the user. The notification includes a user-friendly message created using natural language generation technology, presenting a specific care plan along with relaxation techniques and security advice. This process utilizes a text generation API to provide user-friendly information in real time.

[0194] For example, if analysis reveals a user is at risk of periodontal disease and also identifies a high-stress state, the server sends a notification that includes advice on password management, along with simple stretching and deep breathing exercises the user can perform. In this way, the present invention comprehensively supports the user's oral and mental health.

[0195] An example of a prompt message for the generating AI model would be: "Please describe the security risks and countermeasures based on the user's oral health and emotional analysis results. Please include specific relaxation methods and security measures in your suggestions."

[0196] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0197] Step 1:

[0198] The user uses a device to acquire images and audio information of the oral cavity. Input is real-time data from the smart device's camera and microphone. This data is temporarily stored on the device and converted to an appropriate format. Output is an encrypted data file.

[0199] Step 2:

[0200] The terminal sends the encrypted data obtained in step 1 to the server, which is an external processing unit, using the information transmission means. The input is the encrypted data file, and the output is the completion of the data transfer to the server.

[0201] Step 3:

[0202] The server processes the received data using analytical tools. The input is oral cavity data transmitted from the terminal, and the analysis uses a generative AI model to evaluate oral health status and predict the risk of periodontal disease. The output is the risk assessment data of the analysis results.

[0203] Step 4:

[0204] The server uses an emotion engine to analyze voice tone and facial data to recognize the user's emotional state. The input is the same data as in step 3, and the output is emotion evaluation data indicating the user's stress level. Voice analysis and facial expression recognition technology are used for data processing.

[0205] Step 5:

[0206] The server sends a notification to the user using a notification system based on the analysis results and sentiment evaluation data. The input is the output data from steps 3 and 4, and an appropriate message is generated using natural language generation. The output is a notification that includes specific care and advice for the user.

[0207] Step 6:

[0208] Based on the notifications received, the user performs suggested relaxation techniques and security advice. The input is the notification message from step 5, and the output is the user's behavioral changes and stress reduction. This step includes specific actions such as deep breathing, stretching, and reviewing security settings.

[0209] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0210] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0211] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0212] [Second Embodiment]

[0213] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0214] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0215] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0216] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0217] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0218] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0219] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0220] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0221] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0222] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0223] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0224] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0225] This invention is embodied as a system that periodically records the condition of the oral cavity using the user's smart device, analyzes the data using AI, and detects the risk of periodontal disease early, thereby providing preventive measures. This system mainly consists of the user's smart device, a server that processes the data, and a communication infrastructure that connects them.

[0226] Users capture images of their oral cavity and acquire audio data using a dedicated application on their smart device. This data is temporarily stored on the device and transmitted to a server using a secure communication method. The server employs a generative AI model that utilizes machine learning algorithms to analyze the received data. This AI analyzes the condition of the gums in detail from the image data and detects abnormalities in oral sounds using the audio data.

[0227] Based on the analysis results, the server assesses the user's periodontal disease risk and sends a notification to the device according to the risk level. This notification includes information about the area where abnormalities were detected, the degree of risk, and specific countermeasures. The device displays the received information to the user in an easy-to-understand visual format and provides guidance on appropriate care.

[0228] Furthermore, the server uses natural language generation technology to create a personalized oral care plan, taking into account the user's saved lifestyle data. This plan supports users in practicing self-care without getting lost due to lack of knowledge or time.

[0229] For example, if gum swelling is detected in the early stages from daily image data, the server notifies the terminal of the risk. Based on the user's eating habits, it provides advice on reducing excessive sugar intake and information on recommended oral care products. In this way, the present invention provides an oral care service that is useful for users to proactively manage their health and reduce the risk of developing periodontal disease.

[0230] The following describes the processing flow.

[0231] Step 1:

[0232] The user launches a dedicated app on their smart device. The app instructs the user to take images of their oral cavity and simultaneously records necessary audio data. This obtains data about the user's current oral condition.

[0233] Step 2:

[0234] The device temporarily stores the acquired image and audio data and encrypts them. Encryption is performed to ensure the confidentiality of the data.

[0235] Step 3:

[0236] Encrypted data is sent from the terminal to the server. A secure communication protocol reduces the risk of data leakage during transit.

[0237] Step 4:

[0238] The server uses a generative AI model to analyze the data it receives. The AI ​​analyzes the condition of the gums from image data and detects abnormal patterns. It also detects abnormalities from audio data based on brushing sounds and oral sounds.

[0239] Step 5:

[0240] The server evaluates the user's periodontal disease risk based on the analysis results and determines the risk level. The server then prepares to send an alert to the terminal according to the risk level.

[0241] Step 6:

[0242] The device receives alerts sent from the server and displays them visually to the user. The display includes detailed analysis results and recommended actions. The user can then take self-care actions based on this information.

[0243] Step 7:

[0244] Furthermore, the server generates a personalized oral care plan, taking into account the user's lifestyle and daily data. It utilizes natural language generation technology to send the information to the device in an easy-to-understand format.

[0245] (Example 1)

[0246] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0247] Continuously and regularly monitoring oral health and detecting the risk of periodontal disease early is a challenging task for individual users. Furthermore, providing appropriate oral care plans tailored to individual lifestyles and health conditions is essential for effective self-care. However, conventional methods lacked systems to address these challenges.

[0248] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0249] This invention includes a server that uses a machine learning model to analyze the user's oral health status and predict and evaluate the risk of periodontal disease; a server that sends a notification to the user if an abnormality is detected based on the analysis results; and a server that generates an individualized oral care plan considering the user's lifestyle information. This allows the user to easily understand the condition of their own oral cavity and receive specific health management and preventive measures tailored to their lifestyle.

[0250] "Data acquisition means" refers to a device or method for acquiring the condition of a user's oral cavity as image data and audio data.

[0251] "Data transmission means" refers to a device or method for encrypting acquired image data and audio data and securely transmitting them to an external processing device.

[0252] "Analysis means" refers to a device or method for analyzing the user's oral health status using a machine learning model based on received data, and for predicting and evaluating the risk of periodontal disease.

[0253] "Notification means" refers to a device or method for transmitting information to the user when an anomaly is detected based on the analysis results.

[0254] "Plan generation means" refers to a device or method for generating an individualized oral care plan while taking into account the user's lifestyle information.

[0255] A "machine learning model" is a mathematical model used to compare past and present data, recognize patterns, and make predictions about new data.

[0256] "Natural language generation technology" is a technology that enables computers to generate text in a user-friendly format based on text data.

[0257] This invention is a system that utilizes a user's smart device to periodically acquire images and audio data from inside the oral cavity, analyzes this data, and provides an individualized oral care plan. It mainly consists of a smart device, a server, and a communication network.

[0258] Users install a dedicated application on their smart device and use it to capture images of their oral cavity and collect audio data. The device's built-in camera and microphone are used to acquire both image and audio data. The acquired data is temporarily stored on the smart device and then transmitted to the server using secure protocols such as SSL / TLS.

[0259] The server performs detailed analysis on the received data using a specific machine learning model. This generative AI model has the ability to identify the condition of the gums from image data and detect abnormalities in oral sounds from audio data. This allows for the prediction of periodontal disease risk and enables early intervention as needed.

[0260] Based on the analysis, the server generates information about the user's health status and sends notifications to the user's smart device according to the risk level. These notifications include the specific location and severity of any abnormalities, as well as recommended countermeasures. The smart device also presents the information visually in an easy-to-understand manner, supporting intuitive understanding for the user.

[0261] Furthermore, this system uses natural language generation technology to create a personalized oral care plan, referencing the user's lifestyle information. This plan serves as a guide for self-care that the user can implement on a daily basis.

[0262] For example, if slight gum swelling is detected from captured image data, the server will notify the user of the risk and send advice to reduce sugar intake as a way to improve their diet. It will also provide product information that can be used for actual self-care.

[0263] An example of a prompt message is: "Analyze this user's most recent intraoral image and assess their health status. If any abnormalities are found, indicate their location and risk level, and suggest specific corrective measures."

[0264] In this way, the present invention operates as a system that supports users' autonomy in health management and plays a role in reducing the risk of periodontal disease.

[0265] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0266] Step 1:

[0267] The user launches an application on their smart device, takes an image of their mouth, and records audio. The input consists of image and audio data acquired using the smart device's camera and microphone. Specifically, the user takes a picture of their mouth with the camera and taps the audio input button to record audio. The output—image and audio data—is saved on the smart device.

[0268] Step 2:

[0269] The terminal encrypts the acquired data and sends it to the server. The input consists of image data and audio data stored on the terminal. As part of the data processing, the data is encrypted using the SSL / TLS protocol. Specifically, the terminal executes the data encryption process and establishes a communication channel with the server. As output, the encrypted data is sent to the server via a secure communication path.

[0270] Step 3:

[0271] The server prepares the received data for analysis. The input consists of encrypted image and audio data sent to the server. The data processing involves decrypting the data and converting it into an analyzable format. The output is data suitable for analysis.

[0272] Step 4:

[0273] The server activates a generation AI model and analyzes image and audio data using prompt text as input. For data processing, an image analysis algorithm examines the condition of the gums, and an audio analysis algorithm checks for abnormalities in the audio data. Specifically, the AI ​​model identifies abnormal areas in the image and identifies abnormal patterns in the audio. The output provides analysis results regarding health status and risk levels.

[0274] Step 5:

[0275] The server evaluates the user's periodontal disease risk based on the analysis results, creates a notification message, and sends it to the terminal. The input is the analysis results from the AI ​​model. As data processing, it performs specific actions to generate a message according to the risk level. As output, the notification message is sent to the user's terminal.

[0276] Step 6:

[0277] The terminal displays the received notification to the user. The input is the notification message sent from the server. As a specific operation, the terminal visually presents the message to the user using the display function. As output, information that serves as a guide for the user's next action is provided.

[0278] Step 7:

[0279] The server creates an oral care plan personalized with a generation AI model using the user's lifestyle information. The input is the user's lifestyle data and the analysis results of the AI model. As a specific operation, the care plan is written using natural language generation technology. As output, an oral care plan specialized for the user is generated and sent to the terminal.

[0280] (Application Example 1)

[0281] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0282] Although it is important to appropriately manage the oral health condition and detect the risk of periodontal disease at an early stage, it is not easy to provide specific care proposals based on the individual health condition and lifestyle of each user in real time. Also, in actual stores, there is a lack of means to provide useful oral care information to consumers on the spot and promote the use of products.

[0283] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following respective means.

[0284] In this invention, the server includes data collection means, information transmission means, analysis means, information presentation means, and plan generation means. Thereby, the user can grasp the health condition of his / her oral cavity via his / her own smart device or the video device with voice output function in the actual store, and receive an individually optimized oral care plan in real time.

[0285] The "data collection means" is a device or method for acquiring the oral cavity state of a user as image information and voice information.

[0286] The "information transmission means" is a device or technology for encrypting the collected oral cavity information and transmitting it to an external information processing device.

[0287] The "analysis means" is a technology or device for analyzing the received information, evaluating the oral cavity health state of the user, and predicting the risk of periodontal disease.

[0288] The "notification means" is a device or method for transmitting a notification to the user when an abnormality is detected based on the analysis result.

[0289] The "care plan generation means" is a technology or device for generating an individual oral care plan in consideration of the user's lifestyle pattern information.

[0290] The "information presentation means" is a technology or device for visually and auditorily providing the analysis result and the recommended oral care method to the user by using a video device with a voice output function installed in a physical store.

[0291] This invention constructs a system for monitoring the oral cavity health state and providing an individual care plan to the user. The system utilizes the user's smart device and the digital signage installed in the physical store to acquire information in real time and provide the analysis result and the care plan.

[0292] <00009​​

[0293] The analysis results and care plan are not only notified to the user's smart device, but are also provided as visual and auditory information on video devices with audio output capabilities installed in physical stores. This allows users to receive consistent care advice both at home and in stores.

[0294] For example, a customer who undergoes an oral check at a drugstore might receive a notification about their periodontal disease risk on the spot, prompting them to immediately purchase appropriate oral care products. Another example of a prompt message for the generating AI model is: "Analyze the oral images, assess the periodontal disease risk, and generate recommended care methods based on the risk level."

[0295] This system enhances user convenience and streamlines the maintenance and management of oral health.

[0296] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0297] Step 1:

[0298] The user uses a dedicated application on a smart device to acquire image and audio information from inside their mouth. The user takes pictures of the inside of their mouth with a camera and collects audio data through a microphone. The input is images and audio information of the user's mouth, and the output is this data.

[0299] Step 2:

[0300] The device encrypts the image and audio information it acquires. The device uses encryption technologies such as the SSL / TLS protocol to ensure data security. The input to this process is the captured image and audio information, and the output is the encrypted image and audio information.

[0301] Step 3:

[0302] The terminal sends the encrypted data to the server on the cloud. The terminal uses secure communication means to send the encrypted data to the default server address. The input is encrypted image information and voice information, and the output is data transmission to the server.

[0303] Step 4:

[0304] The server analyzes the received data using a generative AI model based on a machine learning algorithm. The server utilizes the generative AI model to analyze the state of the gums from the image information and detect abnormalities in the voice from the voice information. The input of this process is the decryption result of the received encrypted data, and the output is the analysis result indicating the user's oral health status.

[0305] Step 5:

[0306] The server generates the user's oral care plan based on the analysis result and creates a notification. The server generates a notification considering an individual care plan based on the user's lifestyle and health data, and converts it into text format. The input of this process is the analysis result and the user's lifestyle information, and the output is the individualized care plan and notification message.

[0307] Step 6:

[0308] The server sends the generated notification message to the terminal and the video device in the physical store. The server sends the notification to the terminal via SMS, email, or a dedicated app, and provides it to the user visually and auditorily in the physical store. The input of this process is the generated care plan and notification message, and the output is the information displayed on the terminal screen and the video device.

[0309] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion recognition model 59 and perform specific processing using the user's emotion.

[0310] \( \)This invention is a system for understanding the user's oral health status and providing periodontal disease prevention measures. By combining it with an emotional engine, it realizes customized care that responds to the user's emotional state.

[0311] The user periodically acquires images and audio data of their oral cavity using a dedicated application installed on their smart device. This data is temporarily stored on the device, encrypted, and then sent to a server. The server uses a generative AI model to analyze the received data, examining the condition of the user's gums and the sounds of their brushing.

[0312] Furthermore, this system incorporates an emotion engine that can recognize the user's emotional state. The emotion engine identifies the user's emotions from factors such as voice tone and facial expressions, and evaluates their motivation and stress level regarding oral care. For example, if the system determines that the user is experiencing stress, the plan generation mechanism will provide advice and product suggestions to alleviate that stress.

[0313] The server integrates and analyzes the user's emotional state and oral health data to generate an optimal care plan. Using natural language generation technology, it sends notifications in a format easily understood by the user. These notifications can be emotionally sensitive and include a relaxed tone and encouraging words.

[0314] For example, if user data indicates a risk of periodontal disease and stress levels are also identified, the server will select an approach that promotes self-care in a calming atmosphere. It will also support the user's periodontal disease prevention efforts by recommending simple and effective care methods. In this way, the present invention provides a more effective and personalized care service that comprehensively considers the user's emotions and oral health status.

[0315] The following describes the processing flow.

[0316] Step 1:

[0317] The user launches a dedicated app on their smart device and takes a picture of the inside of their mouth. The app instructs the user to capture the image and record audio, thereby obtaining detailed data about the inside of their mouth.

[0318] Step 2:

[0319] The device encrypts the acquired data and sends it to the server while protecting privacy. Secure protocols such as TLS are used in this process.

[0320] Step 3:

[0321] The server analyzes the received data. Image analysis algorithms are used to assess the condition of the gums, and potential problems are detected from the audio data. At this stage, foundational data for assessing periodontal disease risk is generated.

[0322] Step 4:

[0323] The emotion engine analyzes transmitted voice tone and facial expression data to recognize the user's emotional state. This identifies the user's current mood and stress level.

[0324] Step 5:

[0325] The server integrates the analysis results, combining emotional state and health data to more precisely assess the user's periodontal disease risk. If necessary, it detects anomalies in areas requiring specific attention.

[0326] Step 6:

[0327] A notification is sent from the server to the device. This notification includes the results of the periodontal disease risk assessment, along with personalized advice that takes into account the user's emotional state.

[0328] Step 7:

[0329] The plan generation system considers the user's lifestyle and emotional state, uses natural language generation technology to generate a specific self-care plan, and sends it to the device. The device then presents this plan to the user, supporting their oral care efforts.

[0330] (Example 2)

[0331] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0332] Conventional oral care support systems provide care plans without considering the user's emotional state, which can lead to insufficient user motivation and engagement with care. Therefore, there is a need to provide more effective care plans that take the user's emotional state into account.

[0333] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0334] In this invention, the server includes means for recognizing the user's emotional state from voice and image data, means for integrating and analyzing the emotional state and oral health status to generate an individualized care plan, and means for notifying the user of the generated care plan in natural language. This makes it possible to provide a care plan that takes the user's emotions into consideration.

[0335] A "user" is an individual who uses the system to provide oral care data and receive services.

[0336] "Oral condition" refers to information related to the user's oral health, specifically the condition of the gums and teeth, which is acquired as digital data.

[0337] "Image data" refers to a digital recording of visual information about the user's oral cavity.

[0338] "Audio data" refers to digital recordings of sounds produced when a user performs actions such as brushing their teeth.

[0339] "Device" is a general term for hardware and software used to acquire, transmit, and analyze data.

[0340] "Encryption" is the process of transforming the content of digital data using a specific algorithm in order to transmit it securely.

[0341] An "external device" is another computer system connected via a network for the purpose of processing and storing data.

[0342] "Analysis" is the process of examining the acquired data in detail and evaluating the user's oral health status.

[0343] "Risk" is an indicator that shows the potential for problems with the user's oral health.

[0344] "Notifications" refer to information that communicates analysis results or care plans to the user visually or audibly.

[0345] "Emotional state" refers to the user's psychological state, including their stress levels and relaxation levels.

[0346] "Creation" refers to the process of developing a care plan that is tailored to a specific purpose and is based on the user's characteristics.

[0347] "Natural language" refers to the language that humans use on a daily basis, and it is a means by which computer systems use this language to transmit information to users.

[0348] This system uses smart devices, a dedicated application, an external server, a generative AI model, and an emotion engine to assess the user's oral health and provide a care plan tailored to their emotional state.

[0349] The user utilizes an application installed on their smart device. This application uses the device's camera and microphone to acquire image data of the oral cavity and audio data of brushing. The acquired data is temporarily stored on the device, encrypted using technologies such as AES encryption, and then securely transmitted to an external server.

[0350] The server utilizes generative AI models to analyze the received data. Specifically, it uses machine learning libraries such as TensorFlow and PyTorch to evaluate the user's gum condition and analyze the accuracy and intensity of brushing from the audio data.

[0351] Furthermore, the server implements an emotion engine that recognizes the user's emotional state by analyzing voice tone and facial expression data. This recognition uses OpenCV and NLP techniques to extract features from voice and images and determine stress and relaxation levels.

[0352] The server integrates this data to comprehensively evaluate the user's oral health and emotional state. Then, in the process of generating the optimal care plan for the user, it uses natural language generation technology to create and deliver notifications that are easy for the user to understand.

[0353] For example, if a user is experiencing stress and is at risk of periodontal disease, the server will suggest relaxation techniques along with specific advice to enhance daily self-care. This notification can include a relaxed tone and encouraging words.

[0354] For example, the prompt can be set as follows:

[0355] "Analyze the user's oral cavity images and voice data to generate a customized care plan that takes into account their health status and emotions, and notify them in natural language."

[0356] In this way, the present invention provides more effective and personalized care services.

[0357] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0358] Step 1:

[0359] The user launches the app, takes an image of their oral cavity with their smart device's camera, and records brushing sounds with the microphone. The input consists of image and audio data of the user's oral cavity. This data acquisition provides basic data necessary to understand the user's current condition.

[0360] Step 2:

[0361] The device temporarily stores the image and audio data it acquires and processes the data securely using AES encryption. The input is the image and audio data acquired in step 1, and the output is encrypted data. This prepares the device to securely transmit user data externally.

[0362] Step 3:

[0363] The terminal sends encrypted data to an external server. The input is the encrypted data, and the output is the completion of the data transmission to the server. The terminal's communication function plays a crucial role here, and the data is transmitted using protocols such as TLS.

[0364] Step 4:

[0365] The server decodes the received data and analyzes it through a generative AI model. Here, TensorFlow and PyTorch are used to evaluate the condition of the gums from image data and analyze the quality of brushing from audio data. The input is decoded image and audio data, and the output is the analysis result indicating the user's oral health status.

[0366] Step 5:

[0367] The server uses an emotion engine to analyze voice tone and facial image data to determine the user's emotional state. The input is the same decoded voice and image data, and the output is an evaluation of the user's emotional state. The server uses the engine to analyze stress and relaxation levels.

[0368] Step 6:

[0369] The server integrates health and emotional state data and generates an optimal care plan using an AI model. A personalized care plan is created using natural language generation technology and communicated to the user in the form of a notification. The input is the analysis results of the health state and the evaluation of the emotional state, and the output is a customized care plan. This process uses prompts to create notifications that are easy for the user to remember.

[0370] (Application Example 2)

[0371] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0372] In modern society, maintaining oral and mental health is crucial, yet they often influence each other. In particular, there is a lack of technology to mitigate the impact of stress on oral health while providing effective care plans tailored to individual circumstances. This invention aims to comprehensively analyze a user's oral and emotional state and provide appropriate care while reducing stress-related risks.

[0373] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0374] In this invention, the server includes information acquisition means for acquiring the user's oral cavity condition as image and audio information, information transmission means for encrypting the acquired information and transmitting it to an external processing device, and analysis means for analyzing the oral health condition and stress level based on the received information and the user's emotional state. This makes it possible to effectively provide the user with an individualized care plan and stress management measures.

[0375] "Information acquisition means" refers to a device equipped with the function of acquiring the user's oral cavity condition as image and audio information.

[0376] "Information transmission means" refers to a device equipped with the function of encrypting acquired information and transmitting it to an external processing device.

[0377] The "analysis means" is equipped with the function to analyze the user's oral health status based on the received information and predict the risk of periodontal disease, and further includes the function to identify the user's emotional state by recognizing emotions and to evaluate the stress level.

[0378] A "notification mechanism" is a system equipped with the functionality to send notifications to the user when an anomaly is detected based on the analysis results and emotional state.

[0379] The "plan generation means" is a system that takes into account the user's lifestyle information to generate an individualized oral care plan, and also has the function of providing it in a format that is easy for the user to understand using natural language generation technology.

[0380] A "suggestion tool" is a system equipped with functions to provide relaxation methods and security advice tailored to different stress levels.

[0381] This invention is a system that analyzes the user's oral cavity condition and emotional state, and provides personalized oral care and stress reduction measures. This system includes information acquisition means, information transmission means, analysis means, notification means, plan generation means, and proposal means.

[0382] The system uses a smartphone or other device to periodically acquire image and audio information of the inside of the mouth. It utilizes the camera and microphone of the smart device to acquire data via a dedicated application. The acquired data is encrypted on the spot and sent to an external processing unit, i.e., a server.

[0383] The server is equipped with a generative AI model for analyzing received data. This AI model analyzes the user's oral health status and predicts the risk of periodontal disease. It also has an emotion engine for emotion recognition, which evaluates the user's emotional state, particularly stress level, based on voice tone and facial expression data. This utilizes speech recognition and facial recognition technologies.

[0384] Based on the analysis results, the server sends a notification to the user. The notification includes a user-friendly message created using natural language generation technology, presenting a specific care plan along with relaxation techniques and security advice. This process utilizes a text generation API to provide user-friendly information in real time.

[0385] For example, if analysis reveals a user is at risk of periodontal disease and also identifies a high-stress state, the server sends a notification that includes advice on password management, along with simple stretching and deep breathing exercises the user can perform. In this way, the present invention comprehensively supports the user's oral and mental health.

[0386] An example of a prompt message for the generating AI model would be: "Please describe the security risks and countermeasures based on the user's oral health and emotional analysis results. Please include specific relaxation methods and security measures in your suggestions."

[0387] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0388] Step 1:

[0389] The user uses a device to acquire images and audio information of the oral cavity. Input is real-time data from the smart device's camera and microphone. This data is temporarily stored on the device and converted to an appropriate format. Output is an encrypted data file.

[0390] Step 2:

[0391] The terminal sends the encrypted data obtained in step 1 to the server, which is an external processing unit, using the information transmission means. The input is the encrypted data file, and the output is the completion of the data transfer to the server.

[0392] Step 3:

[0393] The server processes the received data using analytical tools. The input is oral cavity data transmitted from the terminal, and the analysis uses a generative AI model to evaluate oral health status and predict the risk of periodontal disease. The output is the risk assessment data of the analysis results.

[0394] Step 4:

[0395] The server uses an emotion engine to analyze voice tone and facial data to recognize the user's emotional state. The input is the same data as in step 3, and the output is emotion evaluation data indicating the user's stress level. Voice analysis and facial expression recognition technology are used for data processing.

[0396] Step 5:

[0397] The server sends a notification to the user using a notification system based on the analysis results and sentiment evaluation data. The input is the output data from steps 3 and 4, and an appropriate message is generated using natural language generation. The output is a notification that includes specific care and advice for the user.

[0398] Step 6:

[0399] Based on the notifications received, the user performs suggested relaxation techniques and security advice. The input is the notification message from step 5, and the output is the user's behavioral changes and stress reduction. This step includes specific actions such as deep breathing, stretching, and reviewing security settings.

[0400] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0401] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0402] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0403] [Third Embodiment]

[0404] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0405] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0406] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0407] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0408] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0409] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0410] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0411] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0412] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0413] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0414] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0415] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0416] This invention is embodied as a system that periodically records the condition of the oral cavity using the user's smart device, analyzes the data using AI, and detects the risk of periodontal disease early, thereby providing preventive measures. This system mainly consists of the user's smart device, a server that processes the data, and a communication infrastructure that connects them.

[0417] Users capture images of their oral cavity and acquire audio data using a dedicated application on their smart device. This data is temporarily stored on the device and transmitted to a server using a secure communication method. The server employs a generative AI model that utilizes machine learning algorithms to analyze the received data. This AI analyzes the condition of the gums in detail from the image data and detects abnormalities in oral sounds using the audio data.

[0418] Based on the analysis results, the server assesses the user's periodontal disease risk and sends a notification to the device according to the risk level. This notification includes information about the area where abnormalities were detected, the degree of risk, and specific countermeasures. The device displays the received information to the user in an easy-to-understand visual format and provides guidance on appropriate care.

[0419] Furthermore, the server uses natural language generation technology to create a personalized oral care plan, taking into account the user's saved lifestyle data. This plan supports users in practicing self-care without getting lost due to lack of knowledge or time.

[0420] For example, if gum swelling is detected in the early stages from daily image data, the server notifies the terminal of the risk. Based on the user's eating habits, it provides advice on reducing excessive sugar intake and information on recommended oral care products. In this way, the present invention provides an oral care service that is useful for users to proactively manage their health and reduce the risk of developing periodontal disease.

[0421] The following describes the processing flow.

[0422] Step 1:

[0423] The user launches a dedicated app on their smart device. The app instructs the user to take images of their oral cavity and simultaneously records necessary audio data. This obtains data about the user's current oral condition.

[0424] Step 2:

[0425] The device temporarily stores the acquired image and audio data and encrypts them. Encryption is performed to ensure the confidentiality of the data.

[0426] Step 3:

[0427] Encrypted data is sent from the terminal to the server. A secure communication protocol reduces the risk of data leakage during transit.

[0428] Step 4:

[0429] The server uses a generative AI model to analyze the data it receives. The AI ​​analyzes the condition of the gums from image data and detects abnormal patterns. It also detects abnormalities from audio data based on brushing sounds and oral sounds.

[0430] Step 5:

[0431] The server evaluates the user's periodontal disease risk based on the analysis results and determines the risk level. The server then prepares to send an alert to the terminal according to the risk level.

[0432] Step 6:

[0433] The device receives alerts sent from the server and displays them visually to the user. The display includes detailed analysis results and recommended actions. The user can then take self-care actions based on this information.

[0434] Step 7:

[0435] Furthermore, the server generates a personalized oral care plan, taking into account the user's lifestyle and daily data. It utilizes natural language generation technology to send the information to the device in an easy-to-understand format.

[0436] (Example 1)

[0437] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0438] Continuously and regularly monitoring oral health and detecting the risk of periodontal disease early is a challenging task for individual users. Furthermore, providing appropriate oral care plans tailored to individual lifestyles and health conditions is essential for effective self-care. However, conventional methods lacked systems to address these challenges.

[0439] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0440] This invention includes a server that uses a machine learning model to analyze the user's oral health status and predict and evaluate the risk of periodontal disease; a server that sends a notification to the user if an abnormality is detected based on the analysis results; and a server that generates an individualized oral care plan considering the user's lifestyle information. This allows the user to easily understand the condition of their own oral cavity and receive specific health management and preventive measures tailored to their lifestyle.

[0441] "Data acquisition means" refers to a device or method for acquiring the condition of a user's oral cavity as image data and audio data.

[0442] "Data transmission means" refers to a device or method for encrypting acquired image data and audio data and securely transmitting them to an external processing device.

[0443] "Analysis means" refers to a device or method for analyzing the user's oral health status using a machine learning model based on received data, and for predicting and evaluating the risk of periodontal disease.

[0444] "Notification means" refers to a device or method for transmitting information to the user when an anomaly is detected based on the analysis results.

[0445] "Plan generation means" refers to a device or method for generating an individualized oral care plan while taking into account the user's lifestyle information.

[0446] A "machine learning model" is a mathematical model used to compare past and present data, recognize patterns, and make predictions about new data.

[0447] "Natural language generation technology" is a technology that enables computers to generate text in a user-friendly format based on text data.

[0448] This invention is a system that utilizes a user's smart device to periodically acquire images and audio data from inside the oral cavity, analyzes this data, and provides an individualized oral care plan. It mainly consists of a smart device, a server, and a communication network.

[0449] Users install a dedicated application on their smart device and use it to capture images of their oral cavity and collect audio data. The device's built-in camera and microphone are used to acquire both image and audio data. The acquired data is temporarily stored on the smart device and then transmitted to the server using secure protocols such as SSL / TLS.

[0450] The server performs detailed analysis on the received data using a specific machine learning model. This generative AI model has the ability to identify the condition of the gums from image data and detect abnormalities in oral sounds from audio data. This allows for the prediction of periodontal disease risk and enables early intervention as needed.

[0451] Based on the analysis, the server generates information about the user's health status and sends notifications to the user's smart device according to the risk level. These notifications include the specific location and severity of any abnormalities, as well as recommended countermeasures. The smart device also presents the information visually in an easy-to-understand manner, supporting intuitive understanding for the user.

[0452] Furthermore, this system uses natural language generation technology to create a personalized oral care plan, referencing the user's lifestyle information. This plan serves as a guide for self-care that the user can implement on a daily basis.

[0453] For example, if slight gum swelling is detected from captured image data, the server will notify the user of the risk and send advice to reduce sugar intake as a way to improve their diet. It will also provide product information that can be used for actual self-care.

[0454] An example of a prompt message is: "Analyze this user's most recent intraoral image and assess their health status. If any abnormalities are found, indicate their location and risk level, and suggest specific corrective measures."

[0455] In this way, the present invention operates as a system that supports users' autonomy in health management and plays a role in reducing the risk of periodontal disease.

[0456] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0457] Step 1:

[0458] The user launches an application on their smart device, takes an image of their mouth, and records audio. The input consists of image and audio data acquired using the smart device's camera and microphone. Specifically, the user takes a picture of their mouth with the camera and taps the audio input button to record audio. The output—image and audio data—is saved on the smart device.

[0459] Step 2:

[0460] The terminal encrypts the acquired data and sends it to the server. The input consists of image data and audio data stored on the terminal. As part of the data processing, the data is encrypted using the SSL / TLS protocol. Specifically, the terminal executes the data encryption process and establishes a communication channel with the server. As output, the encrypted data is sent to the server via a secure communication path.

[0461] Step 3:

[0462] The server prepares the received data for analysis. The input consists of encrypted image and audio data sent to the server. The data processing involves decrypting the data and converting it into an analyzable format. The output is data suitable for analysis.

[0463] Step 4:

[0464] The server activates a generation AI model and analyzes image and audio data using prompt text as input. For data processing, an image analysis algorithm examines the condition of the gums, and an audio analysis algorithm checks for abnormalities in the audio data. Specifically, the AI ​​model identifies abnormal areas in the image and identifies abnormal patterns in the audio. The output provides analysis results regarding health status and risk levels.

[0465] Step 5:

[0466] The server evaluates the user's periodontal disease risk based on the analysis results, creates a notification message, and sends it to the terminal. The input is the analysis results from the AI ​​model. As data processing, it performs specific actions to generate a message according to the risk level. As output, the notification message is sent to the user's terminal.

[0467] Step 6:

[0468] The terminal displays received notifications to the user. The input is the notification message sent from the server. Specifically, the terminal uses its display function to visually present the message to the user. The output provides information that guides the user's next course of action.

[0469] Step 7:

[0470] The server uses the user's lifestyle information to create a personalized oral care plan using a generative AI model. The input consists of the user's lifestyle data and the results of the AI ​​model's analysis. Specifically, natural language generation technology is used to create the care plan as text. The output is a user-specific oral care plan, which is then sent to the terminal.

[0471] (Application Example 1)

[0472] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0473] While it is important to properly manage oral health and detect the risk of periodontal disease early, providing specific care suggestions based on each user's health condition and lifestyle in real time is not easy. Furthermore, even in physical stores, there is a lack of means to provide consumers with useful oral care information on the spot and promote product use.

[0474] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0475] In this invention, the server includes data collection means, information transmission means, analysis means, information presentation means, and plan generation means. This allows users to understand their oral health status and receive individually optimized oral care plans in real time via their smart devices or video devices with audio output capabilities in physical stores.

[0476] "Data collection means" refers to devices and methods for acquiring the user's oral cavity condition as image and audio information.

[0477] "Information transmission means" refers to devices or technologies for encrypting collected oral cavity information and transmitting it to an external information processing device.

[0478] "Analysis means" refers to a technology or device for analyzing received information, evaluating the user's oral health status, and predicting the risk of periodontal disease.

[0479] A "notification means" is a device or method for sending a notification to a user when an anomaly is detected based on the analysis results.

[0480] "Plan generation means" refers to a technology or device for generating an individualized oral care plan, taking into account the user's lifestyle pattern information.

[0481] "Information presentation means" refers to a technology or device that uses a video device with audio output capabilities installed in a physical store to provide users with analysis results and recommended oral care methods visually and audibly.

[0482] This invention constructs a system for monitoring oral health and providing users with personalized care plans. The system utilizes the user's smart device and digital signage installed in physical stores to acquire information in real time and provide analysis results and care plans.

[0483] First, the user uses a dedicated application on their smart device to collect image and audio information from their oral cavity. The device encrypts this data and sends it to a server in the cloud. The server uses a generative AI model with machine learning algorithms to analyze the received data and assess the risk of periodontal disease and other oral health conditions. This analysis generates a personalized oral care plan based on the user's health condition.

[0484] The analysis results and care plan are not only notified to the user's smart device, but are also provided as visual and auditory information on video devices with audio output capabilities installed in physical stores. This allows users to receive consistent care advice both at home and in stores.

[0485] For example, a customer who undergoes an oral check at a drugstore might receive a notification about their periodontal disease risk on the spot, prompting them to immediately purchase appropriate oral care products. Another example of a prompt message for the generating AI model is: "Analyze the oral images, assess the periodontal disease risk, and generate recommended care methods based on the risk level."

[0486] This system enhances user convenience and streamlines the maintenance and management of oral health.

[0487] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0488] Step 1:

[0489] The user uses a dedicated application on a smart device to acquire image and audio information from inside their mouth. The user takes pictures of the inside of their mouth with a camera and collects audio data through a microphone. The input is images and audio information of the user's mouth, and the output is this data.

[0490] Step 2:

[0491] The device encrypts the image and audio information it acquires. The device uses encryption technologies such as the SSL / TLS protocol to ensure data security. The input to this process is the captured image and audio information, and the output is the encrypted image and audio information.

[0492] Step 3:

[0493] The device sends encrypted data to a server in the cloud. The device uses a secure communication method to send encrypted data to a predetermined server address. The input is encrypted image and audio information, and the output is the transmission of data to the server.

[0494] Step 4:

[0495] The server analyzes the received data using a generative AI model based on machine learning algorithms. The server utilizes this generative AI model to analyze the condition of the gums from image information and detect abnormalities in oral sounds from audio information. The input to this process is the decrypted result of the received encrypted data, and the output is the analysis result indicating the user's oral health status.

[0496] Step 5:

[0497] The server generates an oral care plan for the user based on the analysis results and creates a notification. The server generates a notification that takes into account the user's lifestyle and health data, considering their individual care plan, and converts it into text format. The inputs to this process are the analysis results and the user's lifestyle information, and the outputs are the personalized care plan and notification message.

[0498] Step 6:

[0499] The server generates notification messages and sends them to terminals and in-store video displays. The server sends notifications to terminals via SMS, email, or a dedicated app, and provides them to users visually and audibly in in-store settings. The inputs to this process are the generated care plans and notification messages, and the output is the information displayed on terminal screens and video displays.

[0500] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0501] This invention is a system for understanding the user's oral health status and providing periodontal disease prevention measures. By combining it with an emotional engine, it realizes customized care that responds to the user's emotional state.

[0502] The user periodically acquires images and audio data of their oral cavity using a dedicated application installed on their smart device. This data is temporarily stored on the device, encrypted, and then sent to a server. The server uses a generative AI model to analyze the received data, examining the condition of the user's gums and the sounds of their brushing.

[0503] Furthermore, this system incorporates an emotion engine that can recognize the user's emotional state. The emotion engine identifies the user's emotions from factors such as voice tone and facial expressions, and evaluates their motivation and stress level regarding oral care. For example, if the system determines that the user is experiencing stress, the plan generation mechanism will provide advice and product suggestions to alleviate that stress.

[0504] The server integrates and analyzes the user's emotional state and oral health data to generate an optimal care plan. Using natural language generation technology, it sends notifications in a format easily understood by the user. These notifications can be emotionally sensitive and include a relaxed tone and encouraging words.

[0505] For example, if user data indicates a risk of periodontal disease and stress levels are also identified, the server will select an approach that promotes self-care in a calming atmosphere. It will also support the user's periodontal disease prevention efforts by recommending simple and effective care methods. In this way, the present invention provides a more effective and personalized care service that comprehensively considers the user's emotions and oral health status.

[0506] The following describes the processing flow.

[0507] Step 1:

[0508] The user launches a dedicated app on their smart device and takes a picture of the inside of their mouth. The app instructs the user to capture the image and record audio, thereby obtaining detailed data about the inside of their mouth.

[0509] Step 2:

[0510] The device encrypts the acquired data and sends it to the server while protecting privacy. Secure protocols such as TLS are used in this process.

[0511] Step 3:

[0512] The server analyzes the received data. Image analysis algorithms are used to assess the condition of the gums, and potential problems are detected from the audio data. At this stage, foundational data for assessing periodontal disease risk is generated.

[0513] Step 4:

[0514] The emotion engine analyzes transmitted voice tone and facial expression data to recognize the user's emotional state. This identifies the user's current mood and stress level.

[0515] Step 5:

[0516] The server integrates the analysis results, combining emotional state and health data to more precisely assess the user's periodontal disease risk. If necessary, it detects anomalies in areas requiring specific attention.

[0517] Step 6:

[0518] A notification is sent from the server to the device. This notification includes the results of the periodontal disease risk assessment, along with personalized advice that takes into account the user's emotional state.

[0519] Step 7:

[0520] The plan generation system considers the user's lifestyle and emotional state, uses natural language generation technology to generate a specific self-care plan, and sends it to the device. The device then presents this plan to the user, supporting their oral care efforts.

[0521] (Example 2)

[0522] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0523] Conventional oral care support systems provide care plans without considering the user's emotional state, which can lead to insufficient user motivation and engagement with care. Therefore, there is a need to provide more effective care plans that take the user's emotional state into account.

[0524] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0525] In this invention, the server includes means for recognizing the user's emotional state from voice and image data, means for integrating and analyzing the emotional state and oral health status to generate an individualized care plan, and means for notifying the user of the generated care plan in natural language. This makes it possible to provide a care plan that takes the user's emotions into consideration.

[0526] A "user" is an individual who uses the system to provide oral care data and receive services.

[0527] "Oral condition" refers to information related to the user's oral health, specifically the condition of the gums and teeth, which is acquired as digital data.

[0528] "Image data" refers to a digital recording of visual information about the user's oral cavity.

[0529] "Audio data" refers to digital recordings of sounds produced when a user performs actions such as brushing their teeth.

[0530] "Device" is a general term for hardware and software used to acquire, transmit, and analyze data.

[0531] "Encryption" is the process of transforming the content of digital data using a specific algorithm in order to transmit it securely.

[0532] An "external device" is another computer system connected via a network for the purpose of processing and storing data.

[0533] "Analysis" is the process of examining the acquired data in detail and evaluating the user's oral health status.

[0534] "Risk" is an indicator that shows the potential for problems with the user's oral health.

[0535] "Notifications" refer to information that communicates analysis results or care plans to the user visually or audibly.

[0536] "Emotional state" refers to the user's psychological state, including their stress levels and relaxation levels.

[0537] "Creation" refers to the process of developing a care plan that is tailored to a specific purpose and is based on the user's characteristics.

[0538] "Natural language" refers to the language that humans use on a daily basis, and it is a means by which computer systems use this language to transmit information to users.

[0539] This system uses smart devices, a dedicated application, an external server, a generative AI model, and an emotion engine to assess the user's oral health and provide a care plan tailored to their emotional state.

[0540] The user utilizes an application installed on their smart device. This application uses the device's camera and microphone to acquire image data of the oral cavity and audio data of brushing. The acquired data is temporarily stored on the device, encrypted using technologies such as AES encryption, and then securely transmitted to an external server.

[0541] The server utilizes generative AI models to analyze the received data. Specifically, it uses machine learning libraries such as TensorFlow and PyTorch to evaluate the user's gum condition and analyze the accuracy and intensity of brushing from the audio data.

[0542] Furthermore, the server implements an emotion engine that recognizes the user's emotional state by analyzing voice tone and facial expression data. This recognition uses OpenCV and NLP techniques to extract features from voice and images and determine stress and relaxation levels.

[0543] The server integrates this data to comprehensively evaluate the user's oral health and emotional state. Then, in the process of generating the optimal care plan for the user, it uses natural language generation technology to create and deliver notifications that are easy for the user to understand.

[0544] For example, if a user is experiencing stress and is at risk of periodontal disease, the server will suggest relaxation techniques along with specific advice to enhance daily self-care. This notification can include a relaxed tone and encouraging words.

[0545] For example, the prompt can be set as follows:

[0546] "Analyze the user's oral cavity images and voice data to generate a customized care plan that takes into account their health status and emotions, and notify them in natural language."

[0547] In this way, the present invention provides more effective and personalized care services.

[0548] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0549] Step 1:

[0550] The user launches the app, takes an image of their oral cavity with their smart device's camera, and records brushing sounds with the microphone. The input consists of image and audio data of the user's oral cavity. This data acquisition provides basic data necessary to understand the user's current condition.

[0551] Step 2:

[0552] The device temporarily stores the image and audio data it acquires and processes the data securely using AES encryption. The input is the image and audio data acquired in step 1, and the output is encrypted data. This prepares the device to securely transmit user data externally.

[0553] Step 3:

[0554] The terminal sends encrypted data to an external server. The input is the encrypted data, and the output is the completion of the data transmission to the server. The terminal's communication function plays a crucial role here, and the data is transmitted using protocols such as TLS.

[0555] Step 4:

[0556] The server decodes the received data and analyzes it through a generative AI model. Here, TensorFlow and PyTorch are used to evaluate the condition of the gums from image data and analyze the quality of brushing from audio data. The input is decoded image and audio data, and the output is the analysis result indicating the user's oral health status.

[0557] Step 5:

[0558] The server uses an emotion engine to analyze voice tone and facial image data to determine the user's emotional state. The input is the same decoded voice and image data, and the output is an evaluation of the user's emotional state. The server uses the engine to analyze stress and relaxation levels.

[0559] Step 6:

[0560] The server integrates health and emotional state data and generates an optimal care plan using an AI model. A personalized care plan is created using natural language generation technology and communicated to the user in the form of a notification. The input is the analysis results of the health state and the evaluation of the emotional state, and the output is a customized care plan. This process uses prompts to create notifications that are easy for the user to remember.

[0561] (Application Example 2)

[0562] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0563] In modern society, maintaining oral and mental health is crucial, yet they often influence each other. In particular, there is a lack of technology to mitigate the impact of stress on oral health while providing effective care plans tailored to individual circumstances. This invention aims to comprehensively analyze a user's oral and emotional state and provide appropriate care while reducing stress-related risks.

[0564] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0565] In this invention, the server includes information acquisition means for acquiring the user's oral cavity condition as image and audio information, information transmission means for encrypting the acquired information and transmitting it to an external processing device, and analysis means for analyzing the oral health condition and stress level based on the received information and the user's emotional state. This makes it possible to effectively provide the user with an individualized care plan and stress management measures.

[0566] "Information acquisition means" refers to a device equipped with the function of acquiring the user's oral cavity condition as image and audio information.

[0567] "Information transmission means" refers to a device equipped with the function of encrypting acquired information and transmitting it to an external processing device.

[0568] The "analysis means" is equipped with the function to analyze the user's oral health status based on the received information and predict the risk of periodontal disease, and further includes the function to identify the user's emotional state by recognizing emotions and to evaluate the stress level.

[0569] A "notification mechanism" is a system equipped with the functionality to send notifications to the user when an anomaly is detected based on the analysis results and emotional state.

[0570] The "plan generation means" is a system that takes into account the user's lifestyle information to generate an individualized oral care plan, and also has the function of providing it in a format that is easy for the user to understand using natural language generation technology.

[0571] A "suggestion tool" is a system equipped with functions to provide relaxation methods and security advice tailored to different stress levels.

[0572] This invention is a system that analyzes the user's oral cavity condition and emotional state, and provides personalized oral care and stress reduction measures. This system includes information acquisition means, information transmission means, analysis means, notification means, plan generation means, and proposal means.

[0573] The system uses a smartphone or other device to periodically acquire image and audio information of the inside of the mouth. It utilizes the camera and microphone of the smart device to acquire data via a dedicated application. The acquired data is encrypted on the spot and sent to an external processing unit, i.e., a server.

[0574] The server is equipped with a generative AI model for analyzing received data. This AI model analyzes the user's oral health status and predicts the risk of periodontal disease. It also has an emotion engine for emotion recognition, which evaluates the user's emotional state, particularly stress level, based on voice tone and facial expression data. This utilizes speech recognition and facial recognition technologies.

[0575] Based on the analysis results, the server sends a notification to the user. The notification includes a user-friendly message created using natural language generation technology, presenting a specific care plan along with relaxation techniques and security advice. This process utilizes a text generation API to provide user-friendly information in real time.

[0576] For example, if analysis reveals a user is at risk of periodontal disease and also identifies a high-stress state, the server sends a notification that includes advice on password management, along with simple stretching and deep breathing exercises the user can perform. In this way, the present invention comprehensively supports the user's oral and mental health.

[0577] An example of a prompt message for the generating AI model would be: "Please describe the security risks and countermeasures based on the user's oral health and emotional analysis results. Please include specific relaxation methods and security measures in your suggestions."

[0578] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0579] Step 1:

[0580] The user uses a device to acquire images and audio information of the oral cavity. Input is real-time data from the smart device's camera and microphone. This data is temporarily stored on the device and converted to an appropriate format. Output is an encrypted data file.

[0581] Step 2:

[0582] The terminal sends the encrypted data obtained in step 1 to the server, which is an external processing unit, using the information transmission means. The input is the encrypted data file, and the output is the completion of the data transfer to the server.

[0583] Step 3:

[0584] The server processes the received data using analytical tools. The input is oral cavity data transmitted from the terminal, and the analysis uses a generative AI model to evaluate oral health status and predict the risk of periodontal disease. The output is the risk assessment data of the analysis results.

[0585] Step 4:

[0586] The server uses an emotion engine to analyze voice tone and facial data to recognize the user's emotional state. The input is the same data as in step 3, and the output is emotion evaluation data indicating the user's stress level. Voice analysis and facial expression recognition technology are used for data processing.

[0587] Step 5:

[0588] The server sends a notification to the user using a notification system based on the analysis results and sentiment evaluation data. The input is the output data from steps 3 and 4, and an appropriate message is generated using natural language generation. The output is a notification that includes specific care and advice for the user.

[0589] Step 6:

[0590] Based on the notifications received, the user performs suggested relaxation techniques and security advice. The input is the notification message from step 5, and the output is the user's behavioral changes and stress reduction. This step includes specific actions such as deep breathing, stretching, and reviewing security settings.

[0591] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0592] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0593] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0594] [Fourth Embodiment]

[0595] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0596] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0597] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0598] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0599] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0600] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0601] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0602] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0603] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0604] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0605] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0606] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0607] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0608] This invention is embodied as a system that periodically records the condition of the oral cavity using the user's smart device, analyzes the data using AI, and detects the risk of periodontal disease early, thereby providing preventive measures. This system mainly consists of the user's smart device, a server that processes the data, and a communication infrastructure that connects them.

[0609] Users capture images of their oral cavity and acquire audio data using a dedicated application on their smart device. This data is temporarily stored on the device and transmitted to a server using a secure communication method. The server employs a generative AI model that utilizes machine learning algorithms to analyze the received data. This AI analyzes the condition of the gums in detail from the image data and detects abnormalities in oral sounds using the audio data.

[0610] Based on the analysis results, the server assesses the user's periodontal disease risk and sends a notification to the device according to the risk level. This notification includes information about the area where abnormalities were detected, the degree of risk, and specific countermeasures. The device displays the received information to the user in an easy-to-understand visual format and provides guidance on appropriate care.

[0611] Furthermore, the server uses natural language generation technology to create a personalized oral care plan, taking into account the user's saved lifestyle data. This plan supports users in practicing self-care without getting lost due to lack of knowledge or time.

[0612] For example, if gum swelling is detected in the early stages from daily image data, the server notifies the terminal of the risk. Based on the user's eating habits, it provides advice on reducing excessive sugar intake and information on recommended oral care products. In this way, the present invention provides an oral care service that is useful for users to proactively manage their health and reduce the risk of developing periodontal disease.

[0613] The following describes the processing flow.

[0614] Step 1:

[0615] The user launches a dedicated app on their smart device. The app instructs the user to take images of their oral cavity and simultaneously records necessary audio data. This obtains data about the user's current oral condition.

[0616] Step 2:

[0617] The device temporarily stores the acquired image and audio data and encrypts them. Encryption is performed to ensure the confidentiality of the data.

[0618] Step 3:

[0619] Encrypted data is sent from the terminal to the server. A secure communication protocol reduces the risk of data leakage during transit.

[0620] Step 4:

[0621] The server uses a generative AI model to analyze the data it receives. The AI ​​analyzes the condition of the gums from image data and detects abnormal patterns. It also detects abnormalities from audio data based on brushing sounds and oral sounds.

[0622] Step 5:

[0623] The server evaluates the user's periodontal disease risk based on the analysis results and determines the risk level. The server then prepares to send an alert to the terminal according to the risk level.

[0624] Step 6:

[0625] The device receives alerts sent from the server and displays them visually to the user. The display includes detailed analysis results and recommended actions. The user can then take self-care actions based on this information.

[0626] Step 7:

[0627] Furthermore, the server generates a personalized oral care plan, taking into account the user's lifestyle and daily data. It utilizes natural language generation technology to send the information to the device in an easy-to-understand format.

[0628] (Example 1)

[0629] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0630] Continuously and regularly monitoring oral health and detecting the risk of periodontal disease early is a challenging task for individual users. Furthermore, providing appropriate oral care plans tailored to individual lifestyles and health conditions is essential for effective self-care. However, conventional methods lacked systems to address these challenges.

[0631] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0632] This invention includes a server that uses a machine learning model to analyze the user's oral health status and predict and evaluate the risk of periodontal disease; a server that sends a notification to the user if an abnormality is detected based on the analysis results; and a server that generates an individualized oral care plan considering the user's lifestyle information. This allows the user to easily understand the condition of their own oral cavity and receive specific health management and preventive measures tailored to their lifestyle.

[0633] "Data acquisition means" refers to a device or method for acquiring the condition of a user's oral cavity as image data and audio data.

[0634] "Data transmission means" refers to a device or method for encrypting acquired image data and audio data and securely transmitting them to an external processing device.

[0635] "Analysis means" refers to a device or method for analyzing the user's oral health status using a machine learning model based on received data, and for predicting and evaluating the risk of periodontal disease.

[0636] "Notification means" refers to a device or method for transmitting information to the user when an anomaly is detected based on the analysis results.

[0637] "Plan generation means" refers to a device or method for generating an individualized oral care plan while taking into account the user's lifestyle information.

[0638] A "machine learning model" is a mathematical model used to compare past and present data, recognize patterns, and make predictions about new data.

[0639] "Natural language generation technology" is a technology that enables computers to generate text in a user-friendly format based on text data.

[0640] This invention is a system that utilizes a user's smart device to periodically acquire images and audio data from inside the oral cavity, analyzes this data, and provides an individualized oral care plan. It mainly consists of a smart device, a server, and a communication network.

[0641] Users install a dedicated application on their smart device and use it to capture images of their oral cavity and collect audio data. The device's built-in camera and microphone are used to acquire both image and audio data. The acquired data is temporarily stored on the smart device and then transmitted to the server using secure protocols such as SSL / TLS.

[0642] The server performs detailed analysis on the received data using a specific machine learning model. This generative AI model has the ability to identify the condition of the gums from image data and detect abnormalities in oral sounds from audio data. This allows for the prediction of periodontal disease risk and enables early intervention as needed.

[0643] Based on the analysis, the server generates information about the user's health status and sends notifications to the user's smart device according to the risk level. These notifications include the specific location and severity of any abnormalities, as well as recommended countermeasures. The smart device also presents the information visually in an easy-to-understand manner, supporting intuitive understanding for the user.

[0644] Furthermore, this system uses natural language generation technology to create a personalized oral care plan, referencing the user's lifestyle information. This plan serves as a guide for self-care that the user can implement on a daily basis.

[0645] For example, if slight gum swelling is detected from captured image data, the server will notify the user of the risk and send advice to reduce sugar intake as a way to improve their diet. It will also provide product information that can be used for actual self-care.

[0646] An example of a prompt message is: "Analyze this user's most recent intraoral image and assess their health status. If any abnormalities are found, indicate their location and risk level, and suggest specific corrective measures."

[0647] In this way, the present invention operates as a system that supports users' autonomy in health management and plays a role in reducing the risk of periodontal disease.

[0648] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0649] Step 1:

[0650] The user launches an application on their smart device, takes an image of their mouth, and records audio. The input consists of image and audio data acquired using the smart device's camera and microphone. Specifically, the user takes a picture of their mouth with the camera and taps the audio input button to record audio. The output—image and audio data—is saved on the smart device.

[0651] Step 2:

[0652] The terminal encrypts the acquired data and sends it to the server. The input consists of image data and audio data stored on the terminal. As part of the data processing, the data is encrypted using the SSL / TLS protocol. Specifically, the terminal executes the data encryption process and establishes a communication channel with the server. As output, the encrypted data is sent to the server via a secure communication path.

[0653] Step 3:

[0654] The server prepares the received data for analysis. The input consists of encrypted image and audio data sent to the server. The data processing involves decrypting the data and converting it into an analyzable format. The output is data suitable for analysis.

[0655] Step 4:

[0656] The server activates a generation AI model and analyzes image and audio data using prompt text as input. For data processing, an image analysis algorithm examines the condition of the gums, and an audio analysis algorithm checks for abnormalities in the audio data. Specifically, the AI ​​model identifies abnormal areas in the image and identifies abnormal patterns in the audio. The output provides analysis results regarding health status and risk levels.

[0657] Step 5:

[0658] The server evaluates the user's periodontal disease risk based on the analysis results, creates a notification message, and sends it to the terminal. The input is the analysis results from the AI ​​model. As data processing, it performs specific actions to generate a message according to the risk level. As output, the notification message is sent to the user's terminal.

[0659] Step 6:

[0660] The terminal displays received notifications to the user. The input is the notification message sent from the server. Specifically, the terminal uses its display function to visually present the message to the user. The output provides information that guides the user's next course of action.

[0661] Step 7:

[0662] The server uses the user's lifestyle information to create a personalized oral care plan using a generative AI model. The input consists of the user's lifestyle data and the results of the AI ​​model's analysis. Specifically, natural language generation technology is used to create the care plan as text. The output is a user-specific oral care plan, which is then sent to the terminal.

[0663] (Application Example 1)

[0664] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0665] While it is important to properly manage oral health and detect the risk of periodontal disease early, providing specific care suggestions based on each user's health condition and lifestyle in real time is not easy. Furthermore, even in physical stores, there is a lack of means to provide consumers with useful oral care information on the spot and promote product use.

[0666] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0667] In this invention, the server includes data collection means, information transmission means, analysis means, information presentation means, and plan generation means. This allows users to understand their oral health status and receive individually optimized oral care plans in real time via their smart devices or video devices with audio output capabilities in physical stores.

[0668] "Data collection means" refers to devices and methods for acquiring the user's oral cavity condition as image and audio information.

[0669] "Information transmission means" refers to devices or technologies for encrypting collected oral cavity information and transmitting it to an external information processing device.

[0670] "Analysis means" refers to a technology or device for analyzing received information, evaluating the user's oral health status, and predicting the risk of periodontal disease.

[0671] A "notification means" is a device or method for sending a notification to a user when an anomaly is detected based on the analysis results.

[0672] "Plan generation means" refers to a technology or device for generating an individualized oral care plan, taking into account the user's lifestyle pattern information.

[0673] "Information presentation means" refers to a technology or device that uses a video device with audio output capabilities installed in a physical store to provide users with analysis results and recommended oral care methods visually and audibly.

[0674] This invention constructs a system for monitoring oral health and providing users with personalized care plans. The system utilizes the user's smart device and digital signage installed in physical stores to acquire information in real time and provide analysis results and care plans.

[0675] First, the user uses a dedicated application on their smart device to collect image and audio information from their oral cavity. The device encrypts this data and sends it to a server in the cloud. The server uses a generative AI model with machine learning algorithms to analyze the received data and assess the risk of periodontal disease and other oral health conditions. This analysis generates a personalized oral care plan based on the user's health condition.

[0676] The analysis results and care plan are not only notified to the user's smart device, but are also provided as visual and auditory information on video devices with audio output capabilities installed in physical stores. This allows users to receive consistent care advice both at home and in stores.

[0677] For example, a customer who undergoes an oral check at a drugstore might receive a notification about their periodontal disease risk on the spot, prompting them to immediately purchase appropriate oral care products. Another example of a prompt message for the generating AI model is: "Analyze the oral images, assess the periodontal disease risk, and generate recommended care methods based on the risk level."

[0678] This system enhances user convenience and streamlines the maintenance and management of oral health.

[0679] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0680] Step 1:

[0681] The user uses a dedicated application on a smart device to acquire image and audio information from inside their mouth. The user takes pictures of the inside of their mouth with a camera and collects audio data through a microphone. The input is images and audio information of the user's mouth, and the output is this data.

[0682] Step 2:

[0683] The device encrypts the image and audio information it acquires. The device uses encryption technologies such as the SSL / TLS protocol to ensure data security. The input to this process is the captured image and audio information, and the output is the encrypted image and audio information.

[0684] Step 3:

[0685] The device sends encrypted data to a server in the cloud. The device uses a secure communication method to send encrypted data to a predetermined server address. The input is encrypted image and audio information, and the output is the transmission of data to the server.

[0686] Step 4:

[0687] The server analyzes the received data using a generative AI model based on machine learning algorithms. The server utilizes this generative AI model to analyze the condition of the gums from image information and detect abnormalities in oral sounds from audio information. The input to this process is the decrypted result of the received encrypted data, and the output is the analysis result indicating the user's oral health status.

[0688] Step 5:

[0689] The server generates an oral care plan for the user based on the analysis results and creates a notification. The server generates a notification that takes into account the user's lifestyle and health data, considering their individual care plan, and converts it into text format. The inputs to this process are the analysis results and the user's lifestyle information, and the outputs are the personalized care plan and notification message.

[0690] Step 6:

[0691] The server generates notification messages and sends them to terminals and in-store video displays. The server sends notifications to terminals via SMS, email, or a dedicated app, and provides them to users visually and audibly in in-store settings. The inputs to this process are the generated care plans and notification messages, and the output is the information displayed on terminal screens and video displays.

[0692] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0693] This invention is a system for understanding the user's oral health status and providing periodontal disease prevention measures. By combining it with an emotional engine, it realizes customized care that responds to the user's emotional state.

[0694] The user periodically acquires images and audio data of their oral cavity using a dedicated application installed on their smart device. This data is temporarily stored on the device, encrypted, and then sent to a server. The server uses a generative AI model to analyze the received data, examining the condition of the user's gums and the sounds of their brushing.

[0695] Furthermore, this system incorporates an emotion engine that can recognize the user's emotional state. The emotion engine identifies the user's emotions from factors such as voice tone and facial expressions, and evaluates their motivation and stress level regarding oral care. For example, if the system determines that the user is experiencing stress, the plan generation mechanism will provide advice and product suggestions to alleviate that stress.

[0696] The server integrates and analyzes the user's emotional state and oral health data to generate an optimal care plan. Using natural language generation technology, it sends notifications in a format easily understood by the user. These notifications can be emotionally sensitive and include a relaxed tone and encouraging words.

[0697] For example, if user data indicates a risk of periodontal disease and stress levels are also identified, the server will select an approach that promotes self-care in a calming atmosphere. It will also support the user's periodontal disease prevention efforts by recommending simple and effective care methods. In this way, the present invention provides a more effective and personalized care service that comprehensively considers the user's emotions and oral health status.

[0698] The following describes the processing flow.

[0699] Step 1:

[0700] The user launches a dedicated app on their smart device and takes a picture of the inside of their mouth. The app instructs the user to capture the image and record audio, thereby obtaining detailed data about the inside of their mouth.

[0701] Step 2:

[0702] The device encrypts the acquired data and sends it to the server while protecting privacy. Secure protocols such as TLS are used in this process.

[0703] Step 3:

[0704] The server analyzes the received data. Image analysis algorithms are used to assess the condition of the gums, and potential problems are detected from the audio data. At this stage, foundational data for assessing periodontal disease risk is generated.

[0705] Step 4:

[0706] The emotion engine analyzes transmitted voice tone and facial expression data to recognize the user's emotional state. This identifies the user's current mood and stress level.

[0707] Step 5:

[0708] The server integrates the analysis results, combining emotional state and health data to more precisely assess the user's periodontal disease risk. If necessary, it detects anomalies in areas requiring specific attention.

[0709] Step 6:

[0710] A notification is sent from the server to the device. This notification includes the results of the periodontal disease risk assessment, along with personalized advice that takes into account the user's emotional state.

[0711] Step 7:

[0712] The plan generation system considers the user's lifestyle and emotional state, uses natural language generation technology to generate a specific self-care plan, and sends it to the device. The device then presents this plan to the user, supporting their oral care efforts.

[0713] (Example 2)

[0714] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0715] Conventional oral care support systems provide care plans without considering the user's emotional state, which can lead to insufficient user motivation and engagement with care. Therefore, there is a need to provide more effective care plans that take the user's emotional state into account.

[0716] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0717] In this invention, the server includes means for recognizing the user's emotional state from voice and image data, means for integrating and analyzing the emotional state and oral health status to generate an individualized care plan, and means for notifying the user of the generated care plan in natural language. This makes it possible to provide a care plan that takes the user's emotions into consideration.

[0718] A "user" is an individual who uses the system to provide oral care data and receive services.

[0719] "Oral condition" refers to information related to the user's oral health, specifically the condition of the gums and teeth, which is acquired as digital data.

[0720] "Image data" refers to a digital recording of visual information about the user's oral cavity.

[0721] "Audio data" refers to digital recordings of sounds produced when a user performs actions such as brushing their teeth.

[0722] "Device" is a general term for hardware and software used to acquire, transmit, and analyze data.

[0723] "Encryption" is the process of transforming the content of digital data using a specific algorithm in order to transmit it securely.

[0724] An "external device" is another computer system connected via a network for the purpose of processing and storing data.

[0725] "Analysis" is the process of examining the acquired data in detail and evaluating the user's oral health status.

[0726] "Risk" is an indicator that shows the potential for problems with the user's oral health.

[0727] "Notifications" refer to information that communicates analysis results or care plans to the user visually or audibly.

[0728] "Emotional state" refers to the user's psychological state, including their stress levels and relaxation levels.

[0729] "Creation" refers to the process of developing a care plan that is tailored to a specific purpose and is based on the user's characteristics.

[0730] "Natural language" refers to the language that humans use on a daily basis, and it is a means by which computer systems use this language to transmit information to users.

[0731] This system uses smart devices, a dedicated application, an external server, a generative AI model, and an emotion engine to assess the user's oral health and provide a care plan tailored to their emotional state.

[0732] The user utilizes an application installed on their smart device. This application uses the device's camera and microphone to acquire image data of the oral cavity and audio data of brushing. The acquired data is temporarily stored on the device, encrypted using technologies such as AES encryption, and then securely transmitted to an external server.

[0733] The server utilizes generative AI models to analyze the received data. Specifically, it uses machine learning libraries such as TensorFlow and PyTorch to evaluate the user's gum condition and analyze the accuracy and intensity of brushing from the audio data.

[0734] Furthermore, the server implements an emotion engine that recognizes the user's emotional state by analyzing voice tone and facial expression data. This recognition uses OpenCV and NLP techniques to extract features from voice and images and determine stress and relaxation levels.

[0735] The server integrates this data to comprehensively evaluate the user's oral health and emotional state. Then, in the process of generating the optimal care plan for the user, it uses natural language generation technology to create and deliver notifications that are easy for the user to understand.

[0736] For example, if a user is experiencing stress and is at risk of periodontal disease, the server will suggest relaxation techniques along with specific advice to enhance daily self-care. This notification can include a relaxed tone and encouraging words.

[0737] For example, the prompt can be set as follows:

[0738] "Analyze the user's oral cavity images and voice data to generate a customized care plan that takes into account their health status and emotions, and notify them in natural language."

[0739] In this way, the present invention provides more effective and personalized care services.

[0740] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0741] Step 1:

[0742] The user launches the app, takes an image of their oral cavity with their smart device's camera, and records brushing sounds with the microphone. The input consists of image and audio data of the user's oral cavity. This data acquisition provides basic data necessary to understand the user's current condition.

[0743] Step 2:

[0744] The device temporarily stores the image and audio data it acquires and processes the data securely using AES encryption. The input is the image and audio data acquired in step 1, and the output is encrypted data. This prepares the device to securely transmit user data externally.

[0745] Step 3:

[0746] The terminal sends encrypted data to an external server. The input is the encrypted data, and the output is the completion of the data transmission to the server. The terminal's communication function plays a crucial role here, and the data is transmitted using protocols such as TLS.

[0747] Step 4:

[0748] The server decodes the received data and analyzes it through a generative AI model. Here, TensorFlow and PyTorch are used to evaluate the condition of the gums from image data and analyze the quality of brushing from audio data. The input is decoded image and audio data, and the output is the analysis result indicating the user's oral health status.

[0749] Step 5:

[0750] The server uses an emotion engine to analyze voice tone and facial image data to determine the user's emotional state. The input is the same decoded voice and image data, and the output is an evaluation of the user's emotional state. The server uses the engine to analyze stress and relaxation levels.

[0751] Step 6:

[0752] The server integrates health and emotional state data and generates an optimal care plan using an AI model. A personalized care plan is created using natural language generation technology and communicated to the user in the form of a notification. The input is the analysis results of the health state and the evaluation of the emotional state, and the output is a customized care plan. This process uses prompts to create notifications that are easy for the user to remember.

[0753] (Application Example 2)

[0754] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0755] In modern society, maintaining oral and mental health is crucial, yet they often influence each other. In particular, there is a lack of technology to mitigate the impact of stress on oral health while providing effective care plans tailored to individual circumstances. This invention aims to comprehensively analyze a user's oral and emotional state and provide appropriate care while reducing stress-related risks.

[0756] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0757] In this invention, the server includes information acquisition means for acquiring the user's oral cavity condition as image and audio information, information transmission means for encrypting the acquired information and transmitting it to an external processing device, and analysis means for analyzing the oral health condition and stress level based on the received information and the user's emotional state. This makes it possible to effectively provide the user with an individualized care plan and stress management measures.

[0758] "Information acquisition means" refers to a device equipped with the function of acquiring the user's oral cavity condition as image and audio information.

[0759] "Information transmission means" refers to a device equipped with the function of encrypting acquired information and transmitting it to an external processing device.

[0760] The "analysis means" is equipped with the function to analyze the user's oral health status based on the received information and predict the risk of periodontal disease, and further includes the function to identify the user's emotional state by recognizing emotions and to evaluate the stress level.

[0761] A "notification mechanism" is a system equipped with the functionality to send notifications to the user when an anomaly is detected based on the analysis results and emotional state.

[0762] The "plan generation means" is a system that takes into account the user's lifestyle information to generate an individualized oral care plan, and also has the function of providing it in a format that is easy for the user to understand using natural language generation technology.

[0763] A "suggestion tool" is a system equipped with functions to provide relaxation methods and security advice tailored to different stress levels.

[0764] This invention is a system that analyzes the user's oral cavity condition and emotional state, and provides personalized oral care and stress reduction measures. This system includes information acquisition means, information transmission means, analysis means, notification means, plan generation means, and proposal means.

[0765] The system uses a smartphone or other device to periodically acquire image and audio information of the inside of the mouth. It utilizes the camera and microphone of the smart device to acquire data via a dedicated application. The acquired data is encrypted on the spot and sent to an external processing unit, i.e., a server.

[0766] The server is equipped with a generative AI model for analyzing received data. This AI model analyzes the user's oral health status and predicts the risk of periodontal disease. It also has an emotion engine for emotion recognition, which evaluates the user's emotional state, particularly stress level, based on voice tone and facial expression data. This utilizes speech recognition and facial recognition technologies.

[0767] Based on the analysis results, the server sends a notification to the user. The notification includes a user-friendly message created using natural language generation technology, presenting a specific care plan along with relaxation techniques and security advice. This process utilizes a text generation API to provide user-friendly information in real time.

[0768] For example, if analysis reveals a user is at risk of periodontal disease and also identifies a high-stress state, the server sends a notification that includes advice on password management, along with simple stretching and deep breathing exercises the user can perform. In this way, the present invention comprehensively supports the user's oral and mental health.

[0769] An example of a prompt message for the generating AI model would be: "Please describe the security risks and countermeasures based on the user's oral health and emotional analysis results. Please include specific relaxation methods and security measures in your suggestions."

[0770] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0771] Step 1:

[0772] The user uses a device to acquire images and audio information of the oral cavity. Input is real-time data from the smart device's camera and microphone. This data is temporarily stored on the device and converted to an appropriate format. Output is an encrypted data file.

[0773] Step 2:

[0774] The terminal sends the encrypted data obtained in step 1 to the server, which is an external processing unit, using the information transmission means. The input is the encrypted data file, and the output is the completion of the data transfer to the server.

[0775] Step 3:

[0776] The server processes the received data using analytical tools. The input is oral cavity data transmitted from the terminal, and the analysis uses a generative AI model to evaluate oral health status and predict the risk of periodontal disease. The output is the risk assessment data of the analysis results.

[0777] Step 4:

[0778] The server uses an emotion engine to analyze voice tone and facial data to recognize the user's emotional state. The input is the same data as in step 3, and the output is emotion evaluation data indicating the user's stress level. Voice analysis and facial expression recognition technology are used for data processing.

[0779] Step 5:

[0780] The server sends a notification to the user using a notification system based on the analysis results and sentiment evaluation data. The input is the output data from steps 3 and 4, and an appropriate message is generated using natural language generation. The output is a notification that includes specific care and advice for the user.

[0781] Step 6:

[0782] Based on the notifications received, the user performs suggested relaxation techniques and security advice. The input is the notification message from step 5, and the output is the user's behavioral changes and stress reduction. This step includes specific actions such as deep breathing, stretching, and reviewing security settings.

[0783] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0784] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0785] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0786] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0787] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0788] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0789] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0790] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0791] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0792] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0793] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0794] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0795] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0796] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0797] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0798] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0799] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0800] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0801] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0802] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0803] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0804] The following is further disclosed regarding the embodiments described above.

[0805] (Claim 1)

[0806] A data acquisition means for acquiring the user's oral cavity condition as image data and audio data,

[0807] A data transmission means that encrypts the acquired data and sends it to an external server,

[0808] An analytical means that analyzes the user's oral health status based on received data and predicts the risk of periodontal disease,

[0809] A notification system that sends a notification to the user when an anomaly is detected based on the analysis results,

[0810] A plan generation means that generates an individualized oral care plan considering the user's lifestyle information,

[0811] A system that includes this.

[0812] (Claim 2)

[0813] The system according to claim 1, wherein the analysis means compares past data with current data using a machine learning algorithm.

[0814] (Claim 3)

[0815] The system according to claim 1, wherein the plan generation means utilizes natural language generation technology to provide a care plan in a format that is easy for the user to understand.

[0816] "Example 1"

[0817] (Claim 1)

[0818] A data acquisition means for acquiring the user's oral cavity condition as image data and audio data,

[0819] A data transmission means for encrypting the acquired data and transmitting it to an external processing device,

[0820] An analytical means using a machine learning model to analyze the user's oral health status based on received data and predict and evaluate the risk of periodontal disease,

[0821] A notification system that sends a notification to the user when an anomaly is detected based on the analysis results,

[0822] A plan generation means that generates an individualized oral care plan considering the user's lifestyle information,

[0823] A means of supporting health management by presenting countermeasures tailored to the user's health condition,

[0824] A system that includes this.

[0825] (Claim 2)

[0826] The system according to claim 1, wherein the analysis means compares past data with current data using a machine learning algorithm and analyzes it using a prompt sentence with a generative AI model.

[0827] (Claim 3)

[0828] The system according to claim 1, wherein the plan generation means utilizes natural language generation technology to provide a care plan that is easy for the user to understand and is personalized.

[0829] "Application Example 1"

[0830] (Claim 1)

[0831] A data collection means for acquiring the user's oral cavity condition as image and audio information,

[0832] Information transmission means for encrypting the collected information and transmitting it to an external information processing device,

[0833] An analytical means that analyzes the user's oral health status based on received information and predicts the risk of periodontal disease,

[0834] A notification means that sends a notification to the user when an anomaly is detected based on the analysis results,

[0835] A plan generation means that generates an individualized oral care plan considering the user's lifestyle pattern information,

[0836] An information presentation means that provides the user with the analysis results and recommended oral care methods visually and audibly using a video device with audio output function installed in a physical store,

[0837] A system that includes this.

[0838] (Claim 2)

[0839] The system according to claim 1, wherein the analysis means compares past information with current information using a machine learning algorithm.

[0840] (Claim 3)

[0841] The system according to claim 1, wherein the plan generation means utilizes natural language generation technology to provide a care plan in a format that is easy for the user to understand.

[0842] "Example 2 of combining an emotion engine"

[0843] (Claim 1)

[0844] A device that acquires the condition of the user's oral cavity as image data and audio data,

[0845] A device that encrypts the acquired data and transmits it to an external device,

[0846] A device that analyzes the user's oral health status based on received data and predicts risks,

[0847] A device that sends a notification to the user if an anomaly is detected based on the analysis results,

[0848] A device that recognizes the user's emotional state from voice and image data,

[0849] A device that integrates and analyzes emotional state and oral health status to generate an individualized oral care plan,

[0850] A device that notifies the user of a care plan generated in natural language,

[0851] A system that includes this.

[0852] (Claim 2)

[0853] The system according to claim 1, wherein the analyzing device uses a learning algorithm to compare past data with current data and evaluate the emotional state.

[0854] (Claim 3)

[0855] The system according to claim 1, wherein the generating device utilizes the generation model to provide a care plan in a format that is easy for the user to understand.

[0856] "Application example 2 when combining with an emotional engine"

[0857] (Claim 1)

[0858] Information acquisition means for acquiring the user's oral cavity condition as image information and audio information,

[0859] Information transmission means for encrypting the acquired information and transmitting it to an external processing device,

[0860] An analytical means that analyzes the user's oral health status based on received information and predicts the risk of periodontal disease,

[0861] The aforementioned analysis means includes an analysis means that uses an emotion recognition function to identify the user's emotional state and evaluate the stress level,

[0862] A notification means that sends a notification to the user when an anomaly is detected based on the analysis results and emotional state,

[0863] A plan generation means that generates an individualized oral care plan taking into account the user's lifestyle information,

[0864] A means of providing suggestions for relaxation methods and security advice tailored to stress levels,

[0865] A system that includes this.

[0866] (Claim 2)

[0867] The system according to claim 1, wherein the analysis means compares past information with current information using a machine learning algorithm.

[0868] (Claim 3)

[0869] The system according to claim 1, wherein the plan generation means utilizes natural language generation technology to provide a care plan in a format that is easy for the user to understand. [Explanation of symbols]

[0870] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A data acquisition means for acquiring the user's oral cavity condition as image data and audio data, A data transmission means that encrypts the acquired data and sends it to an external server, An analytical means that analyzes the user's oral health status based on received data and predicts the risk of periodontal disease, A notification system that sends a notification to the user when an anomaly is detected based on the analysis results, A plan generation means that generates an individualized oral care plan considering the user's lifestyle information, A system that includes this.

2. The system according to claim 1, wherein the analysis means compares past data with current data using a machine learning algorithm.

3. The system according to claim 1, wherein the plan generation means utilizes natural language generation technology to provide a care plan in a format that is easy for the user to understand.

Images